Research project SR/00/320 (Research action SR)
CONTEXT AND OBJECTIVES
In Flanders, fruit-growing is an important and key economic activity. Fruit orchard productivity monitoring and a site specific management thereof becomes more important than ever due to the increasing demand for uniform batches of quality fruits, and the reduced use of pesticides and fertilizers for environmental sustainability. The common way to acquire tree productivity information is through visual inspections, which are subjective, demanding and time consuming. In many sectors, including fruit sorting, visual inspection is already replaced by computer vision for faster, cheaper and more objective information on number, size and quality of products. Image processing therefore seems the best option to automate the visual inspections in orchards as well. The hypothetical ‘optimal’ plant vitality remains a theoretical concept. However, it is generally accepted that plants experiencing stress differ in some characteristics (e.g., biomass, LAI, biochemical parameter content, and photosynthetic efficiency) from plants growing under optimal conditions, and these characteristics can therefore be considered as indices of productivity.
The HyperTemp project aims at developing an optimized orchard tree productivity monitoring system by integrating multi-date and multi-sensor remote sensing (RS) and in-situ data. An accurate productivity monitoring system provides the opportunity to increase profitability and reduce the environmental effects of farming by more closely matching the application of inputs such as pesticides and fertilizers with actual conditions in specific parts of the field.
The needs of the farmer will be addressed, i.e.:
- Which parcels do need specific management actions (a.o., irrigation, fertilization)?
- What are the yield expectations per management zone/ How many people to hire for harvest?
- How to maximize profit? Reduce the inputs by variable rate irrigation and fertigation?
- How to reduce the environmental effects of pesticides and fertilizers?
Objectives
This project proposal aims at acquiring detailed information about tree productivity in fruit orchards by integrating multi date and multisensor remote sensing data making optimal use of the Belair data. Plant vitality parameters which determine the productivity, and that will be taken into account in this study, include plant biomass, LAI, biochemical parameter content (chlorophyll, anthocyanins), photosynthetic efficiency (PRI index).
Following scientific issues can be defined and our possible answers to solve these issues are presented here :
- How to handle ‘Big Data’, i.e., excessive computing time, complex data handling, high dimensionality?
To address this issue, a robust object-based hierarchical image analysis procedure will be developed on single and a series of RPAS images using RGB, multispectral and hyperspectral cameras. For the latter, a multimodal segmentation technique will be developed.
- How to solve the constraint of ‘Single Date’ information which is often not informative enough to enable a distinction between different elements in an agricultural image scene.
Multitemporal imagery will be used. We will evaluate the added value of multidate to single date image processing. By applying a multitemporal classification algorithm including feature extraction, we will include spectral and texture features of all RS data, gathered throughout the growth season. Correlated and less informative features will be excluded.
- What is the impact of misregistration on the use of multidate, multisensor imagery?
Taking an optimal advantage of the complementarity of information offered by the use of multimodal and/or multidate data is already a scientific challenge by itself. It is essential that the different data sets are co-registered very accurately. To address this, we will investigate the impact of a potential misregistration of multidate multisensor images on the actual output of the project. This will be achieved by applying and optimizing a method to improve geometric accuracy and geometric correspondence of multisensor and multidate imagery by local geometric matching. Obviously, when applied to multidate imagery, the method has to cope with time-induced changes to the scenes. If these are masking the invariant features, correspondence will be less accurate. Any remaining misregistration reduces the capability to extract information at the highest resolutions, so it is a potential threat of the method (see the SWOT analysis). However, the estimation of productivity at the tree scale, which aggregates information at a larger scale, should not cause a significant impact.
- How to obtain vitality maps by RS data?
Parameters related to plant vitality include plant biomass, Leaf Area Index (LAI), biochemical parameter content (chlorophyll, anthocyanins), and photosynthetic efficiency (PRI index). First, all these parameters will be extracted from single date images via OBIA and standard classification algorithms. Subsequently, it will be investigated if additional multidate and multisensor data provide an added value, i.e., can generate more accurate vitality maps.
- How are the different vitality parameters related to the productivity of the orchard tree?
The vitality maps of the orchard will be related to the overall productivity, measured at the end of the growing season during harvest. The phenological stage of the trees at the time of image acquisition will be taken into account to determine which vitality parameters need to be acquired during which phenological stage for optimal productivity estimation. This will provide insights into which vitality parameter is influential for the productivity, and hence provide information on how to optimize the production process.
- Can the vitality maps be used to predict the productivity?
Based on the insight from the previous research question, the key vitality parameters influencing the productivity will be known. By measuring these parameters in the right phenological stage, it will be evaluated if these findings can be used to predict the final yield.
- How can we validate the algorithms?
The vitality maps (LAI, PRI, Chl, and biomass) will be validated by the vitality parameters measured by PcFruit, as well as the end of season yield maps (i.e. the prodcutivity). For this, additional data will be gathered at key stages during the growing season and ground reference data will be provided by PcFruit. Additionally, a commercial fruit grower expressed his interest to monitor his orchard encountering drought stress due to an impervious ground layer, which can yield additional validation data.
- How can we valorize our research?
Having PcFruit as stakeholder in the project ensures that the knowledge and experience obtained will be transferred to end users. This will be done by presenting results at workshops specifically targeting end users. The research results will also be brought to the attention of the international research community by peer reviewed publications and presentations at international conferences. A consecutive application project should allow the development of an operational framework for accurate orchard productivity mapping, serving the fruit growers.
By addressing above mentioned research questions, we aim at providing an accurate productivity map based on: (i) an improved UAV preprocessing chain, (ii) an object-based hierarchical multimodal image analysis technique, and (ii) a dedicated multitemporal classification method.
METHODOLOGY
An accurate monitoring of orchard productivity is essential to make the right decision on site specific orchard management and thus to maximize the profitability. Traditionally, this is performed with visual observations which are labor intensive and prone to human errors. Automating this process based on RS observations can lead to a less time consuming and more cost effective solution. An automated, objective collection of this information about the fruit tree productivity would therefore be an enormous added value for taking the right decisions e.g. for fruit thinning.
We hereby outline the way in which research will be undertaken to achieve the preset goals. Four main work packages are defined: data collection (WP2000), image preprocessing (WP3000), image processing (WP4000), and productivity mapping (WP5000). Two other work packages are specified to deal with project management (WP1000) and dissemination (WP6000).
Data collection (WP2000): At PcFruit, two experimental plots are designed to evaluate differences in tree productivity.
Besides the data collected during the Belair campaigns, additional information on the productivity of the fruit trees will be collected (e.g. yield data and hyperspectral RPAS images). Similar to the experimental set-up of 2013, two orchard blocks will be measured. Belair will provide RPAS, airborne hyperspectral and satellite RS data. Ground measurements that will be used from Belair include ASD spectroradiometer readings, chlorophyll data, physiological measurements, and pictures.
Additional ground measurements and RPAS flights will be planned in a commercial fruit orchard experiencing drought stress. As such, we want to validate our algorithms and make them useful to the end-user.
Image Preprocessing (WP3000): Image preprocessing is of utmost importance when using multi-date and multi-sensor data. An accurate radiometric and geometric correction is needed to allow fusion and an absolute interpretation of the images. Belair imagery is supposed to be delivered in preprocessed mode. However, an improved geometric and radiometric correction of RPAS images which allow fusion of different datasets still requires a large effort and additional, innovative research within this project. To anticipate potential pitfalls before an operational development, the impact of potential misregistration will be evaluated.
Radiometric accuracy will be established by imaging reference targets throughout the time series. During the first acquisitions, both real field measurements and measurements of the reference targets will be carried out. For subsequent acquisitions, it will be investigated if sufficient radiometric consistency can be established by using imaging of reference targets without additional field measurements.
Image processing (WP4000): In order to handle the ‘big data’, we will first concentrate on single image processing to derive information on tree vitality parameters. This single image processing will be performed with standard techniques of segmentation, vegetation index development and classification. Later on, the innovative multidate, multisensor processing will be added, investigated and evaluated.
WP4100: Single date RPAS image processing (VITO, INP, KUL)
In 2013, RGB images were collected over the orchards of interest. In 2015 and subsequent years, not only RGB, but also red edge and MultiSpec RPAS data will be collected at a spatial resolution of 5 cm. Additional hyperspectral RPAS imagery will be collected in the framework of this project.
This high spatial detailed data requires an object based analysis instead of the common pixel based analysis [14]. An object-based representation of the RPAS imagery will be given, in which the image is clustered into ‘objects’ or segments of connected pixels of similar context in an unsupervised manner. To find such a representation, a partitional clustering algorithm, which produces an exhaustive partitioning of the set of image pixels into clusters, will be applied. One of the most widely used partitional clustering algorithms is the k-means algorithm. In this algorithm, each observation is assigned to the cluster whose centroid (its mean) is closest according to some distance measure, such as the squared euclidean distance.
Specific, innovative developments are required in order to handle directly very high dimensional data such as hyperspectral images (each pixel is a vector with several hundreds of dimensions). In order to obtain a structured representation of the information contained in the image, the Binary Partition Tree (BPT) will be applied to represent an image as a set of hierarchical regions [15]. Subsequently, an application driven pruning step will accurately segment the image.
Vegetation indices, known to be closely correlated to the vitality parameters, i.e., chlorophyll, LAI and photosynthetic efficiency will be calculated on the tree objects extracted from the previous step. As such, tree vitality maps can be established. Also standard classification algorithms (e.g., tree based models, SVM) will be applied to group the segments in different vitality classes. The digital elevation models obtained by processing RPAS imagery, will allow for biomass estimation.
By doing this for all images throughout the season, an insight in the most optimal period for vitality detection could be obtained. We believe that adding more temporal information will improve the vitality map accuracy (WP 4300).
WP4200: Single date airborne image processing (KUL, INP, VITO)
APEX data are available within the Belair project. The drawback of this data in orchards is that due to the spatial resolution (i.e. 2m), both the trees and background are present within a single pixel. As seen in previous studies, mixing different elements in an image scene has a huge impact on the vitality prediction (eg Hypermix). Signal unmixing and VI correction methods will therefore be used to analyze these airborne data. By unmixing the tree signatures from the surrounding image elements, or correction of VI values for the background influence, a more realistic insight in the tree vitality will become possible.
Productivity mapping (WP5000)
WP5100: Multi date, multi sensor productivity modelling (INP, KUL, VITO)
Productivity information will be provided by PC fruit, encompassing the number of fruits, size, quality as well as weight. These productivity parameters will be linked to the vitality parameters obtained in WP4000. As the different single-date vitality maps were obtained throughout the growing season, not only will the main influencial vitality parameters be determined, but also the phenological stage at which it is measured. It has been shown, for example, that water stress during the fruitset period is less determining for total yield than during the fruit filling period. To identify the key vitality parameters and phenological stage, a multi-date, multi-sensor modelling approach must be applied.
Multimodal segmentation: The segmentation of a multimodal image should benefit from the complementarity of its modes to ensure a more accurate delineation of its regions, in particular when they share common attributes in one mode but not in another one. An innovative approach of energy minimization for segmenting hierarchies of segmentations issued from different modalities will be investigated. In particular, we will look into the recently introduced concept of braids of partitions [16, 17] and associated monitor hierarchies with an adapted dynamic program procedure to perform energy minimization over hierarchies.
Multitemporal classification: From all multidate images, spectral features will be extracted and used as classifiers in a multitemporal classification method. The Colibri ENVI-IDL code, written by Luc Bertels from VITO, will be applied and optimized for this purpose.
Vegetation index values calculated in WP 4100 will be recalculated for the objects extracted from multimodal segmentation. Temporal profiles of index values for vital and less vital trees will be compared. Conclusions will be drawn on how different vitality levels can be recognized based on these profiles, and used in the productivity modelling.
Regression models of vitality maps and ground reference and final yield data will disclose the relation between the vitality parameters and the productivity maps. Investigation into the optimal amount of information to create an accurate productivity map will be done. An answer will be given on ‘what is the added value of additional RS data’ in the creation of an orchard productivity map. What is the best time period to predict vitality or yield?
Algorithms will be evaluated on commercial orchard fields. Additional ground reference data and RPAS flight campaigns will be organized in a commercial fruit orchard under drought stress.
WP5200: Prediction of tree productivity based on single vs multidate multisensor image data (KUL, INP, VITO)
Based on the findings of WP5100, vitality maps obtained during the growing season can be used in a forward prediction mode to estimate the productivity, and thus provide predictions of the yield. WP5100 will determine which vitality parameter will need to be measured at which phenological stage, and through the obtained regression models the influence of the vitality parameter on the final yield can be determined. Sub-optimal vitality values will thus be translated to losses in yield, enabling the farmers to optimize the production process.
This yield prediction will be evaluated on the 2013 and 2015 BELAIR data, and in a second stage be validated on the data to be acquired in the commercial orchard. Predictions during the growing will then be compared to the yield obtained at the end of the growing season.
