1st Call for Funding: Dr Ji Zhou – NIAB

Developing 2D-3D fusion to enable high-throughput phenotypic analysis of key yield-related traits of bread wheat using cost-effective UAV imagery.

Total Fund Requested: £24,940

2. Project Description 2.1 Project aim and objectives In the proposed project, we aim to develop a standardised imaging protocol to acquire high-quality and high-frequency aerial images of wheat crops using low-cost UAVs, based on which a proof-of-concept analytic platform will be developed to enable the automation of plot segmentation as well as plot-level analysis of key yield-related traits. We plan to focus on three key yield-related traits, including (1) plot height (improving from existing methods published before10), (2) 3D canopy structure (including texture compactness for canopy closure and canopy orientation for accessing lodging), and (3) 3D leaf area for photosynthetic effects at the canopy level. We plan to develop the analysis algorithms to process multiple aerial image series based on 2D-3D fusion mapping, which combines 2D orthomosaic images of hundreds of wheat plots with 3D point cloud data reconstructed using cloud-based platform such as Pix4D Cloud. We will use the wheat lines selected by the BBSRC’s Designing Future Wheat (DFW) programme, at field trail centres in both JIC and Rothamsted Research. A number of key objectives are listed below: • To establish a standard protocol for using low-cost UAVs to conduct organ-level, plot-level, and field-level aerial imagery, so that acquired images can be used for both precise 2D orthomosaic and high-density 3D point cloud reconstruction.
   • To build a 2D-3D fusion algorithm to reliably merge pixels (from 2D orthomosaic) and voxels (from 3D point cloud) in a high-throughput manner, so that 3D height information can be assigned to each 2D pixel in the 2D-3D mapping for 3D phenotypic analysis.
   • To establish a learning model (e.g. convolutional neural networks, CNNs or You Only Look Twice, YOLT) to enable large-scale plot segmentation (e.g. on thousands of wheat plots) from the 2D-3D fusion mapping.
   • To develop phenotypic analysis algorithms to measure three key yield-related traits at the canopy level, i.e. plot height, 3D canopy structure, and 3D leaf area index.
   • To assess and link analysis outcomes with manual measurement as well as methods previously published, which are being conducted during the DFW programme in both WP1 and WP3.

2.2 Aerial imagery and pre-processing This project will make extensive use of aerial imagery data collected from DFW wheat lines at both JIC and Rothamsted Research field trail stations. The 2018-2019 aerial image series has already been collected by the Zhou laboratory, which will be used as a foundation of this project. To collect wheat organ, plot, and field level aerial imagery, we will test a range of altitudes and flying modes using flying control software such as DJI GO and Pix4D Capture, with an autonomous GPS-guided flight path over the DFW regions, taking photos at defined intervals. The acquired images (the front and back overlap is at least 80% and a side overlap is 70% minimum) and the associated GPS location data from the drone will be processed using cloud-based photogrammetry software (e.g. Pix4D Cloud) to find common key points in the imagery to produce 2D orthomosaic and reconstruct 3D point cloud models of the whole field.

Figure 1 The workflow of aerial imagery collection and data pre-processing Figure 1 shows the initial workflow, from aerial imaging to data pre-processing. We will test data collection and optimal altitudes, photo overlap, and oblique angles during the project, so that clear and high-resolution datasets can be produced in a standard manner for the following automated phenotypic analysis. The target resolution is 0.25-0.5 cm/pixel for organ level, 0.5-1 cm/pixel for plot level, and 2-5 cm/pixel for field level analysis, based on which 2D and 3D imagery will be produced to represent hundreds of DFW wheat plots (e.g. 720 six-meter plots for different drilling day experiments, i.e. early, normal and late drilling dates) during the 2019-2020 growing season. When collecting the aerial images, we plan to cover five key growth stages, including early establishment, tillering, heading, flowering, and the grain filling stage.

2.3 2D-3D fusion and 3D point cloud analysis 2D-3D data fusion calibration and point cloud analysis of the aerial imagery datasets will be carried out by the Zhou laboratory at EI. Based on the constructed high-density 3D point cloud data at different key growth stages (e.g. heading and flowering in Fig. 2A), we plan to firstly assess voxels (3D points) using their original Z value positions and then register them to the computed fitted positions to remove noise

voxels using point density and elevation filters provided by VTK (Visualization Toolkit) library (Fig. 2B). Then, we will cast remaining 3D points to a “hyperplane” by estimating the average 3D normal of the hyperplane, i.e. mean values of ordinary least squares, displaying the height elevation in a pseudo heat map (the first attempt can be seen in Fig. 2C). This step will assign height information of individual plots and their associated canopies. Figure 2C shows an example of our current approach that calibrates wheat plot point clouds in order to flatten the slope of field, where blue colour represents low z values (i.e. the plot height) and red colour for high z values.

Figure 2 Point Clouds visualised using Python and VTK libraries. Applying VTK elevation and point density filters to the point cloud to flat the field and then presents the plot height in a ‘heat map’.

2.4 3D Traits Analysis After the hyperplane fitting and colour-code height mapping for hundreds of wheat plots in the field, the variation in wheat heights and canopy structures can be visualised in the 2D-3D fusion heat map. To carry out 3D phenotypic analysis of traits such as plot height, 3D canopy structure, and 3D leaf area, we will build upon our expertise and experience with large-scale crop phenotyping projects. We will establish a proof-of-concept analytic platform to process image series in a high-throughput fashion. To accomplish this objective, we plan to utilise 3D-2D fusion mapping and feed them into the ongoing deep learning research derived from the DFW programme. First, we plan to soundly segment thousands of plots in the field using deep learning (e.g. CNNs or YOLT); then, we will conduct computer-vision based 3D traits measurement at the plot level, so that 2D/3D phenotypic information can be quantified for associated plots; finally, we will employ parallel computing and profiling to accelerate the analysis to realise the high-throughput requirement. Figure 3 shows the architecture of a CNN network we plan to use for segmenting thousands of plots in the field, a “shallow” learning architecture that conducts a binary classification of soil and non-soil. To trigger high-throughput plot segmentation, we will train the learning model with colour, texture and height information from the 2D-3D fusion map (Fig. 3A). To enable a speedy classification, we will avoid larger and deeper neural network architectures that often require more time to train and hence are slower to execute (Fig. 3B). In the proposed project, we plan to identify soil pixels in a broad range of field

conditions and in a timely manner, which will allow us to recognise plots by inverting the soil signals (coloured red in Fig. 3C).

Figure 3 An example of a CNN learning architecture established for segmenting soil and non-soil pixels for identifying wheat plots. After segmenting individual plots, 3D trait analyses can be conducted by using computer-vision based methods published by the Zhou laboratory11–13, as well as some latest solutions14,15: • 2D morphological features, including patterns, can be calculated based on green pixel regions (e.g. excess greenness) contained by the canopy;
• Canopy stockiness and compactness can be calculated by the ratio between the projected canopy area and the canopy perimeter.
• The height channel, compensating by the green colour channel, can be used to calculate Shannon entropy, Anisotropy, and several GLCM features to represent 3D canopy structure. The algorithms developed in this project will be used to help crop researchers and breeders with 3D yield-related trait analysis that enables better crop selection and improved understanding of dyanmic phenotypic changes. It could also be used to tackle the scalability challenge that crop researchers and breeders are facing due to multi-sites and multi-time phenotyping needs. The proof-of-concept analytic platform developed in the proposed project will be hosted by the CropSight system16, openly available for the community (https://github.com/Crop-Phenomics-Group/CropSight).