1st Call for Funding: Dr Wenhao Zhang – Centre for Machine Vision, Bristol Robotics Laboratory, UWE, Bristol.

An intelligent, low-cost adaptive 3D multi-scale imaging system for advanced plant phenotyping (PS-Plant+)

Total Fund Requested – £24,881

Project Summary

Overview: We will develop a multi-scale 3D imaging and modelling system to identify and characterise key growth developmental stages in the model plant Arabidopsis and the important crop oilseed rape. The system will combine two complementary stereo imaging techniques to 1) record plant growth at a medium resolution that will be used to train a neural network to identify broad growth traits (e.g. rosette area and leaf number) and 2) capture high resolution images for targeted analyses of traits of interest (e.g. the surface and/or movement of an individual leaf, development of a new bud). To generate sufficient data for robust neural network training and analysis, two systems will be installed at the University of Edinburgh and on an automated platform at the National Plants Phenomics Centre (NPPC, Aberystwyth University), respectively. We aim to capture phenotypic data from a sufficient number of natural accessions of Arabidopsis (>100) and varieties of oilseed rape (about 120) to enable Genome Wide Association studies of the extracted trait variation.

Deliverables: Two proof-of-concept working systems; plant model databases for a model plant and important crop species; a journal paper; a follow-on bid.

The project team: This project offers an interdisciplinary collaboration opportunity between machine vision engineers at the Centre for Machine Vision in the Bristol Robotics Laboratory and plant and crop scientists at both the Edinburgh Predictive Plant Centre, Edinburgh University, and the NPPC, Aberystwyth University.

The need for whole plant 3D imaging and feature extraction: High throughput phenotyping platforms, such as the PSI system at the National Plants Phenomics Centre (NPPC), Aberystwyth University, involve the quantitative study of complex plant traits related to growth, yield and adaptation to biotic and abiotic stress. Leaf-level features of interest may include, leaf count, shape, area, angle, curvature and stem height, along with associated temporal aspects, such as leaf expansion rate and movement. Whole plant canopy features may include biomass, asymmetry and compactness. Together these features can be used to establish key developmental stages (e.g. the BBCH scale – a universal system used to delineate stages of plant growth), which in turn can be used to improve crop yield predictions. For example, the key points used to evaluate performance for oilseed rape(Brassica napus) are: first leaf emergence (stage 10) and unfolding (stage 11), second leaf unfolding (stage 12), eighth leaf unfolding (stage 18), and the rosette diameter at 20% of the mature rosette (stage 32). How plant development through these stages is impacted by environmental stress and climate change is of significant agronomic interest. A better understanding could help to improve predictive models of crop performance and inform future breeding strategies. Here we will build on an established image system to explore a new approach for automating the capture of features and developmental stages that have an important biological meaning in both simple and more complex plant canopies.

The challenges with whole plant 3D imaging: The resolution of 3D whole plant models for non-invasive phenotyping analyses is still low in terms of both image quality (spatial resolution) and imaging over time (temporal resolution) (Gibbs et al., 2018; Wu et al., 2018). Time series growth analysis at a resolution necessary to detect small features, such as emerging buds, needs appropriate hardware and generates huge quantities of data, requiring high-cost data storage and processing capabilities – this remains an unmet challenge in most species. Unlike the model plant Arabidopsis, which forms a relatively simple rosette during maturation, the majority of plants develop a complex 3D canopy. To tackle these problems, we propose to borrow from an approach already established in medical imaging that achieves an enhanced descriptive and discriminative ability via multi-scale feature representation (Tanga et al., 2017). Here we will explore a similar hybrid multi-scale methodology but add a machine learning component to generate compact (i.e. in terms of data size) integrated multiscale plant models, in which the scale and resolution can be adapted to target sub-regions of interest in high resolution. In this manner, traits of interest can be observed and characterised at different scales, for example, from detailed leaf properties (e.g. texture or morphology), to lower resolution canopy features (e.g. asymmetry or compactness) to represent more global, large-scale plant features.

Figure 1. Image of a whole Arabidopsis thaliana plant (a), close up of a soybean (Glycine max) leaf with visible veins and bud (b), and a detailed 3D reconstruction of a mint leaf (Mentha spicata) surface (using photometric stereo).

The benefits of a multiscale approach: Plants are comprised of complex 3D structures at multiple scales (Camargo et al., 2014; Gibbs et al., 2018; Lou et al., 2014). Thus, an inherent attribute of plant images is the presence of multiscale structure in both space and time (Tabb et al., 1997; Godin et al., 1998). As an example of space, Figure 1a depicts an image of a single of Arabidopsis plant. Here the individual leaves and petioles form identifiable regions at a scale that allows for significant intensity variation. However, if the sensitivity to spatial intensity variation is increased, then individual ribs and veins (and netted veins) may also be identified as sub-regions, together with stipules and buds (Figure 1b). Similarly, the dynamics of plant growth may also be perceived at differing scales of time resolution. For example, on a large scale, fixed whole plants appear, while on a smaller scale we can resolve important growth stage related parameters, that each can be affected by the growth environment in different ways, such as relative leaf growth rate, emergence of features (buds), changes in morphology (e.g. leaf / petiole curvature), and aspects of tropisms. Therefore, the identification of plant structure is inherently a spatial and temporal problem, where image structure is recursive, and where regions may contain substructures, which themselves contain substructure (a concept described by fractals).

To date, most plant imaging and modelling techniques have largely ignored the issue of scale and so can identify only a limited variety of structures. Techniques that attempt to deploy a single model over a wide range of scales tend to be expensive and impractical and may result in huge quantities of largely irrelevant data that is challenging to process for useful information (Rousseau et al., 2015). Thus, a need exists for an approach able to adaptively adjust spatial and temporal scales to simplify the identification and targeting of traits of interest. For example, to capture the whole plant canopy, a low-resolution large-scale format might be deployed, while for regions of greater interest, such as individual leaves, a regional, higher high-resolution small spatial scale data capture and modelling approach lends itself.  Because scale selection and structure detection are closely linked, we use the concept of scale (and not the image per se) to represent image structures at different resolutions.

Building on our established PSPlant imaging system (Bernotas et al., 2019), we will develop a multiscale analysis platform (PSPlant+) that will combine the use of machine learning (to help identify plant regions of interest) with two complementary 3D imaging techniques – stereo triangulation (ST) at a large scale, and photometric stereo (PS) at a small scale. This will enable identification, segmentation and modelling of features at significantly different scales (i.e. ranging from meters to micrometres), for which the processes of structure identification and adaptive scale selection will be integrated and, using machine learning, performed automatically.

Implementing a targeted multi-scale approach: Most practical imaging techniques to date have limited spatial resolution and only target measurements at a single scale, such as individual plant organs (e.g. leaf or stem), a single whole plant or perhaps a group of plants (Spalding et al 2013). However, image analysis is driving a renaissance in growth analysis and our prior work to develop the PS-Plant imaging system has demonstrated the utility of a low-cost photometric stereo (PS) technique in capturing local plant organ features at very high (sub-pixel level) 3D resolution (Figure 1c) (Zhang et al., 2018; Bernotas et al., 2019). PS-Plant is inherently low-cost as it uses relatively simple structured LED illuminations and a single camera to capture fine plant organ 3D morphology and albedo (true colour) data. However, PS has a limited capacity to recover low spatial frequency or whole plant 3D geometry, due to surface discontinuities and PS modelling errors. In contrast to PS, conventional stereo triangulation (ST) (i.e. the addition of a second camera) can potentially recover low spatial frequency plant data across the whole visible plant. This range of scale is important in monitoring principal BBCH growth stages. For example, sprouting, bud development, tillering and leaf development initiate at a sub-mm to mm scale, while stem elongation, leaf unfolding, and rosette growth tend to occur at a larger scale of mm to cm (or more).

We therefore propose to develop a hybrid PS-ST imaging technique (PS-Plant+), where ST provides gross whole plant geometry and PS captures local fine plant features (i.e. organ details) – all within a single fully integrated imaging framework. A common problem with stereo imaging is the correspondence problem (i.e. the matching of the same features between ST images). We aim to solve this issue by exploiting the combination of PS and ST to identify matching points between ST images across featureless regions (e.g. within leaves – featureless to ST but not PS) using photometric ratios in PS (i.e. the ratio of light reflected under differing PS illuminates). In addition, the gross positional data provided by ST will allow local calibration of illuminate positions for PS across whole plant geometry (to account for change in relative illuminate positions at different plant locations), which will help to improve PS accuracy over a larger field of view. Together, the synergistic combination of PS and ST will allow exploitation of the advantages of each imaging modality, while overcoming their respective disadvantages – that is, PS is good for high spatial frequency (e.g. small plant features) but poor at low spatial frequency (e.g. whole plant geometry), while ST offers the opposite.

Adding interactive, adaptive resolution: By capturing whole plant geometry as a ST 3D point cloud with local features recovered using PS, users would be able to interact with the plant model at different scales. For example, a location can be specified at low resolution in 3D space and the local 3D morphology could then be recovered at much high resolution – effectively zooming in on subtle details only where needed. We have already developed and tested an ST-PS prototype based on a RealSense (RGB+D) camera, and have achieved promising early results with Arabidopsis (Figure 2). Our first goal will be to further develop our system to improve resolutions in Arabidopsis. We will then switch to higher-resolution PointGrey cameras and a larger, more complex crop plant – oilseed rape. Work on the latter important crop is linked with the proposed follow-on bid – see Expected Outcomes section below for full justification.

Figure 2. Combining PS and ST data in a single 3D plant model. Higher-resolution PS was acquired for leaf texture, which we were able to superimpose on the lower-resolution ST whole plant model.

Deep learning to intelligently locate and describe features of interest: The previous deep learning approach we developed demonstrated a high performance on plant leaf segmentation and tracking on PS data (Bernotas et al., 2019). To further develop the deep learning approach, a convolutional neural network (CNN) will be trained to recognise and track plant leaves (and other organs) in large-scale low-resolution ST images. These local features will subsequently be rendered as sub-regions using PS data to accurately extract their shape, area, angle, and topography/curvature data. Hence, in this manner, the CNN will find local features at low-resolution, which PS will then reconstruct, render and quantify in high resolution – achieving a plant model where scale and resolution adapt as needed to efficiently (in terms of model data storage space) describe important features of interest.

Integration into existing phenotyping platforms:  We propose to create two identical imaging rigs and install one in Edinburgh, to initially image Arabidopsis, and a second in the small plant platform at the NPPC in Aberystwyth to image Arabidopsis and oilseed rape.  The installation at NPPC is able to benefit from plants installed as part of their existing BBSRC LoLa project ‘BRAVO’ (grant number BB/P003095/1). This robotic platform will allow us to address the big data aspect as it can capture images from up to 2000 Arabidopsis plants per day, or 500 oilseed rape plants, providing a test-bed for high throughput applications as might be required for genetic mapping experiments.  For both Arabidopsis and oilseed rape, the aim will be to train the CNN to automatically identify key growth developmental stages based on a combination of low- and high-resolution features and to obtain quantitative measurements of organs as a function of time. For example, in the case of oilseed rape: first leaf emergence (stage 10) and unfolding (stage 11), second leaf unfolding (stage 12), eighth leaf unfolding (stage 18), and the rosette diameter at 20% of the mature rosette (stage 32). Performance will be validated against manually labelling. Evaluating the system and associated feature extraction pipelines in 2 different locations will test its robustness and general applicability to large scale biology. This proof of concept will form the basis for a publication and provide preliminary work for a follow-on bid.