This activity will develop detection networks locate perimeters of objects in images, density networks trained to generate a numerical value, and semantic segmentation networks to assign class labels to image pixels so that contiguous pixels with the same label form segmented regions. By applying these automated processes to plants, computers can replace humans in tedious manual tasks such as observing the emergence of seedlings, counting the number of flowers in a drone image, and separating vegetation from features such as bare soil in rectangular and non-rectangular plots.

Digital phenotypes may be directly related to yield traits but information can also be extracted from abstract phenotypes, whose relationship with yield is not initially clear. This activity is developing unsupervised learning methods to extract abstract phenotypes for use in breeding programs. We are also using semi-supervised learning to identify signatures that predict performance across multiple environments during early-generation yield trials, leading to better predictions of GxE effects without test replications. We are working on enhanced early-generation yield trial designs that augment statistical analysis with digital information to better account for spatial gradients.

End-of-season phenotypes are important targets for plant breeding but information from earlier in the season is also valuable. We are developing state-of-the-art recurrent neural network architectures to analyze images of plots throughout the growing season to better predict end-of-season phenotypes. We are using latent-space phenotyping to automatically detect and quantify responses to treatment directly from images of plants, and we are using high-temporal-resolution images of crops for the extraction of direct phenotypes related to growth and abstract phenotypes related to other traits.

To uncover prospective phenotypes that may be useful for selection, we are developing algorithms that incorporate knowledge of how roots grow for the segmentation of 3D root images, automated systems for the reconstruction of 3D image sequences from gradually rotated samples, and architectural models of segmented root systems. We are using generative adversarial networks to derive richer datasets from low-resolution images, and we are investigating deep learning methods that combine imaging data and differences in the spatial arrangement of shoots and roots to predict the maturity and shatter rate of canola pods.

Visualizations help us to understand deep neural networks because they focus on the same regions as human annotators, but existing generic visualization tools do not directly support specialized analysis. We are developing visual analytics tools for deep neural networks, including a testing and evaluation framework for deep learning models to make it easier for non-computer scientists to train and deploy these phenotyping tools. We have taken on personnel to build/adapt software for image annotation and establish ground-truth measurements with images to evaluate annotations for reliability.