# US 12,119,086 B2 · Method and system for performing data analysis for plant phenotyping

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,119,086 B2
- Document ID: US12119086B2
- Status: Granted
- Kind: B2
- Application number: 16/753,044
- Filed: 2018-10-04
- Published: 2020-09-17
- Granted: 2024-10-15
- Earliest claimed: 2017-10-04
- Assignees: KWS SAAT SE & Co. KGaA
- Applicants: KWS SAAT SE & CO. KGaA
- Inventors: Christoph Bauer; Christian Jebsen; Sabine Gubatz; Ludmilla Dahl
- Web page: https://patentgrep.com/patent/US12119086B2

## CPC

- A01B 79/005: {Precision agriculture}
- G16B 20/20: Allele or variant detection, e.g. single nucleotide polymorphism [SNP] detection
- C12Q 1/6895: for plants, fungi or algae
- G01N 21/31: Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry {(G01N21/72 takes precedence)}
- G01N 33/0098: {Plants or trees (wood G01N33/46)}
- G06V 20/17: taken from planes or by drones
- G06V 20/188: {Vegetation}
- A01G 7/00: Botany in general
- C12Q 2600/13: Plant traits
- C12Q 2600/158: Expression markers
- G01N 21/6456: {Spatial resolved fluorescence measurements; Imaging}
- G01N 33/245: {for agricultural purposes}
- G06V 20/194: {using hyperspectral data, i.e. more or other wavelengths than RGB}
- G06V 2201/10: Recognition assisted with metadata
- A01B: SOIL WORKING IN AGRICULTURE OR FORESTRY; PARTS, DETAILS, OR ACCESSORIES OF AGRICULTURAL MACHINES OR IMPLEMENTS, IN GENERAL (making or covering furrows or holes for sowing, planting or manuring A01C5/00; mowers convertible to soil working apparatus or capable of soil working A01D42/04; mowers combined with soil working implements A01D43/12; soil working for engineering purposes E01, E02, E21)
- G16B: BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY
- C12Q: MEASURING OR TESTING PROCESSES INVOLVING ENZYMES, NUCLEIC ACIDS OR MICROORGANISMS (immunoassay G01N33/53); COMPOSITIONS OR TEST PAPERS THEREFOR; PROCESSES OF PREPARING SUCH COMPOSITIONS; CONDITION-RESPONSIVE CONTROL IN MICROBIOLOGICAL OR ENZYMOLOGICAL PROCESSES
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A01G: HORTICULTURE; CULTIVATION OF VEGETABLES, FLOWERS, RICE, FRUIT, VINES, HOPS OR SEAWEED; FORESTRY; WATERING (picking of fruits, vegetables, hops or the like A01D46/00; propagating unicellular algae C12N1/12)

## Abstract

The invention relates to a method for performing data analysis for plant phenotyping of single plants in a field and a data acquisition and evaluation system for performing data analysis for plant phenotyping of single plants in a field. Further, the invention relates to a mobile platform for use in the data acquisition and evaluation system. The method comprises the steps of capturing spectral data via a hyperspectral imaging sensor, capturing image data via an image sensor, capturing georeference data via an inertial measurement unit, spatializing the image data to generate georeferenced image data and a digital surface model, spatializing the spectral data, generating georeferenced spectral data based on the spatialized spectral data and the digital surface model and overlaying the georeferenced image data and georeferenced spectral data with field plan information to generate a high-resolution analysis data set.

## Claims

### Claim 1

A computer-aided method for performing data analysis for plant phenotyping of one or more single plants in each of a plurality of distributed fields, comprising the steps of:
simultaneously capturing spectral data associated with the one or more single plants in each of the plurality of distributed fields via a hyperspectral imaging sensor;
simultaneously capturing image data associated with the one or more single plants in each of the plurality of distributed fields via an image sensor, wherein the image sensor is a color sensor for selective detection and evaluation of a visible spectral range;
simultaneously capturing high-resolution georeference data associated with the one or more single plants in each of the plurality of distributed fields via an inertial measurement unit comprising acceleration sensors and rotational speed sensors, the high-resolution georeference data comprising georeferenced coordinates;
transmitting the captured spectral data, image data, and high-resolution georeference data to one or more remote servers;
spatializing, via the one or more remote servers, the image data to generate georeferenced image data and a high-resolution digital surface model, wherein spatializing the image data comprises assigning spatial coordinates to the image data and spatially correcting the image data such that spatial information is assigned to individual image pixels, wherein the high-resolution digital surface model includes height information comprising an orientation of growth of a plant, a direction of growth of the plant, a direction of growth of leaves of the plant, a height of the plant, or combinations thereof, wherein generating the high-resolution digital surface model comprises:
obtaining multiple recordings of an individual picture element by capturing the image data; and
combining the multiple recorded picture elements in a three-dimensional image;
generating, via the one or more remote servers, one or more visual indications of diseased leaves, healthy leaves, soil, or combinations thereof from the spectral data by comparing the spectral data to referenced spectral data from a spectral database;
spatializing, via the one or more remote servers, the spectral data;
generating, via the one or more remote servers, georeferenced spectral data from the spatialized spectral data using the high-resolution digital surface model;
overlaying, via the one or more remote servers, the one or more visual indications, the georeferenced image data, and the georeferenced spectral data with field plan information to generate a high-resolution analysis data set by using a merging algorithm, wherein the overlaying of the georeferenced image data and the georeferenced spectral data with the field plan information comprises an assignment of field piece information according to the georeferenced coordinates; and
localizing, via the one or more remote servers, the one or more single plants in a range of 10 cm around one or more real positions of the one or more single plants in each of the plurality of distributed fields.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the image sensor is an RGB sensor.

### Claim 3 (depends on 1)

The method according to claim 1, wherein spatializing the spectral data comprises:
a first step of spatializing the spectral data, which comprises;
assigning spatial coordinates to spectral data; and
radiometrically correcting the spectral data; and
a second step of spatializing the spectral data, which comprises spatially correcting the spectral data.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the field plan information comprises field information for defining field locations and field dimensions, in particular field piece information for defining field piece locations and field piece dimensions.

### Claim 5 (depends on 1)

The method according to claim 1 further comprises the step of capturing additional data via at least one additional sensor, via a thermal sensor and/or an electro-magnetic sensor.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the method further comprises the step of using a computer algorithm for phenotyping that identifies direct traits and/or leaf diseases and/or insect damages and/or virus infections by symptoms and/or abiotic stress effects.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the hyperspectral imaging sensor for capturing spectral data and the image sensor for capturing image data and the inertial measurement unit for capturing georeference data are arranged on a mobile platform, wherein the mobile platform is an aerial device configured to be positioned at an altitude 1-100 meters above a canopy of the one or more single plants in each of the plurality of distributed fields.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the method further comprises pre-processing and/or processing the spectral data on the mobile platform and/or an agricultural station and/or a main server during an operating process and/or in a separate step offline.

### Claim 9 (depends on 8)

The method according to claim 8, wherein captured data and/or pre-processed data and/or the processed spectral data are transferred from the mobile platform to the main server and/or from the agricultural station to the main server via a wire connection and/or a wireless connection.

### Claim 10 (depends on 1)

A mobile platform for use in the method according to claim 1, comprising:
the hyperspectral imaging sensor for capturing spectral data;
the image sensor for capturing image data; and
the inertial measurement unit for capturing georeference data.

### Claim 11

A data acquisition and evaluation system for performing data analysis for plant phenotyping of one or more single plants in each of a plurality of distributed fields, comprising:
a hyperspectral imaging sensor for simultaneously capturing spectral data associated with the one or more single plants in each of the plurality of distributed fields;
an image sensor for simultaneously capturing image data associated with the one or more single plants in each of the plurality of distributed fields, wherein the image sensor is a color sensor for selective detection and evaluation of a visible spectral range;
an inertial measurement unit comprising acceleration sensors and rotational speed sensors for simultaneously capturing high-resolution georeference data associated with the one or more single plants in each of the plurality of distributed fields, the high-resolution georeference data comprising georeferenced coordinates; and
a control unit, which is remote from the hyperspectral imaging sensor, the image sensor, and the inertial measurement unit, and adapted to:
spatialize the image data to generate georeferenced image data and a high-resolution digital surface model, wherein spatializing the image data comprises assigning spatial coordinates to the image data and spatially correcting the image data, whereby spatial information is assigned to individual image pixels, wherein the high-resolution digital surface model includes height information comprising an orientation of growth of a plant, a direction of growth of the plant, a direction of growth of leaves of the plant, a height of the plant, or combinations thereof, wherein generating the high-resolution digital surface model comprises:
obtaining recordings of multiple individual picture elements by capturing the image data;
combining the recorded multiple individual picture elements in a three-dimensional image;
generating one or more visual indications of diseased leaves, healthy leaves, soil, or combinations thereof from the spectral data by comparing the spectral data to referenced spectral data from a spectral database;
spatialize the spectral data;
generate georeferenced spectral data from the spatialized spectral data using the high-resolution digital surface model;
overlay the one or more visual indications, the georeferenced image data, and georeferenced spectral data with field plan information to generate a high-resolution analysis data set, wherein the overlaying of the georeferenced image data and the georeferenced spectral data with the field plan information comprises an assignment of field piece information according to the georeferenced coordinates; and
localize the one or more single plants in a range of 10 cm around one or more real positions of the one or more single plants in each of the plurality of distributed fields.

### Claim 12 (depends on 11)

A mobile platform for use in a data acquisition and evaluation system according claim 11, comprising:
the hyperspectral imaging sensor for capturing spectral data;
the image sensor for capturing image data; and
the inertial measurement unit for capturing georeference data,
wherein the mobile platform is an aerial device configured to be positioned at an altitude 1-100 meters above a canopy of the one or more single plants in each of the plurality of distributed fields.

### Claim 13

A method for selecting a plant, said method comprising:
a) growing a plant population;
b) simultaneously capturing spectral data associated with one or more single plants in each of a plurality of distributed fields via a hyperspectral imaging sensor;
c) simultaneously capturing image data associated with the one or more single plants in each of the plurality of distributed fields via an image sensor, wherein the image sensor is a color sensor for selective detection and evaluation of a visible spectral range;
d) simultaneously capturing high-resolution georeference data associated with the one or more single plants in each of the plurality of distributed fields via an inertial measurement unit comprising accelerate sensors and rotational speed sensors, the high-resolution georeference data comprising georeference coordinates;
e) transmitting the captured spectral data, image data, and high-resolution georeference data to one or more remote servers;
f) generating, via the one or more remote servers, one or more visual indications of diseased leaves, healthy leaves, soil, or combinations thereof from the spectral data by comparing the spectral data to referenced spectral data from a spectral database;
g) spatializing, via the one or more remote servers, the image data to generate georeferenced image data and a high-resolution digital surface model, wherein spatializing the image data comprises assigning spatial coordinates to the image data and spatially correcting the image data, wherein spatial information is assigned to individual image pixels, wherein the high-resolution digital surface model includes height information comprising an orientation of growth of a plant, a direction of growth of the plant, a direction of growth of the leaves of the plant, a height of the plant, or combinations thereof, wherein generating the high-resolution digital surface model comprises:
obtaining multiple recordings of an individual picture element by capturing the image data; and
combining the recorded multiple individual picture elements in a three-dimensional image;
h) spatializing, via the one or more remote servers, the spectral data;
i) generating, via the one or more remote servers, georeferenced spectral data from the spatialized spectral data using the high-resolution digital surface model;
j) overlaying, via the one or more remote servers, the one or more visual indications, the georeferenced image data, and the georeferenced spectral data with field plan information to generate a high-resolution analysis data set by using a merging algorithm, wherein the overlaying of the georeferenced image data and the georeferenced spectral data with the field plan information comprises an assignment of field piece information according to the georeference coordinates;
k) localizing, via the one or more remote servers, the one or more single plants in a range of 10 cm around one or more real positions of the one or more single plants in each of the plurality of distributed fields; and
l) selecting a plant from the plant population having a desired phenotype.

### Claim 14

A method for selecting plant individuals in a breeding program, said method comprising:
a) growing a plant population of training individuals;
b) simultaneously capturing spectral data associated with one or more single plants in each of a plurality of distributed fields via a hyperspectral imaging sensor;
c) simultaneously capturing image data associated with the one or more single plants in each of the plurality of distributed fields via an image sensor, wherein the image sensor is a color sensor for selective detection and evaluation of a visible spectral range;
d) simultaneously capturing high-resolution georeference data associated with the one or more single plants in each of the plurality of distributed fields via an inertial measurement unit comprising accelerate sensors and rotational speed sensors, the high-resolution georeference data comprising georeference coordinates;
e) transmitting the captured spectral data, image data, and high-resolution georeference data to one or more remote servers;
f) spatializing, via the one or more remote servers, the image data to generate georeferenced image data and a high-resolution digital surface model, wherein spatializing the image data comprises assigning spatial coordinates to the image data and spatially correcting the image data, wherein spatial information is assigned to individual image pixels, wherein the high-resolution digital surface model includes height information comprising an orientation of growth of a plant, a direction of growth of the plant, a direction of growth of the leaves of the plant, a height of the plant, or combinations thereof, wherein generating the high-resolution digital surface model comprises:
obtaining multiple recordings of an individual picture element by capturing the image data; and
combining the recorded multiple individual picture elements in a three-dimensional image;
g) generating, via the one or more remote servers, one or more visual indications of diseased leaves, healthy leaves, soil, or combinations thereof from the spectral data by comparing the spectral data to referenced spectral data from a spectral database;
h) spatializing, via the one or more remote servers, the spectral data;
i) generating, via the one or more remote servers, georeferenced spectral data from the spatialized spectral data using the high-resolution digital surface model;
j) overlaying, via the one or more remote servers, the one or more visual indications, the georeferenced image data, and the georeferenced spectral data with field plan information to generate a high-resolution analysis data set by using a merging algorithm, wherein the overlaying of the georeferenced image data and the georeferenced spectral data with the field plan information comprises an assignment of field piece information according to the georeference coordinates;
k) localizing, via the one or more remote servers, the one or more single plants in a range of 10 cm around one or more real positions of the one or more single plants in each of the plurality of distributed fields; and
l) generating a phenotype training data set;
m) associating the phenotype training data set with a genotype training data set comprising genetic information across a genome of each training individual;
n) genotyping a population of breeding individuals; and
o) selecting breeding pairs from the population of breeding individuals based on plant genotypes using the associated training data set to select breeding pairs likely or able to generate offspring with one or more desired traits.

### Claim 15 (depends on 14)

The method of claim 14, wherein said genetic information for a candidate is obtained by genotyping using SNP markers.

### Claim 16 (depends on 14)

The method of claim 14, wherein said genetic information for a candidate is obtained by analyses of gene expression, metabolite concentration, or protein concentration.

### Claim 17 (depends on 14)

The method of claim 14, further comprising:
crossing the breeding pairs to generate offspring; and
growing the offspring with the one or more desired traits,
wherein said breeding individuals are homozygous.

### Claim 18 (depends on 14)

The method of claim 14, further comprising a genetically diverse population that includes individuals carrying one or more transgenes or a genetically diverse population that includes individuals with DNA edited with random or targeted mutagenesis.

### Claim 19 (depends on 14)

The method of claim 14, wherein said plant population of training individuals and/or the population of breeding individuals is genetically diverse.

### Claim 20

A method for selecting an inbred plant, the method comprising:
a) quantitatively assessing a distribution of two or more traits in a population of inbred plants, wherein assessing the distribution of at least one trait is performed on bases of a high-resolution analysis data set generated by:
simultaneously capturing spectral data associated with one or more single plants in each of a plurality of distributed fields via a hyperspectral imaging sensor;
simultaneously capturing image data associated with the one or more single plants in each of the plurality of distributed fields via an image sensor, wherein the image sensor is a color sensor for selective detection and evaluation of a visible spectral range;
simultaneously capturing high-resolution georeference data associated with the one or more single plants in each of the plurality of distributed fields via an inertial measurement unit comprising acceleration sensors and rotational speed sensors, the high-resolution georeference data comprising georeferenced coordinates;
transmitting the captured spectral data, image data, and high-resolution georeference data to one or more remote servers;
spatializing, via the one or more remote servers, the image data to generate georeferenced image data and a high-resolution digital surface model, wherein spatializing the image data comprises assigning spatial coordinates to the image data and spatially correcting the image data such that spatial information is assigned to individual image pixels, wherein the high-resolution digital surface model includes height information comprising an orientation of growth of a plant, a direction of growth of the plant, a direction of growth of the leaves of the plant, a height of the plant, or combinations thereof, wherein generating the high-resolution digital surface model comprises:
obtaining multiple recordings of an individual picture element by capturing the image data; and
combining the multiple recorded picture elements in a three-dimensional image;
generating, via the one or more remote servers, one or more visual indications of diseased leaves, healthy leaves, soil, or combinations thereof from the spectral data by comparing the spectral data to referenced spectra data from a spectral database;
spatializing, via the one or more remote servers, the spectral data;
generating, via the one or more remote servers, georeferenced spectral data from the spatialized spectral data using the high-resolution digital surface model; and
overlaying, via the one or more remote servers, the one or more visual indications, the georeferenced image data, and the georeferenced spectral data with field plan information to generate the high-resolution analysis data set by using a merging algorithm, wherein the overlaying of the georeferenced image data and the georeferenced spectral data with the field plan information comprises an assignment of field piece information according to the georeferenced coordinates;
localizing, via the one or more remote servers, the one or more single plants in a range of 10 cm around one or more real positions of the one or more single plants in each of the plurality of distributed fields;
b) constructing a relationship matrix for each inbred plant parent for the two or more traits of interest;
c) applying the relationship matrix in a multivariate mixed model analysis for the population of inbred plants;
d) obtaining a predicted value for said inbred plant; and
e) selecting one or more inbred plants based on the predicted value.

### Claim 21 (depends on 20)

The method of claim 20, wherein the population of inbred plants is separated into male and female lines.

### Claim 22 (depends on 20)

The method of claim 20, wherein the traits comprise a plurality of correlated attributes.

### Claim 23 (depends on 22)

The method of claim 22, wherein the plurality of correlated attributes comprises grain yield, moisture content, total leaf number and/or biomass.

### Claim 24 (depends on 20)

The method of claim 20, further comprising determining the general combining ability and/or a specific combining ability for said plant.

### Claim 25 (depends on 20)

The method of claim 20, further comprising calculating a BLUP using the high-resolution digital surface model.

### Claim 26 (depends on 20)

The method of claim 20, further comprising calculating an accuracy of prediction for each said predicted value.

### Claim 27 (depends on 20)

The method of claim 20, further comprising selecting a hybrid progeny plant based on predicted values obtained from two parent inbred plants.