# US 2022/0036070 A1 · Inferring moisture from color

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036070 A1
- Document ID: US20220036070A1
- Status: Published application
- Kind: A1
- Application number: 16/943,247
- Filed: 2020-07-30
- Published: 2022-02-03
- Earliest claimed: 2020-07-30
- Assignees: X Development LLC
- Applicants: X Development LLC
- Inventors: Bodi Yuan; Zhiqiang Yuan; Ming Zheng
- Web page: https://patentgrep.com/patent/US20220036070A1

## CPC

- G06V 10/82: using neural networks
- G06V 20/188: {Vegetation}
- A01B 79/005: {Precision agriculture}
- A01C 1/025: {Testing seeds for determining their viability or germination capacity}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06T 7/60: Analysis of geometric attributes
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/90: Determination of colour characteristics
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06V 10/56: relating to colour
- A01D 75/00: Accessories for harvesters or mowers
- A01G 25/16: Control of watering
- A01M 21/00: Apparatus for the destruction of unwanted vegetation, e.g. weeds (control of undesirable vegetation on roads or permanent ways of railways E01H11/00)
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30188: Vegetation; Agriculture
- G06T 2207/30242: Counting objects in image
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A01B: SOIL WORKING IN AGRICULTURE OR FORESTRY; PARTS, DETAILS, OR ACCESSORIES OF AGRICULTURAL MACHINES OR IMPLEMENTS, IN GENERAL (making or covering furrows or holes for sowing, planting or manuring A01C5/00; mowers convertible to soil working apparatus or capable of soil working A01D42/04; mowers combined with soil working implements A01D43/12; soil working for engineering purposes E01, E02, E21)
- A01C: PLANTING; SOWING; FERTILISING (parts, details or accessories of agricultural machines or implements, in general A01B51/00 - A01B75/00)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A01D: HARVESTING; MOWING
- A01G: HORTICULTURE; CULTIVATION OF VEGETABLES, FLOWERS, RICE, FRUIT, VINES, HOPS OR SEAWEED; FORESTRY; WATERING (picking of fruits, vegetables, hops or the like A01D46/00; propagating unicellular algae C12N1/12)
- A01M: CATCHING, TRAPPING OR SCARING OF ANIMALS (appliances for catching swarms or drone-catching A01K57/00; fishing A01K69/00 - {A01K99/00}; biocides, pest repellants or attractants A01N); APPARATUS FOR THE DESTRUCTION OF NOXIOUS ANIMALS OR NOXIOUS PLANTS

## Abstract

Techniques are described herein for using artificial intelligence to predict crop yields based on observational crop data. A method includes: obtaining a first digital image of at least one plant; segmenting the first digital image of the at least one plant to identify at least one seedpod in the first digital image; for each of the at least one seedpod in the first digital image: determining a color of the seedpod; determining a number of seeds in the seedpod; inferring, using one or more machine learning models, a moisture content of the seedpod based on the color of the seedpod; and estimating, based on the moisture content of the seedpod and the number of seeds in the seedpod, a weight of the seedpod; and predicting a crop yield based on the moisture content and the weight of each of the at least one seedpod.

## Claims

### Claim 1

A method implemented by one or more processors, the method comprising:
obtaining a first digital image of at least one plant;
segmenting the first digital image of the at least one plant to identify at least one seedpod in the first digital image;
for each of the at least one seedpod in the first digital image:
determining a color of the seedpod;
determining a number of seeds in the seedpod;
inferring, using one or more machine learning models, a moisture content of the seedpod based on the color of the seedpod; and
estimating, based on the moisture content of the seedpod and the number of seeds in the seedpod, a weight of the seedpod; and
predicting a crop yield based on the moisture content and the weight of each of the at least one seedpod.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising determining a size of each of the at least one seedpod in the first digital image,
wherein, for each of the at least one seedpod in the first digital image, the estimating the weight of the seedpod is further based on the size of the seedpod.

### Claim 3 (depends on 1)

The method according to claim 1, further comprising determining a number of seedpods on each of the at least one plant in the first digital image,
wherein the predicting the crop yield is further based on the number of seedpods on each of the at least one plant in the first digital image.

### Claim 4 (depends on 1)

The method according to claim 1, wherein:
the weight that is estimated based on the moisture content is a wet weight; and
the predicting the crop yield comprises predicting a dry weight based on the wet weight of each of the at least one seedpod and the moisture content of each of the at least one seedpod.

### Claim 5 (depends on 1)

The method according to claim 1, wherein one or more of the machine learning models is a convolutional neural network model.

### Claim 6 (depends on 1)

The method according to claim 1, wherein:
the first digital image is one of a plurality of digital images of the at least one plant obtained using a multi-camera array, and
the segmenting, the determining the color, the determining the number of seeds, the inferring, and the estimating are performed using the plurality of digital images.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the plurality of digital images comprises digital images obtained at a plurality of positions along a length of a row of plants.

### Claim 8

A computer program product comprising one or more non-transitory computer-readable storage media having program instructions collectively stored on the one or more computer-readable storage media, the program instructions executable to:
receive training data comprising a plurality of digital images of a plurality of plants, wherein each of the plurality of digital images is labeled based on a ground truth moisture content of a seedpod;
generate preprocessed training data using the training data by:
for each of the plurality of digital images, segment the digital image to identify at least one seedpod in the digital image; and
for each of the at least one seedpod in each of the plurality of digital images:
determine a color of the seedpod; and
determine a number of seeds in the seedpod; and
train one or more machine learning models to predict one or both of a moisture content of the seedpod and a weight of the seedpod, using the preprocessed training data and the ground truth moisture content.

### Claim 9 (depends on 8)

The computer program product according to claim 8, wherein the generating the preprocessed training data further comprises, for each of the plurality of digital images, determining a size of each of the at least one seedpod in the digital image.

### Claim 10 (depends on 8)

The computer program product according to claim 8, wherein the weight of the seedpod is a wet weight.

### Claim 11 (depends on 8)

The computer program product according to claim 8, wherein one or more of the machine learning models is a convolutional neural network model.

### Claim 12 (depends on 8)

The computer program product according to claim 8, wherein the plurality of digital images is obtained using a multi-camera array.

### Claim 13 (depends on 12)

The computer program product according to claim 12, wherein the plurality of digital images comprises digital images obtained at a plurality of positions along a length of a row of plants.

### Claim 14

A system comprising:
a processor, a computer-readable memory, one or more computer-readable storage media, and program instructions collectively stored on the one or more computer-readable storage media, the program instructions executable to:
obtain a first digital image of at least one plant;
segment the first digital image of the at least one plant to identify at least one seedpod in the first digital image;
for each of the at least one seedpod in the first digital image:
determine a color of the seedpod;
determine a number of seeds in the seedpod; and
infer, using one or more machine learning models, one or both of a moisture content of the seedpod and a weight of the seedpod, based on the color of the seedpod and the number of seeds in the seedpod; and
predict a crop yield based on the moisture content and the weight of each of the at least one seedpod.

### Claim 15 (depends on 14)

The system according to claim 14, wherein:
the program instructions are further executable to determine a size of each of the at least one seedpod in the first digital image; and
for each of the at least one seedpod in the first digital image, the weight of the seedpod is determined based on the size of the seedpod.

### Claim 16 (depends on 14)

The system according to claim 14, wherein:
the program instructions are further executable to determine a number of seedpods on each of the at least one plant in the first digital image; and
the predicting the crop yield is further based on the number of seedpods on each of the at least one plant in the first digital image.

### Claim 17 (depends on 14)

The system according to claim 14, wherein:
the weight is a wet weight; and
the predicting the crop yield comprises predicting a dry weight based on the wet weight of each of the at least one seedpod and the moisture content of each of the at least one seedpod.

### Claim 18 (depends on 14)

The system according to claim 14, wherein one or more of the machine learning models is a convolutional neural network model.

### Claim 19 (depends on 14)

The system according to claim 14, wherein:
the first digital image is one of a plurality of digital images of the at least one plant obtained using a multi-camera array, and
the segmenting, the determining the color, the determining the number of seeds, and the inferring are performed using the plurality of digital images.

### Claim 20 (depends on 19)

The system according to claim 19, wherein the plurality of digital images comprises digital images obtained at a plurality of positions along a length of a row of plants.
