# US 2024/0177470 A1 · Machine learning systems and methods for generating structural representations of plants

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0177470 A1
- Document ID: US20240177470A1
- Status: Published application
- Kind: A1
- Application number: 18/285,193
- Filed: 2022-03-29
- Published: 2024-05-30
- Earliest claimed: 2021-03-30
- Assignees: Terramera, Inc.
- Applicants: Terramera, Inc.
- Inventors: Darren Bryce SUTTON; Ghassan HAMARNEH; Carolina PARTIDA
- Web page: https://patentgrep.com/patent/US20240177470A1

## CPC

- G06V 10/82: using neural networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/86: using syntactic or structural representations of the image or video pattern, e.g. symbolic string recognition; using graph matching
- G06N 3/045: Combinations of networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods for training a machine learning model for generating a structural representation of a plant are provided, as well as systems and methods for generating a structural representation of a plant via such a model. The training method involves encoding a plant image into a structural representation of the plant (e.g. a “skeleton”), decoding the structural representation of the plant into a reconstructed image of the plant, and classifying the reconstructed image as having been generated based on a ground-truth structural representation or output of the encoder. Such classification incentivizes the encoder to produce structural representations which do not “smuggle” texture information (e.g. appearance, such as color). Texture information may be separately represented. The encoder, once trained, may be used to generate structural representations from plant images without necessarily requiring decoding or classification.

## Claims

### Claim 1

A method for training a machine learning model for generating a structural representation of a plant, the method performed by a processor and comprising:
transforming, by an encoder, an input plant image to a predicted structural representation based on parameters of the encoder, the input plant image one of a plurality of plant images, each plant image representing a plant having a plurality of connected plant structures, and the predicted structural representation representing predicted connections between the plant structures of the input plant image;
transforming, by a decoder, the predicted structural representation to a reconstructed plant image based on parameters of the decoder;
classifying, by a discriminator, the reconstructed plant image as having been generated based on one of: the ground truth training dataset or parameters of the encoder to generate a classification based on parameters of the discriminator; and
training parameters of the encoder based on the classification.

### Claim 2 (depends on 1)

The method according to claim 1 comprising training the discriminator by:
transforming, by the decoder, a ground-truth structural representation from a ground truth training dataset of predetermined structural representations to a second reconstructed plant image;
transforming, by the decoder, a second predicted structural representation to a third reconstructed plant image based on parameters of the decoder, the second predicted structural representation generated by the encoder based on parameters of the encoder;
classifying, by the discriminator, each of the second and third reconstructed plant images as having been generated based on one of: the ground truth training dataset or parameters of the encoder to generate respective second and third classifications based on parameters of the discriminator; and
training parameters of the discriminator based on the second and third classifications.

### Claim 3 (depends on 1)

The method according to claim 1 wherein transforming the predicted structural representation to the reconstructed plant image comprises transforming the predicted structural representation to the first reconstructed plant image based on a texture representation separate from the predicted structural representation.

### Claim 4 (depends on 3)

The method according to claim 3 comprising transforming, by a texture encoder, the input plant image to the texture representation, the texture representation comprising a relatively lower-dimensional encoding than the input plant image.

### Claim 5 (depends on 4)

The method according to claim 4 comprising training parameters of the texture encoder based on the input plant image, the reconstructed plant image, and the classification.

### Claim 6 (depends on 1)

The method according to claim 1 wherein training parameters of the encoder comprises training parameters of the encoder based on the input plant image and the reconstructed plant image.

### Claim 7 (depends on 6)

The method according to claim 6 wherein training the parameters of the encoder comprises determining a value of an objective function based on a difference between the input plant image and the reconstructed plant image; determining a gradient of the objective function relative to parameters of the encoder based on the difference; and updating parameters of the encoder based on the gradient.

### Claim 8 (depends on 6)

The method according to claim 6 wherein training parameters of the encoder comprises semisupervised training of parameters of the encoder, the semisupervised training comprising:
transforming a first number of the plurality of plant images to a plurality of predicted plant structural representations by the encoder;
transforming a second number of ground-truth structural representations from the ground truth training dataset to a plurality of reconstructed plant images by the decoder, the first number greater than the second number; and
training the parameters of the encoder based on the plurality of reconstructed plant images and the plurality of plant images.

### Claim 9 (depends on 8)

The method according to claim 8 wherein the ground truth training dataset comprises at most a number of ground-truth structural representations no greater than: 1%, 2%, 3%, 4%, 5%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, or 90% of a number of plant images of the plurality of plant images.

### Claim 10 (depends on 9)

The method according to claim 9 wherein the ground-truth structural representations of the ground-truth training dataset represent a plurality of plants other than the plants represented by the plurality of plant images.

### Claim 11 (depends on 1)

The method according to claim 1 wherein training parameters of the encoder comprises, for each of the predicted connections of the predicted structural representation, determining a plant morphology of the predicted connection and training the encoder based on the plant morphology.

### Claim 12 (depends on 11)

The method according to claim 11 wherein determining the plant morphology comprises determining a branching characteristic, the branching characteristic comprising at least one of: number of branches, branch length, average branch length, branch type, branch curvature, and/or branch endpoint location.

### Claim 13 (depends on 12)

The method according to claim 12 wherein the branching characteristic comprises branch type and determining the branching characteristic comprises determining, for at least one predicted connection, that the at least one predicted connection comprises a branch type of: tip-tip, tip-junction, junction-junction, path-path.

### Claim 14 (depends on 13)

The method according to claim 13 wherein training parameters of the encoder comprises training parameters of the encoder to incentivize tip-junction branch types relative to one or more of tip-tip, junction-junction, and path-path branch types.

### Claim 15 (depends on 12)

The method according to claim 12 wherein training parameters of the encoder comprises halting training based on the plant morphology.

### Claim 16 (depends on 12)

The method according to claim 12 wherein training parameters of the encoder comprises determining a value of an objective function based on the plant morphology and training parameters of the encoder based on the value of the objective function.

### Claim 17 (depends on 11)

The method according to claim 11 wherein the plant morphology comprises connectivity of the predicted connections, the method comprises determining a value of an objective function based on a shape prior representing connectivity of the predicted connections, and training parameters of the encoder comprises training parameters of the encoder based on the value of the objective function.

### Claim 18 (depends on 17)

The method according to claim 17 wherein the method comprises transforming the predicted structural representation to a polar predicted structural representation having polar coordinates and determining the value of the objective function comprises determining a difference between the polar predicted structural representation and the shape prior in a polar domain.

### Claim 19 (depends on 18)

The method according to claim 18 wherein transforming the predicted structural representation to the polar predicted structural representation comprises determining a center point of the predicted structural representation and transforming the predicted structural representation to the polar predicted structural representation based on the center point.

### Claim 20 (depends on 19)

The method according to claim 19 wherein determining the center point of the predicted structural representation comprises determining a center point of the input plant image and identifying a corresponding point in the predicted structural representation as the center point of the predicted structural representation.

### Claim 21 (depends on 1)

The method according to claim 1 wherein training parameters of the encoder comprises determining a value of an objective function based on one or more structural classifications generated by classifying the predicted structural representation, by one or more structure discriminators, as having been drawn from the ground truth training dataset or generated by the encoder, the value of the objective function based on the one or more structural classifications.

### Claim 22 (depends on 21)

The method according to claim 21 wherein the one or more structure discriminators comprises a cartesian structure discriminator and classifying the predicted structural representation comprises generating a cartesian classification based on an image of the predicted structural representation comprising cartesian coordinates.

### Claim 23 (depends on 21)

The method according to claim 21 wherein the one or more structure discriminators comprises a polar structure discriminator and classifying the predicted structural representation comprises generating a polar classification based on an image of the predicted structural representation comprising polar coordinates.

### Claim 24 (depends on 23)

The method according to claim 23 comprising determining a center point of a cartesian predicted structural representation and transforming the cartesian predicted structural representation to the predicted structural representation comprising polar coordinates.

### Claim 25 (depends on 24)

The method according to claim 24 wherein determining the center point of the cartesian predicted structural representation comprises determining a center point of the input plant image and identifying a corresponding point in the cartesian predicted structural representation as the center point of the cartesian predicted structural representation.

### Claim 26 (depends on 1)

The method according to claim 1 wherein training parameters of the encoder comprises determining a value of an objective function based on one or more reconstruction classifications generated by classifying the reconstructed plant image, by one or more reconstruction discriminators, as having been generated by the decoder or drawn from the plurality of plant images, the value of the objective function based on the one or more reconstruction classifications.

### Claim 27 (depends on 26)

The method according to claim 26 wherein the one or more reconstruction discriminators comprises a first reconstruction discriminator and classifying the reconstructed plant representation by the first reconstruction discriminator comprises classifying the reconstructed plant image as having been generated by the decoder based on a ground-truth structural representation from the ground-truth training dataset or drawn from the plurality of plant images.

### Claim 28 (depends on 27)

The method according to claim 27 wherein the one or more reconstruction discriminators comprises a first reconstruction discriminator and classifying the reconstructed plant representation by the first reconstruction discriminator comprises classifying the reconstructed plant image as having been generated by the decoder based on the predicted structural representation generated by the encoder or drawn from the plurality of plant images.

### Claim 29 (depends on 1)

A method for generating a structural representation of a plant by a machine learning model, the method performed by a processor and comprising:
transforming, by an encoder, an input plant image to a predicted structural representation based on parameters of the encoder, the parameters of the encoder trained according to claim 1.

### Claim 30

A method for training a machine learning model for generating a structural representation for an object, the method performed by a processor and comprising:
transforming, by an encoder, an input image depicting the object to a predicted structural representation based on parameters of the encoder, the input image one of a plurality of images, each image representing an object having a structure and a texture, and the predicted structural representation representing the structure of the object;
transforming, by a decoder, the predicted structural representation to a reconstructed image based on parameters of the decoder;
classifying, by a discriminator, the reconstructed image as having been generated based on one of: a ground truth training dataset or parameters of the encoder to generate a classification based on parameters of the discriminator; and
training parameters of the encoder based on the classification.

### Claim 31 (depends on 30)

The method according to claim 30 comprising training the discriminator by:
transforming, by the decoder, a ground-truth structural representation from a ground truth training dataset of predetermined structural representations to a second reconstructed image;
transforming, by the decoder, a second predicted structural representation to a third reconstructed plant image based on parameters of the decoder, the second predicted structural representation generated by the encoder based on parameters of the encoder;
classifying, by the discriminator, each of the second and third reconstructed images as having been generated based on one of: the ground truth training dataset or parameters of the encoder to generate respective second and third classifications based on parameters of the discriminator; and
training parameters of the discriminator based on the second and third classifications.

### Claim 32 (depends on 30)

The method according to claim 30 wherein transforming the predicted structural representation to the reconstructed image comprises transforming the predicted structural representation to the first reconstructed image based on a texture representation separate from the predicted structural representation.

### Claim 33 (depends on 32)

The method according to claim 32 comprising transforming, by a texture encoder, the input image to the texture representation, the texture representation comprising a relatively lower-dimensional encoding than the input image.

### Claim 34 (depends on 33)

The method according to claim 33 comprising training parameters of the texture encoder based on the input image, the reconstructed image, and the classification.

### Claim 35 (depends on 30)

The method according to claim 30 wherein training parameters of the encoder comprises training parameters of the encoder based on the input image and the reconstructed image.

### Claim 36 (depends on 35)

The method according to claim 35 wherein training the parameters of the encoder comprises determining a value of an objective function based on a difference between the input image and the reconstructed image; determining a gradient of the objective function relative to parameters of the encoder based on the difference; and updating parameters of the encoder based on the gradient.

### Claim 37 (depends on 35)

The method according to claim 35 wherein training parameters of the encoder comprises semisupervised training of parameters of the encoder, the semisupervised training comprising:
transforming a first number of the plurality of images to a plurality of predicted object structural representations by the encoder;
transforming a second number of ground-truth structural representations from the ground truth training dataset to a plurality of reconstructed images by the decoder, the first number greater than the second number; and
training the parameters of the encoder based on the plurality of reconstructed images and the plurality of images.

### Claim 38 (depends on 37)

The method according to claim 37 wherein the ground truth training dataset comprises at most a number of ground-truth structural representations no greater than: 1%, 2%, 3%, 4%, 5%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, or 90% of a number of images of the plurality of images.

### Claim 39 (depends on 38)

The method according to claim 38 wherein the ground-truth structural representations of the ground-truth training dataset represent a plurality of objects other than the objects represented by the plurality of images.

### Claim 40 (depends on 30)

The method according to claim 30 wherein training parameters of the encoder comprises determining a value of an objective function based on one or more structural classifications generated by classifying the predicted structural representation, by one or more structure discriminators, as having been drawn from the ground truth training dataset or generated by the encoder, the value of the objective function based on the one or more structural classifications.

### Claim 41 (depends on 40)

The method according to claim 40 wherein the one or more structure discriminators comprises a cartesian structure discriminator and classifying the predicted structural representation comprises generating a cartesian classification based on an image of the predicted structural representation comprising cartesian coordinates.

### Claim 42 (depends on 40)

The method according to claim 40 wherein the one or more structure discriminators comprises a polar structure discriminator and classifying the predicted structural representation comprises generating a polar classification based on an image of the predicted structural representation comprising polar coordinates.

### Claim 43 (depends on 42)

The method according to claim 42 comprising determining a center point of a cartesian predicted structural representation and transforming the cartesian predicted structural representation to the predicted structural representation comprising polar coordinates.

### Claim 44 (depends on 43)

The method according to claim 43 wherein determining the center point of the cartesian predicted structural representation comprises determining a center point of the input image and identifying a corresponding point in the cartesian predicted structural representation as the center point of the cartesian predicted structural representation.

### Claim 45 (depends on 30)

The method according to claim 30 wherein training parameters of the encoder comprises determining a value of an objective function based on one or more reconstruction classifications generated by classifying the reconstructed image, by one or more reconstruction discriminators, as having been generated by the decoder or drawn from the plurality of images, the value of the objective function based on the one or more reconstruction classifications.

### Claim 46 (depends on 45)

The method according to claim 45 wherein the one or more reconstruction discriminators comprises a first reconstruction discriminator and classifying the reconstructed object representation by the first reconstruction discriminator comprises classifying the reconstructed image as having been generated by the decoder based on a ground-truth structural representation from the ground-truth training dataset or drawn from the plurality of images.

### Claim 47 (depends on 46)

The method according to claim 46 wherein the one or more reconstruction discriminators comprises a first reconstruction discriminator and classifying the reconstructed object representation by the first reconstruction discriminator comprises classifying the reconstructed image as having been generated by the decoder based on the predicted structural representation generated by the encoder or drawn from the plurality of images.

### Claim 48 (depends on 1)

A method for generating a structural representation for an object by a machine learning model, the method performed by a processor and comprising:
transforming, by an encoder, an input image to a predicted structural representation based on parameters of the encoder, the parameters of the encoder trained according to claim 1.

### Claim 49 (depends on 1)

A computer system comprising:
one or more processors; and
a memory storing instructions which cause the one or more processors to perform operations comprising the method of claim 1.
