# US 2020/0401883 A1 · Individual plant recognition and localization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401883 A1 - Document ID: US20200401883A1 - Status: Published application - Kind: A1 - Application number: 16/450,238 - Filed: 2019-06-24 - Published: 2020-12-24 - Earliest claimed: 2019-06-24 - Assignees: X Development LLC - Applicants: X Development LLC - Inventors: Jie Yang; Zhiqiang Yuan; Hongxu Ma; Cheng-en Guo; Elliott Grant; Yueqi Li - Web page: https://patentgrep.com/patent/US20200401883A1 ## CPC - G06T 7/001: {using an image reference approach} - G05D 1/0033: {by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle} - G05D 1/2235: {involving the operator tracking the vehicle by direct line of sight} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/285: {Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system} - G06N 3/0418: {using chaos or fractal principles} - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes} - G06V 10/74: Image or video pattern matching; Proximity measures in feature spaces - G06V 10/764: using classification, e.g. of video objects - G06V 10/776: Validation; Performance evaluation - G06V 10/82: using neural networks - G06V 20/188: {Vegetation} - G06V 20/20: in augmented reality scenes - G06T 2207/10032: Satellite or aerial image; Remote sensing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30188: Vegetation; Agriculture - G06V 2201/10: Recognition assisted with metadata - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Implementations are described herein for training and applying machine learning models to digital images capturing plants, and to other data indicative of attributes of individual plants captured in the digital images, to recognize individual plants in distinction from other individual plants. In various implementations, a digital image that captures a first plant of a plurality of plants may be applied, along with additional data indicative of an additional attribute of the first plant observed when the digital image was taken, as input across a machine learning model to generate output. Based on the output, an association may be stored in memory, e.g., of a database, between the digital image that captures the first plant and one or more previously-captured digital images of the first plant. ## Claims ### Claim 1 A method implemented using one or more processors, comprising: obtaining a digital image that captures at least a first plant of a plurality of plants; determining, based on a sensor signal generated by a sensor, additional data indicative of an additional attribute of the first plant; applying the digital image and the additional data as input across a machine learning model to generate output, wherein the machine learning model is trained using a plurality of training instances, and wherein each training instance includes one or more training digital images of a particular plant and additional data indicative of an additional attribute of the particular plant when the one or more training digital images were taken; and based on the output, storing in memory an association between the digital image that captures the first plant and one or more previously-captured digital images of the first plant. ### Claim 2 (depends on 1) The method of claim 1, wherein the obtaining includes: operating an autonomous or semiautonomous vehicle equipped with a vision sensor to traverse along a path that is proximate to at least a subset of the plurality of plants; and operating the vision sensor to acquire the digital image that captures the first plant. ### Claim 3 (depends on 1) The method of claim 1, wherein the machine learning model comprises a convolutional neural network. ### Claim 4 (depends on 3) The method of claim 3, wherein the convolutional neural network is trained on the plurality of training instances using triplet loss. ### Claim 5 (depends on 1) The method of claim 1, wherein the sensor comprises a position coordinate sensor, and the additional attribute of the first plant comprises a position coordinate indicative of a location of the first plant. ### Claim 6 (depends on 1) The method of claim 1, wherein the sensor comprises a vision sensor. ### Claim 7 (depends on 6) The method of claim 6, wherein the additional attribute of the first plant comprises a spatial dimension of a bounding shape, determined from one or more digital images generated by the vision sensor, that encloses at least a portion of the first plant. ### Claim 8 (depends on 7) The method of claim 7, wherein the bounding shape comprises a minimum bounding shape that encloses an entirety of the first plant. ### Claim 9 (depends on 6) The method of claim 6, wherein the additional attribute of the first plant comprises an environmental context of the first plant captured in one or more digital images generated by the vision sensor. ### Claim 10 (depends on 6) The method of claim 6, wherein the additional attribute of the first plant comprises a time interval since a milestone in a life of the first plant. ### Claim 11 (depends on 1) The method of claim 1, further comprising: receiving, from a graphical user interface rendered on a computing device, user selection of a user interface element that corresponds to the first plant; and based on the receiving, causing the computing device to output first plant information based on the digital image that captures the first plant and one or more of the previously-captured digital images of the first plant. ### Claim 12 (depends on 11) The method of claim 11, wherein the first plant information includes a time-based sequence of digital images that includes the digital image that captures the first plant and one or more of the previously-captured digital images of the first plant. ### Claim 13 (depends on 11) The method of claim 11, wherein the first plant information includes first plant growth or health statistics generated based on the digital image that captures the first plant and one or more of the previously-captured digital images of the first plant. ### Claim 14 (depends on 1) The method of claim 1, further comprising: analyzing the digital image that captures the first plant and one or more of the previously-captured digital images of the first plant; and based on the analyzing, predicting a growth rate or yield of the first plant. ### Claim 15 A system comprising one or more processors and memory storing instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to: obtain a digital image that captures at least a first plant of a plurality of plants; determine, based on a sensor signal generated by a sensor, additional data indicative of an additional attribute of the first plant; applying the digital image and the additional data as input across a machine learning model to generate output, wherein the machine learning model is trained using a plurality of training instances, and wherein each training instance includes one or more training digital images of a particular plant and additional data indicative of an additional attribute of the particular plant when the one or more training digital images were taken; and based on the output, store in memory an association between the digital image that captures the first plant and one or more previously-captured digital images of the first plant. ### Claim 16 (depends on 15) The system of claim 15, wherein the obtaining includes: operating an autonomous or semiautonomous vehicle equipped with a vision sensor to traverse along a path that is proximate to at least a subset of the plurality of plants; and operating the vision sensor to acquire the digital image that captures the first plant. ### Claim 17 (depends on 15) The system of claim 15, wherein the machine learning model comprises a convolutional neural network. ### Claim 18 (depends on 17) The system of claim 17, wherein the convolutional neural network is trained on the plurality of training instances using triplet loss. ### Claim 19 (depends on 11) The system of claim 11, wherein the sensor comprises a position coordinate sensor, and the additional attribute of the first plant comprises a position coordinate indicative of a location of the first plant. ### Claim 20 At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations: obtaining a digital image that captures at least a first plant of a plurality of plants; determining, based on a sensor signal generated by a sensor, additional data indicative of an additional attribute of the first plant; applying the digital image and the additional data as input across a machine learning model to generate output, wherein the machine learning model is trained using a plurality of training instances, and wherein each training instance includes one or more training digital images of a particular plant and additional data indicative of an additional attribute of the particular plant when the one or more training digital images were taken; and based on the output, storing in memory an association between the digital image that captures the first plant and one or more previously-captured digital images of the first plant.