# US 10,813,295 B2 · Method for monitoring growth of plants and generating a plant grow schedule

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,813,295 B2
- Document ID: US10813295B2
- Status: Granted
- Kind: B2
- Application number: 15/955,651
- Filed: 2018-04-17
- Published: 2018-10-18
- Granted: 2020-10-27
- Earliest claimed: 2017-04-17
- Assignees: Iron Ox, Inc.
- Applicants: Iron Ox, Inc.
- Inventors: Brandon Ace Alexander; Jonathan BInney; Winnie Ding; Zachary Wielgosz
- Web page: https://patentgrep.com/patent/US10813295B2

## CPC

- A01G 5/00: Floral handling
- A01G 9/243: {Collecting solar energy (in general F24S)}
- A01G 7/00: Botany in general
- A01G 31/02: Special apparatus therefor (self-acting watering devices A01G27/00)
- B25J 9/1679: {characterised by the tasks executed}
- G01N 33/0098: {Plants or trees (wood G01N33/46)}
- G05B 19/416: characterised by control of velocity, acceleration or deceleration (G05B19/19 takes precedence)
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/0016: {involving temporal comparison}
- G05B 2219/39369: Host and robot controller and vision processing
- G05B 2219/40233: Portable robot
- G06T 2207/10024: Color image
- G06T 2207/30188: Vegetation; Agriculture
- Y10S 901/01: Mobile robot
- Y10S 901/47: Optical
- A01G: HORTICULTURE; CULTIVATION OF VEGETABLES, FLOWERS, RICE, FRUIT, VINES, HOPS OR SEAWEED; FORESTRY; WATERING (picking of fruits, vegetables, hops or the like A01D46/00; propagating unicellular algae C12N1/12)
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- Y10S: TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS

## Abstract

One variation of method for monitoring growth of plants within a facility includes: aggregating global ambient data recorded by a suite of fixed sensors, arranged proximal a grow area within the facility, at a first frequency during a grow period; extracting interim outcomes of a set of plants, occupying a module in the grow area, from module-level images recorded by a mover at a second frequency less than the first frequency while interfacing with the module during the period of time; dispatching the mover to autonomously deliver the module to a transfer station; extracting interim outcomes of the set of plants from plant-level images recorded by the transfer station while sequentially transferring plants out of the module at the conclusion of the grow period; and deriving relationships between ambient conditions, interim outcomes, and final outcomes from a corpus of plant records associated with plants grown in the facility.

## Claims

### Claim 1

A method for monitoring growth of plants within a facility comprising:
accessing a series of global ambient data recorded by a suite of fixed sensors, arranged proximal a grow area within the facility, at a first frequency during a first period of time;
writing global ambient data to plant records associated with plants occupying a group of modules within the grow area based on known locations of modules within the grow area and known locations of plants within the group of modules during the first period of time;
dispatching a mover to autonomously navigate to the group of modules during the first period of time;
accessing a series of module-level images of a first module, in the group of modules, recorded by the mover at a second frequency less than the first frequency during the first period of time;
extracting interim outcomes of a first set of plants, occupying the first module, from the series of module-level images;
writing interim outcomes of plants in the first set of plants to plant records associated with plants in the first set of plants based on known locations of the first set of plants within the first module during the first period of time;
dispatching the mover to autonomously deliver the first module to a transfer station;
for each plant in the first set of plants:
accessing a plant-level image of the plant recorded by the transfer station while transferring the plant out of the first module during a second period of time succeeding the first period of time;
extracting a final outcome of the plant from the plant-level image; and
writing the final outcome to a plant record associated with the plant; and
deriving relationships between ambient conditions, interim outcomes, and final outcomes from a corpus of plant records associated with plants grown in the facility.

### Claim 2 (depends on 1)

The method of claim 1:
wherein writing global ambient data to plant records comprises, for a first plant in the first set of plants in the first module occupying the grow area during the first period of time, writing representations of high-frequency time series of ambient conditions recorded by the suite of fixed sensors during the first period of time to a first multi-dimensional vector associated with the first plant;
wherein writing interim outcomes of plants to plant records comprises writing moderate-frequency, moderate-resolution quantitative representations of visual characteristics of the first plant, extracted from the module-level image, to the first multi-dimensional vector;
wherein writing final outcomes to plant records comprises labeling the first multi-dimensional vector with low-frequency, high-resolution quantitative representations of visual characteristics of the first plant, extracted from a first plant-level image of the first plant; and
wherein deriving relationships between ambient conditions, interim outcomes, and final outcomes from plant records associated with plants grown in the facility comprises:
detecting clusters of multi-dimensional vectors, in the corpus of plant records, labeled with similar final outcomes; and
deriving relationships between ambient conditions, interim outcomes, and final outcomes from differences in ambient conditions and final outcomes represented in clusters of multi-dimensional vectors.

### Claim 3 (depends on 1)

The method of claim 1:
wherein deriving relationships between ambient conditions, interim outcomes, and final outcomes from plant records associated with plants grown in the facility comprises deriving strengths of relationships between ambient conditions, recorded at the first frequency, and final outcomes of plants grown in the facility; and
further comprising:
generating a new grow schedule defining target ambient conditions for a new plant over a growth cycle of the new plant and predicted to yield a positive final outcome for the new plant based on derived strengths of relationships between ambient conditions and final outcomes of plants grown in the facility; and
for a second module loaded with a second set of plants and occupying the grow area during a third period of time succeeding the second period of time, autonomously adjusting actuators within the facility according to the new grow schedule based on global ambient data recorded by the suite of fixed sensors during the third period of time.

### Claim 4 (depends on 3)

The method of claim 3:
wherein deriving relationships between ambient conditions, interim outcomes, and final outcomes from plant records associated with plants grown in the facility further comprises deriving strengths of relationships between interim outcomes, extracted from module-level images recorded by the mover at the second frequency per module, and final outcomes of plants grown in the facility; and
further comprising:
dispatching the mover to autonomously navigate to the grow area during the third period of time;
accessing a second module-level image of the second module recorded by the mover at a particular time during the third period of time;
extracting interim outcomes of the second set of plants from the second module-level image;
calculating a predicted interim outcome of the second set of plants at the particular time based on global ambient data recorded by the suite of fixed sensors during the third period of time up to the particular time and derived relationships between ambient conditions and final outcomes of plants grown in the facility; and
in response to interim outcomes of the second set of plants extracted from the second module-level image differing from the predicted interim outcome, modifying the new grow schedule for a remainder of a growth cycle of the second set of plants based on derived relationships between ambient conditions, interim outcomes, and final outcomes.

### Claim 5 (depends on 3)

The method of claim 3, further comprising:
dispatching the mover to autonomously deliver the first module to a transfer station;
for each plant in the second set of plants:
accessing a plant-level image of the plant recorded by the transfer station while transferring the plant out of the second module during a fourth period of time succeeding the third period of time; and
extracting a final outcome of the plant from the plant-level image; and
calculating a predicted final outcome of the second set of plants at the particular time based on global ambient data recorded by the suite of fixed sensors during the third period of time and derived relationships between ambient conditions and final outcomes of plants grown in the facility; and
in response to final outcomes of the second set of plants aligning with the predicted final outcome, assigning the new grow schedule to modules, loaded with new plants, throughout the facility.

### Claim 6 (depends on 1)

The method of claim 1:
wherein accessing the series of global ambient data recorded by the suite of fixed sensors comprises accessing the series of global ambient data recorded by the suite of fixed sensors at multiple instances per day;
wherein dispatching the mover to autonomously navigate to the group of modules during the first period of time comprises dispatching the mover to autonomously navigate to each module in the group of modules once per day;
wherein accessing the series of module-level images of the first module comprises accessing one module-level image of the first module recorded by the mover per day;
wherein dispatching the mover to autonomously deliver the first module to the transfer station comprises dispatching the mover to autonomously deliver the first module to the transfer station for harvest of the first set of plants in response to conclusion of a growth cycle of the first set of plants; and
wherein accessing the plant-level image of each plant in the first set of plants comprises accessing plant-level images of each plant in the first set of plants recorded by a robotic manipulator at the transfer station during automated removal of the first set of plants from the first module.

### Claim 7 (depends on 1)

The method of claim 1:
further comprising:
accessing a series of global images of the group of modules, occupying the grow area, recorded by the suite of fixed sensors at approximately the first frequency during the first period of time;
segmenting the series of global images into a series of image segments representing singular modules in the group of modules;
extracting low-resolution interim outcomes of the first set of plants from image segments representing the first module during the first period of time; and
writing low-resolution interim outcomes of plants in the first set of plants to plant records associated with plants in the first set of plants; and
wherein extracting interim outcomes of the first set of plants from the series of module-level images comprises extracting moderate-resolution interim outcomes of the first set of plants from the series of module-level images.

### Claim 8 (depends on 7)

The method of claim 7:
wherein extracting final outcomes of plants in the first set of plants from plant-level images comprises extracting high-resolution final outcomes of plants in the first set of plants from plant-level images;
further comprising:
triggering a robotic manipulator at the transfer station to transfer plants, in the first set of plants, from the first module into a harvest container;
triggering the robotic manipulator to record the plant-level images of plants in the first set of plants while transferring plants in the first set of plants from the first module into the harvest container;
selecting a representative plant in the first set of plants;
triggering the transfer station to record a three-dimensional optical scan of the representative plant; and
extracting an ultra-high-resolution final outcome of the representative plant from the three-dimensional optical scan; and
wherein deriving relationships between ambient conditions, interim outcomes, and final outcomes from the corpus of plant records comprises:
assigning a first weight to low-resolution interim outcomes, a second weight greater than the first weight to moderate-resolution interim outcomes, a third weight greater than the second weight to high-resolution final outcomes, and a fourth weight greater than the third weight to ultra-high-resolution final outcomes; and
deriving relationships between ambient conditions, interim outcomes, and final outcomes based on weighted outcomes in the corpus of plant records.

### Claim 9 (depends on 7)

The method of claim 7:
wherein extracting low-resolution interim outcomes of the first set of plants from image segments representing the first module during the first period of time comprises:
detecting the first set of plants in a first subset of image segments corresponding to the first module;
extracting low-resolution quantitative representations of colors and sizes of plants in the first set of plants during the first period of time from the first subset of image segments;
wherein writing low-resolution interim outcomes of plants in the first set of plants to plant records comprises writing low-resolution quantitative representations of colors and sizes of plants in the first set of plants during the first period of time to plant records associated with plants in the first set of plants based on a plant map specifying positions of the first set of plants in the first module;
wherein extracting moderate-resolution interim outcomes of the first set of plants from the series of module-level images comprises extracting moderate-resolution quantitative representations of colors, sizes, and pest indicators of plants in the first set of plants at instances throughout the first period of time from module-level images of the first module recorded by the mover during the first period of time;
wherein writing moderate-resolution interim outcomes of plants in the first set of plants to plant records comprises writing moderate-resolution quantitative representations of colors, sizes, and pest indicators of plants in the first set of plants at instances throughout the first period of time to plant records associated with plants in the first set of plants based on the plant map;
wherein extracting final outcomes of plants from plant-level images of plants in the first set of plants comprises extracting high-resolution quantitative representations of colors, sizes, and pest indicators of plants in the first set of plants from plant-level images recorded by the transfer station during harvest of the first set of plants from the first module; and
wherein writing high-resolution interim outcomes of plants in the first set of plants to plant records comprises writing high-resolution quantitative representations of colors, sizes, and pest indicators of plants in the first set of plants and a weight of plants in the first set of plants, measured by the transfer station, during harvest of the first set of plants to plant records associated with plants in the first set of plants.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
accessing module-level ambient data recorded by the mover while recording module-level images of the first module at the second frequency during the first period of time; and
writing module-level ambient data to plant records associated with plants in the first set of plants.

### Claim 11 (depends on 10)

The method of claim 10:
wherein accessing the series of global ambient data recorded by the suite of fixed sensors comprises accessing a series of global temperature data recorded by the suite of fixed sensors at the first frequency during the first period of time;
wherein accessing module-level ambient data recorded by the mover comprises accessing module-level temperature data recorded by a mobile temperature sensor supported by the mover proximal a module, in the group of modules, while recording a module-level image of the module during the first period of time; and
further comprising:
calculating a temperature correction map that transforms a global temperature value into module-level temperature values at discrete module locations within the grow area based on differences between global temperature data recorded by the suite of fixed sensors and concurrent module-level temperature data recorded by the mover; and
correcting global ambient data stored in plant records associated with plants in the first set of plants based on the temperature correction map and a known location of the first module in the grow area during the first period of time.

### Claim 12 (depends on 1)

The method of claim 1:
further comprising:
accessing water quality data recorded by the mover though a probe inserted into the first module by the mover while recording module-level images of the first module at the second frequency during the first period of time; and
writing water quality data of plants in the first set of plants to plant records associated with plants in the first set of plants based on known locations of the first set of plants within the first module during the first period of time; and
wherein deriving relationships between ambient conditions, interim outcomes, and final outcomes comprises deriving relationships between a set of input parameters, interim outcomes, and final outcomes from the corpus of plant records, the set of input parameters comprising ambient conditions and water qualities.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
dispatching the mover to autonomously navigate to the grow area during a third period of time succeeding the second period of time;
accessing a second module-level image of a second module recorded by the mover at a particular time during the third period of time;
extracting a second interim outcome of a second plant, in a second set of plants occupying the second module, from the second module-level image, the second plant at a particular stage of development at the particular time;
identifying a first plant, in the first set of plants, exhibiting a first interim outcome, at the particular stage of development, nearest the second interim outcome of the second plant; and
in response to a final outcome of the first plant differing from a target final outcome, flagging the second plant for culling from the second module by the transfer station prior to a scheduled completion of a growth cycle of the second plant.

### Claim 14 (depends on 13)

The method of claim 13:
wherein extracting final outcomes of plants in the first set of plants from plant-level images comprises characterizing sizes, colors, and indicators of pest presence of plants in the first set of plants from plant-level images recorded by the transfer station; and
wherein flagging the second plant for culling comprises flagging the second plant for culling in response to the final outcome of the first plant comprising one of a size different from a target plant size, a color differing from a target plant color, and a degree of pest presence exceeding a threshold pest presence.

### Claim 15 (depends on 13)

The method of claim 13, further comprising, in response to flagging the second plant for culling:
dispatching the mover to autonomously deliver the second module to the transfer station; and
triggering a robotic manipulator at the transfer station to:
optically scan the second module;
detect the second plant in the second module;
navigate an end effector to the second plant;
remove the second plant from the second module; and
discard the second plant.

### Claim 16 (depends on 1)

The method of claim 1, further comprising:
dispatching the mover to autonomously navigate to the grow area during a third period of time succeeding the second period of time;
accessing a second module-level image of a second module recorded by the mover at a particular time during the third period of time;
extracting interim outcomes of a second set of plants occupying the second module, from the second module-level image, the second set of plants at a particular stage of development at the particular time;
identifying a first plant, in the first set of plants, exhibiting a first interim outcome, at the particular stage of development, nearest the interim outcomes of the second set of plants and associated with a final outcome approximating a target final outcome;
aligning target ambient conditions for the second module following the particular time to ambient conditions proximal the first plant, from the particular stage of development to harvest, stored in a first plant record associated with the first plant; and
autonomously adjusting actuators within the facility according to the target ambient conditions for the second module based on global ambient data recorded by the suite of fixed sensors following the particular time.

### Claim 17 (depends on 1)

The method of claim 1, further comprising:
triggering a robotic manipulator at the transfer station to transfer plants, in the first set of plants, from the first module into a harvest container;
triggering the robotic manipulator to record the plant-level images of plants in the first set of plants while transferring plants in the first set of plants from the first module into the harvest container;
selecting a representative plant in the first set of plants;
triggering the robotic manipulator to transfer the representative plant from the first module into a test container; and
storing a taste test result of the representative plant as a final outcome in a plant record associated with the representative plant.

### Claim 18 (depends on 1)

The method of claim 1:
wherein extracting final outcomes of plants in the first set of plants from plant-level images comprises extracting final colors, sizes, and geometries of plants in the first set of plants from plant-level images recorded by the transfer station;
further comprising:
selecting a representative plant in the first set of plants; and
in response to final colors, sizes, and geometries of plants in the first set of plants differing from final colors, sizes, and geometries of a corpus of plants grown previously in the facility, as stored in the corpus of plant records, prompting a human operator to indicate visual appeal of the representative plant; and
wherein deriving relationships between ambient conditions, interim outcomes, and final outcomes comprises deriving relationships between ambient conditions during the growth cycles of plants and visual appeal of plants at conclusion of a growth cycle from the corpus of plant records associated with plants grown in the facility.

### Claim 19 (depends on 1)

The method of claim 1:
wherein accessing the series of global ambient data comprises accessing a series of global humidity data recorded by the suite of fixed sensors during the first period of time;
wherein deriving relationships between ambient conditions, interim outcomes, and final outcomes comprises deriving relationships between ambient humidity and final outcomes of plants grown in the facility;
further comprising:
setting a target humidity range in the grow area based on derived relationships between ambient humidity and final outcomes of plants grown in the facility;
estimating leaf areas of plants occupying modules in the grow area based on features extracted from module-level images recently recorded by the mover;
predicting a humidity in the grow area at a future time based on a sum of leaf areas of plants occupying modules in the grow area and a plant transpiration model;
in response to the humidity in the grow area at the future time exceeding the target humidity range, identifying a second module, occupied by a second set of plants nearest a target outcome, in module-level images recently recorded by the mover;
dispatching the mover to autonomously deliver the second module to the transfer station; and
triggering the transfer station to harvest the second set of plants from the second module.

### Claim 20 (depends on 1)

The method of claim 1:
wherein accessing the series of global ambient data comprises accessing a series of global humidity data recorded by the suite of fixed sensors during the first period of time;
wherein deriving relationships between ambient conditions, interim outcomes, and final outcomes comprises deriving relationships between ambient humidity and interim outcomes and deriving relationships between interim outcomes and final outcomes of plants grown in the facility;
further comprising:
setting a target humidity range in the grow area based on derived relationships between ambient humidity and interim outcomes and derived relationships between interim outcomes and final outcomes of plants grown in the facility;
estimating leaf areas of plants occupying modules in the grow area based on features extracted from module-level images recently recorded by the mover;
predicting a humidity in the grow area at a future time based on a sum of leaf areas of plants occupying modules in the grow area and a plant transpiration model;
in response to the humidity in the grow area at the future time exceeding the target humidity range, identifying a second module, occupied by a second set of plants exhibiting interim outcomes indicating poor final outcomes, in module-level images recently recorded by the mover;
dispatching the mover to autonomously deliver the second module to the transfer station; and
triggering the transfer station to cull and discard the second set of plants from the second module.

### Claim 21

A method for monitoring growth of plants within a facility comprising:
aggregating global ambient data recorded by a suite of fixed sensors, arranged proximal a grow area within the facility, at a first frequency during a grow period;
extracting interim outcomes of a set of plants, occupying a module in the grow area, from module-level images recorded by a mover at a second frequency less than the first frequency while interfacing with the module during the grow period;
dispatching the mover to autonomously deliver the module to a transfer station;
extracting final outcomes of the set of plants from plant-level images recorded by the transfer station while sequentially transferring plants out of the module at the conclusion of the grow period; and
deriving relationships between ambient conditions, interim outcomes, and final outcomes from a corpus of plant records associated with plants grown in the facility.