# US 12,008,744 B2 · Mapping field anomalies using digital images and machine learning models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,744 B2 - Document ID: US12008744B2 - Status: Granted - Kind: B2 - Application number: 16/707,355 - Filed: 2019-12-09 - Published: 2020-06-18 - Granted: 2024-06-11 - Earliest claimed: 2018-12-10 - Assignees: Climate LLC - Applicants: Climate LLC - Inventors: Boyan Peshlov; Weilin Wang - Web page: https://patentgrep.com/patent/US12008744B2 ## CPC - G06T 7/0004: {Industrial image inspection} - A01B 69/008: {automatic} - B64C 39/02: characterised by special use - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/096: Transfer learning - G06T 5/60: using machine learning, e.g. neural networks - G06V 10/82: using neural networks - G06V 10/955: {using specific electronic processors} - G06V 20/188: {Vegetation} - A01B 79/005: {Precision agriculture} - A01G 25/16: Control of watering - B64U 2101/30: for imaging, photography or videography - B64U 2101/32: for cartography or topography - G06T 2207/10024: Color image - G06T 2207/10032: Satellite or aerial image; Remote sensing - G06T 2207/10036: Multispectral image; Hyperspectral image - G06T 2207/10048: Infrared image - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30188: Vegetation; Agriculture - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 20/194: {using hyperspectral data, i.e. more or other wavelengths than RGB} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - A01B: SOIL WORKING IN AGRICULTURE OR FORESTRY; PARTS, DETAILS, OR ACCESSORIES OF AGRICULTURAL MACHINES OR IMPLEMENTS, IN GENERAL (making or covering furrows or holes for sowing, planting or manuring A01C5/00; mowers convertible to soil working apparatus or capable of soil working A01D42/04; mowers combined with soil working implements A01D43/12; soil working for engineering purposes E01, E02, E21) - B64C: AEROPLANES; HELICOPTERS - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - A01G: HORTICULTURE; CULTIVATION OF VEGETABLES, FLOWERS, RICE, FRUIT, VINES, HOPS OR SEAWEED; FORESTRY; WATERING (picking of fruits, vegetables, hops or the like A01D46/00; propagating unicellular algae C12N1/12) - B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR ## Abstract A computer-implemented method for generating an improved map of field anomalies using digital images and machine learning models is disclosed. In an embodiment, a method comprises: obtaining a shapefile that defines boundaries of an agricultural plot and boundaries of the field containing the plot; obtaining a plurality of plot images within the field from one or more image capturing devices that are located within the boundaries of the field; calibrating and pre-processing the plurality of plot images to create a plot map of the agricultural plot at a plot level; based on the plot map of the agricultural plot, generating a plot grid; based on the plot grid and the plot map, generating a plurality of plot tiles; based on the plurality of plot tiles, generating, using a first machine learning model and a plurality of first image classifiers corresponding to one or more first anomalies, a set of classified plot images that depicts at least one anomaly; based on the set of classified plot images, generating a plot anomaly map for the agricultural plot; transmitting the plot anomaly map to one or more controllers that control one or more agricultural machines or database systems to perform agricultural functions on the agricultural plot. ## Claims ### Claim 1 A computer-implemented method for generating an improved map of field anomalies using digital images and machine learning models, the method comprising: obtaining a shapefile that defines boundaries of an agricultural plot; based on the shapefile, obtaining a plurality of ground based plot images from one or more image capturing devices mounted at a fixed ground location or a ground vehicle at the agricultural plot; calibrating the plurality of ground based plot images; stitching the plurality of calibrated ground based plot images into a plot map of the agricultural plot at a plot level; generating a plot grid; based on the plot grid and the plot map, defining a plurality of plot tiles for the agricultural plot, each of the plurality of plot tiles including multiple pixels of the plurality of calibrated ground based plot images; classifying the plurality of plot tiles, using a first machine learning model and a plurality of first image classifiers corresponding to one or more first anomalies, into a set of classified plot images that depicts at least one anomaly, wherein each of the plurality of plot tiles is classified into classifications at least corresponding to a crop, a weed, trees, and inter-row damage; determining, for each image in the set of classified plot images, a probability that the image is correctly classified, and further comparing the probability to an acceptable probability; based on the set of classified plot images, generating a plot anomaly map for the agricultural plot; and transmitting the plot anomaly map to one or more controllers that control one or more agricultural machines to perform agricultural functions on the agricultural plot. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the shapefile is generated by performing: obtaining a plurality of aerial images of an agricultural field; calibrating the plurality of aerial images stitching the plurality of calibrated aerial images into a field map of the agricultural field at a field level; based on the field map of the agricultural field, generating a field grid; based on the field grid and the field map, defining a plurality of field tiles; classifying the plurality of field tiles, using a second machine learning model and a plurality of second image classifiers corresponding to one or more second anomalies, into a set of classified field images that depicts at least one field anomaly; based on the set of classified field images, generating a field anomaly map for the agricultural field; and based on the field anomaly map, generating the boundaries for the agricultural plot defined in the shapefile. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the agricultural plot is a part of the agricultural field. ### Claim 4 (depends on 2) The computer-implemented method of claim 2, wherein the plot anomaly map has a higher-level of detail than the field anomaly map; wherein the plurality of first image classifiers has a higher-level of detail than the plurality of second image classifiers; and wherein the plurality of first image classifiers includes two or more of: one or more interrow image classifiers, one or more weed image classifiers, one or more bare soil classifiers, one or more lodging classifiers, or one or more standing water classifiers; and wherein the one or more first anomalies have a higher-level of detail than the one or more second anomalies. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the shapefile is used to control the one or more image capturing devices configured to capture the plurality of ground based plot images from the agricultural plot defined by the boundaries. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the plurality of ground based plot images is captured by the one or more image capturing devices as the one or more image capturing devices are controlled based on contents of the shapefile specifying the boundaries of the agricultural plot. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the one or more image capturing devices are installed on any of: moving farming equipment, stationary farming equipment, stationary posts, stationary structures, handheld devices, or mobile devices. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the calibrating of the plurality of ground based plot images comprises correcting one or more colors depicted in the plurality of ground based plot images. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the plot anomaly map for the agricultural plot comprises one or more specific anomaly maps, each specific anomaly map depicting a specific anomaly identified for the agricultural plot. ### Claim 10 One or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause the one or more computing devices to perform: obtaining a shapefile that defines boundaries of an agricultural plot; based on the shapefile, obtaining a plurality of ground based plot images from one or more image capturing devices mounted at a fixed ground location or a ground vehicle at the agricultural plot; calibrating the plurality of ground based plot images; stitching the plurality of calibrated ground based plot images into a plot map of the agricultural plot at a plot level; generating a plot grid; based on the plot grid and the plot map, defining a plurality of plot tiles for the agricultural plot, each of the plurality of plot tiles including multiple pixels of the plurality of calibrated ground based plot images; classifying the plurality of plot tiles using a first machine learning model and a plurality of first image classifiers corresponding to one or more first anomalies, into a set of classified plot images that depicts at least one anomaly, wherein each of the plurality of plot tiles is classified into classifications at least corresponding to a crop, a weed, trees, and inter-row damage; determining, for each image in the set of classified plot images, a probability that the image is correctly classified, and further comparing the probability to an acceptable probability; based on the set of classified plot images, generating a plot anomaly map for the agricultural plot; and transmitting the plot anomaly map to one or more controllers that control one or more agricultural machines to perform agricultural functions on the agricultural plot. ### Claim 11 (depends on 10) The one or more non-transitory storage media of claim 10, storing additional instructions for generating the shapefile, which, when executed by the one or more computing devices, cause the one or more computing devices to perform: obtaining a plurality of aerial images of an agricultural field; calibrating the plurality of aerial images stitching the plurality of calibrated aerial images into a field map of the agricultural field at a field level; based on the field map of the agricultural field, generating a field grid; based on the field grid and the field map, defining a plurality of field tiles; classifying the plurality of field tiles using a second machine learning model and a plurality of second image classifiers corresponding to one or more second anomalies, into a set of classified field images that depicts at least one field anomaly; based on the set of classified field images, generating a field anomaly map for the agricultural field; and based on the field anomaly map, generating the boundaries for the agricultural plot defined in the shapefile. ### Claim 12 (depends on 11) The one or more non-transitory storage media of claim 11, wherein the agricultural plot is a part of the agricultural field. ### Claim 13 (depends on 11) The one or more non-transitory storage media of claim 11, wherein the plot anomaly map has a higher-level of detail than the field anomaly map; wherein the plurality of first image classifiers has a higher-level of detail than the plurality of second image classifiers; and wherein the plurality of first image classifiers includes two or more of: one or more interrow image classifiers, one or more weed image classifiers, one or more bare soil classifiers, one or more lodging classifiers, or one or more standing water classifiers; and wherein the one or more first anomalies have a higher-level of detail than the one or more second anomalies. ### Claim 14 (depends on 10) The one or more non-transitory storage media of claim 10, wherein the shapefile is used to control the one or more image capturing devices configured to capture the plurality of ground based plot images from the agricultural plot defined by the boundaries. ### Claim 15 (depends on 10) The one or more non-transitory storage media of claim 10, wherein the plurality of ground based plot images is captured by the one or more image capturing devices as the one or more image capturing devices are controlled based on contents of the shapefile specifying the boundaries of the agricultural plot. ### Claim 16 (depends on 10) The one or more non-transitory storage media of claim 10, wherein the one or more image capturing devices are installed on any of: moving farming equipment, stationary farming equipment, stationary posts, stationary structures, handheld devices, or mobile devices. ### Claim 17 (depends on 10) The one or more non-transitory storage media of claim 10, wherein the calibrating of the plurality of ground based plot images comprises correcting one or more colors depicted in the plurality of ground based plot images. ### Claim 18 (depends on 10) The one or more non-transitory storage media of claim 10, wherein the plot anomaly map for the agricultural plot comprises one or more specific anomaly maps, each specific anomaly map depicting a specific anomaly identified for the agricultural plot.