# US 2022/0044044 A1 · Plant identification using heterogenous multi-spectral stereo imaging > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0044044 A1 - Document ID: US20220044044A1 - Status: Published application - Kind: A1 - Application number: 17/396,454 - Filed: 2021-08-06 - Published: 2022-02-10 - Earliest claimed: 2020-08-07 - Assignees: Blue River Technology Inc. - Applicants: Blue River Technology Inc. - Inventors: John William Peake; James Patrick Ostrowski - Web page: https://patentgrep.com/patent/US20220044044A1 ## CPC - G06V 10/143: Sensing or illuminating at different wavelengths - A01B 79/005: {Precision agriculture} - A01C 21/005: {Following a specific plan, e.g. pattern} - A01M 7/0089: {Regulating or controlling systems (the delivery being related to the movement of a vehicle B05B9/06)} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/25: Fusion techniques - G06T 7/593: from stereo images - G06T 7/90: Determination of colour characteristics - G06V 10/56: relating to colour - G06V 10/803: {of input or preprocessed data} - G06V 10/82: using neural networks - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/60: Type of objects - G06V 20/647: {by matching two-dimensional images to three-dimensional objects} - H04N 13/25: using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor - H04N 13/257: Colour aspects - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - H04N 23/11: for generating image signals from visible and infrared light wavelengths - H04N 25/131: including elements passing infrared wavelengths - H04N 25/134: {based on three different wavelength filter elements} - G06T 2207/10024: Color image - G06T 2207/10036: Multispectral image; Hyperspectral image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30188: Vegetation; Agriculture - G06V 10/58: relating to hyperspectral data - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - H04N 2013/0092: {Image segmentation from stereoscopic image signals} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - A01B: SOIL WORKING IN AGRICULTURE OR FORESTRY; PARTS, DETAILS, OR ACCESSORIES OF AGRICULTURAL MACHINES OR IMPLEMENTS, IN GENERAL (making or covering furrows or holes for sowing, planting or manuring A01C5/00; mowers convertible to soil working apparatus or capable of soil working A01D42/04; mowers combined with soil working implements A01D43/12; soil working for engineering purposes E01, E02, E21) - A01C: PLANTING; SOWING; FERTILISING (parts, details or accessories of agricultural machines or implements, in general A01B51/00 - A01B75/00) - A01M: CATCHING, TRAPPING OR SCARING OF ANIMALS (appliances for catching swarms or drone-catching A01K57/00; fishing A01K69/00 - {A01K99/00}; biocides, pest repellants or attractants A01N); APPARATUS FOR THE DESTRUCTION OF NOXIOUS ANIMALS OR NOXIOUS PLANTS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A farming machine identifies and treats a plant as the farming machine travels through a field. The farming machine includes a pair of image sensors for capturing images of a plant. The image sensors are different, and their output images are used to generate a depth map to improve the plant identification process. A control system identifies a plant using the depth map. The control system captures images, identifies a plant, and actuates a treatment mechanism in real time. ## Claims ### Claim 1 A method for taking a farming action in a field, comprising: accessing a color image corresponding to a first field of view and comprising a plurality of color pixels comprising (i) one or more color pixels representing a plant in the field, and (ii) one or more color pixels for a particular spectral band; accessing a multispectral image corresponding to a second field of view and comprising a plurality of multispectral pixels comprising (i) one or more multispectral pixels representing the plant in the field, and (ii) one or more multispectral pixels for the particular spectral band; generating a depth map comprising a plurality of depth pixels representing stereo depth information, the plurality of depth pixels including one or more depth pixels representing the plant in the field, by: identifying an overlapping region between the first field of view and the second field of view; generating stereo depth information from the multispectral pixels and color pixels for the particular spectral band that correspond to the overlapping region; and encoding the stereo depth information into the depth pixels of the depth map; identifying the plant in the depth pixels of the depth map; and taking a farming action in the field based on the identified plant in the depth pixels of the depth map. ### Claim 2 (depends on 1) The method of claim 1, wherein a first image sensor is a digital camera with red, green, and blue color channels, and the color pixels include red, green, and blue color pixels. ### Claim 3 (depends on 1) The method of claim 1, wherein a second image sensor is a digital camera with red, green, and near-multispectral color channels, and the multispectral pixels include, red, green, and near-multispectral color pixels. ### Claim 4 (depends on 1) The method of claim 1, wherein identifying the plant further comprises applying a plant identification model to at least one of the accessed color image, the accessed multispectral image, and the depth map. ### Claim 5 (depends on 1) The method of claim 1, wherein the first field of view and the second field of view at least partially overlap on a plane of the field. ### Claim 6 (depends on 1) The method of claim 1, wherein identifying the plant further comprises: determining at least one of a normalized difference vegetation index calculation and an enhanced vegetation index calculation using the multispectral image; and identifying the plant based on the at least one of the determined normalized difference vegetation index calculation and the determined enhanced vegetation index calculation. ### Claim 7 (depends on 1) The method of claim 1, wherein the particular spectral band is red or green. ### Claim 8 (depends on 1) The method of claim 1, wherein generating depth pixels comprises: estimating a first set of blue pixels for the multispectral image; and combining the first set of blue pixels with a second set of blue pixels from the color image. ### Claim 9 (depends on 8) The method of claim 8, wherein estimating the blue channel for the multispectral image comprises: inputting the multispectral image into a machine learned neural network, the machine learned neural network configured to estimate the first set of blue pixels for the multispectral image based on the color image and the multispectral image; wherein the machine learned neural network is trained using overlapping regions of a plurality of previously accessed color images and multispectral images. ### Claim 10 (depends on 1) The method of claim 1, wherein the overlapping region is determined based on at least one of (i) a hardware configuration of the image acquisition system, (ii) a configuration of the farming machine, and (iii) one or more image processing techniques comprising area-based image registration and feature-based image registration. ### Claim 11 (depends on 1) The method of claim 1, wherein identifying the plant further comprises: identifying one or more depth pixels having depth information a threshold distance from an indicative depth, the identified one or more depth pixels comprising pixels representing the plant. ### Claim 12 (depends on 11) The method of claim 11, wherein the indicative depth represents depth corresponding to any of: at least one of other plants in the field, a variation in depth over a time period, an expected depth of one or more substrates for the plant, and an expected depth of other plants having a same type as the plant. ### Claim 13 A farming machine comprising: an image acquisition system configured to capture a pair of images of a plant in a field as the farming machine moves through the field, wherein the image acquisition system comprises: a first image sensor configured to produce color images comprising a plurality of color pixels, the plurality of color pixels comprising (i) one or more color pixels representing the plant in the field, and (ii) one or more color pixels for a particular spectral band, and a second image sensor configured to produce multispectral images comprising a plurality of multispectral pixels, the plurality of multispectral pixels comprising (i) one or more multispectral pixels representing the plant in the field, and (ii) one or more multispectral pixels for the particular spectral band; and a controller for: generating a depth map comprising a plurality of depth pixels representing stereo depth information, the plurality of depth pixels including one or more depth pixels representing the plant in the field, the generating comprising: identifying an overlapping region between the first field of view and the second field of view; generating stereo depth information from the multispectral pixels and color pixels for the particular spectral band that correspond to the overlapping region; and encoding the stereo depth information into the depth pixels of the depth map; identifying the plant in the depth pixels of the depth map; and taking a farming action in the field based on the identified plant in the depth pixels of the depth map. ### Claim 14 (depends on 13) The farming machine of claim 13, wherein identifying the plant further comprises: determining at least one of a normalized difference vegetation index calculation and an enhanced vegetation index calculation using the multispectral image; and identifying the plant based on the determined at least one of the normalized difference vegetation index calculation and the enhanced vegetation index calculation. ### Claim 15 (depends on 13) The farming machine of claim 13, wherein the particular spectral band is red or green. ### Claim 16 (depends on 13) The farming machine of claim 13, wherein generating depth pixels comprises: estimating a first set of blue pixels for the multispectral image; and combining the first set of blue pixels with a second set of blue pixels from the color image. ### Claim 17 (depends on 16) The farming machine of claim 16, wherein estimating the blue channel for the multispectral image comprises: maintaining a machine learned neural network, the machine learned neural network trained upon overlapping color images and multispectral images; and applying the maintained machine learned neural network to the multispectral image. ### Claim 18 (depends on 13) The farming machine of claim 13, wherein determining the overlapping region is based on at least one of a hardware configuration and one or more image processing techniques comprising area-based image registration and feature-based image registration. ### Claim 19 (depends on 13) The farming machine of claim 13, wherein identifying the plant further comprises: identifying one or more depth pixels having depth information a threshold distance from an indicative depth, the identified one or more depth pixels comprising pixels representing the plant. ### Claim 20 (depends on 19) The method of claim 19, wherein the indicative depth represents depth corresponding to any of: at least one of other plants in the field, a variation in depth over a time period, an expected depth of one or more substrates for the plant, and an expected depth of other plants having a same type as the plant.