# US 11,998,000 B2 · Camera system visualization and control for an agricultural spraying machine > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,998,000 B2 - Document ID: US11998000B2 - Status: Granted - Kind: B2 - Application number: 17/188,320 - Filed: 2021-03-01 - Published: 2022-05-05 - Granted: 2024-06-04 - Earliest claimed: 2020-10-30 - Assignees: Deere & Company - Applicants: Deere & Company - Inventors: Michelle Kwak; Barbra Cavanaugh; Tarik Loukili; Scott M. Buse - Web page: https://patentgrep.com/patent/US11998000B2 ## CPC - A01M 7/0089: {Regulating or controlling systems (the delivery being related to the movement of a vehicle B05B9/06)} - G06T 7/0004: {Industrial image inspection} - A01C 23/007: {Metering or regulating systems} - A01C 23/047: {Spraying of liquid fertilisers} - A01M 7/0042: {Field sprayers, e.g. self-propelled, drawn or tractor-mounted} - B05B 12/122: {responsive to presence or shape of target (B05B12/124 takes precedence)} - B05B 12/16: for controlling the spray area (B05B3/00 takes precedence) - B60R 1/00: Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles - G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06V 10/141: Control of illumination - G06V 10/143: Sensing or illuminating at different wavelengths - G06V 10/235: {based on user input or interaction} - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 20/188: {Vegetation} - G06V 20/194: {using hyperspectral data, i.e. more or other wavelengths than RGB} - H04N 5/265: Mixing - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - B05B 1/20: Perforated pipes or troughs, e.g. spray booms; Outlet elements therefor - B60R 2300/105: using multiple cameras - B60R 2300/30: characterised by the type of image processing - B60R 2300/303: using joined images, e.g. multiple camera images - B60R 2300/80: characterised by the intended use of the viewing arrangement - G06F 18/24: Classification techniques - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2207/10012: Stereo images - G06T 2207/20104: Interactive definition of region of interest [ROI] - G06T 2207/30188: Vegetation; Agriculture - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06V 20/68: Food, e.g. fruit or vegetables - A01M: CATCHING, TRAPPING OR SCARING OF ANIMALS (appliances for catching swarms or drone-catching A01K57/00; fishing A01K69/00 - {A01K99/00}; biocides, pest repellants or attractants A01N); APPARATUS FOR THE DESTRUCTION OF NOXIOUS ANIMALS OR NOXIOUS PLANTS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - A01C: PLANTING; SOWING; FERTILISING (parts, details or accessories of agricultural machines or implements, in general A01B51/00 - A01B75/00) - B05B: SPRAYING APPARATUS; ATOMISING APPARATUS; NOZZLES (spray-mixers with nozzles B01F25/72; processes for applying liquids or other fluent materials to surfaces by spraying B05D) - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An agricultural machine includes a product application system comprising a plurality of actuatable applicator mechanisms, each actuatable applicator mechanism configured to apply a substance on an agricultural surface, an imaging system configured to obtain image data indicative of an image of plant matter on the agricultural surface, and a control system configured to determine a characteristic of the plant matter in an image region defined within the image, control one or more of the actuatable applicator mechanisms to apply the substance on the agricultural surface based on the determined characteristic, and generate a user interface display that displays the image and includes a display element that visually represents the image region within the image. ## Claims ### Claim 1 An agricultural machine comprising: a plurality of applicators, each applicator of the plurality of applicators being actuatable to apply a substance on an agricultural surface; one or more imaging sensors configured to capture images of plant matter on the agricultural surface; and a control system configured to: map each applicator, in the plurality of applicators, to a corresponding region of interest (ROI) within a field of view of at least one imaging sensor of the one or more imaging sensors; for each respective applicator, in the plurality of applicators, determine a characteristic of the plant matter based on image processing on the corresponding region of interest mapped to the respective applicator, and control the respective applicator to apply the substance on the agricultural surface based on the characteristic of the plant matter in the corresponding region of interest; and generate a user interface display configured to display the images and, for a selected applicator in the plurality of applicators, a region of interest indicator that identifies the corresponding region of interest that is mapped to the selected applicator. ### Claim 2 (depends on 1) The agricultural machine of claim 1, wherein the agricultural machine comprises an agricultural sprayer and the plurality of applicators comprise a plurality of controllable spray nozzles. ### Claim 3 (depends on 2) The agricultural machine of claim 2, wherein the control system is configured to control each spray nozzle, of the plurality of controllable spray nozzles, to spray the substance in a dispersal area corresponding to the plant matter in the corresponding region of interest that is mapped to the spray nozzle. ### Claim 4 (depends on 1) The agricultural machine of claim 1, wherein the one or more imaging sensors comprises a plurality of cameras, and further comprising at least one image processor configured to perform the image processing on the corresponding region of interest. ### Claim 5 (depends on 4) The agricultural machine of claim 4, wherein the user interface display includes a video feed from one or more camera of the plurality of the camera. ### Claim 6 (depends on 4) The agricultural machine of claim 4, wherein the user interface display includes a ROI modification user input mechanism, and the control system is configured to: receive an indication of actuation of the ROI modification user input mechanism; and modify the ROI based on the indication of actuation of the ROI modification user input mechanism. ### Claim 7 (depends on 1) The agricultural machine of claim 1, wherein the one or more imaging sensors comprise a first camera configured to obtain a first image and a second camera configured to obtain a second image, and wherein the control system is configured to: generate a stitched image based on the first and second images, and display the stitched image in the user interface display with a first display element that visually identifies a first ROI associated with the first camera and a second display element that visually identifies a second ROI associated with the second camera. ### Claim 8 (depends on 7) The agricultural machine of claim 7, wherein the control system is configured to: control a first set of applicators, of the plurality of applicators, based on a characteristic of plant matter in the first ROI; and control a second set of applicators, of the plurality of applicators, based on a characteristic of plant matter in the second ROI. ### Claim 9 (depends on 1) The agricultural machine of claim 1, wherein the control system is configured to generate the characteristic based on a sensitivity setting associated with the one or more imaging sensors, wherein the user interface display includes a sensitivity adjustment user input mechanism configured to adjust the sensitivity setting. ### Claim 10 (depends on 9) The agricultural machine of claim 9, wherein the user interface display includes a sensitivity setting display element that indicates the sensitivity setting. ### Claim 11 (depends on 4) The agricultural machine of claim 4, wherein the control system is configured to generate image processing diagnostic data, and to generate the user interface display to display a indication of the image processing diagnostic data. ### Claim 12 (depends on 11) The agricultural machine of claim 11, wherein the image processing diagnostic data comprises diagnostic data representing an operational status of at least one of: one or more cameras of the plurality of cameras, or the image processor. ### Claim 13 (depends on 12) The agricultural machine of claim 12, wherein the operational status comprises a fault condition. ### Claim 14 (depends on 1) The agricultural machine of claim 1, wherein the control system is configured to generate a detection confidence metric based on the image processing and to automatically switch control of the plurality of applicators between a precision application mode and a broadcast application mode based on detection confidence metric. ### Claim 15 (depends on 14) The agricultural machine of claim 14, wherein the user interface display includes a mode display map that identities an area of the agricultural surface on which the precision application mode was utilized and an area of the agricultural surface on which the broadcast application mode was utilized. ### Claim 16 An agricultural machine comprising: a plurality of applicators, each applicator of the plurality of applicators being actuatable to apply a substance on an agricultural surface; one or more imaging sensors configured to capture images of plant matter on the agricultural surface; and a control system configured to: map each applicator, in the plurality of applicators, to a corresponding region of interest within a field of view of at least one imaging sensor of the one or more imaging sensors; for each respective applicator, in the plurality of applicators, determine a characteristic of the plant matter based on image processing on the corresponding region of interest mapped to the respective applicator, generate image processing diagnostic data related to the image processing, control the respective applicator to apply the substance on the agricultural surface based on the characteristic of the plant matter in the corresponding region of interest; and generate a user interface display configured to display an indication of the image processing diagnostic data. ### Claim 17 (depends on 16) The agricultural machine of claim 16, wherein the one or more imaging sensors comprises a plurality of cameras, and the image processing diagnostic data comprises diagnostic data representing an operational status of at least one of: one or more cameras of the plurality of cameras, or an image processor that performs the image processing. ### Claim 18 (depends on 17) The agricultural machine of claim 17, wherein the operational status comprises a fault condition. ### Claim 19 A method performed by an agricultural system, the method comprising: mapping each applicator, in a plurality of applicators, configured to apply a substance on an agricultural surface to a corresponding region of interest within an imaging sensor field of view; for each respective applicator in the plurality of applicators, determining a characteristic of plant matter based on image processing on the corresponding region of interest mapped to the respective applicator, and controlling the respective applicator to apply the substance on the agricultural surface based on the characteristic of the plant matter in the corresponding region of interest; and generate a user interface display configured to display, for a selected applicator in the plurality of applicators, a region of interest indicator that identifies the corresponding region of interest that is mapped to the selected applicator. ### Claim 20 (depends on 19) The method of claim 19, wherein the plurality of applicators comprises a plurality of controllable spray nozzles, and controlling the respective applicator comprises controlling each spray nozzle, of the plurality of controllable spray nozzles, to spray the substance in a dispersal area corresponding to the plant matter in the corresponding region of interest that is mapped to the spray nozzle.