# US 11,076,589 B1 · Autonomous agricultural treatment system using map based targeting of agricultural objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,076,589 B1
- Document ID: US11076589B1
- Status: Granted
- Kind: B1
- Application number: 17/073,236
- Filed: 2020-10-16
- Granted: 2021-08-03
- Earliest claimed: 2020-10-16
- Assignees: Verdant Robotics, Inc.
- Applicants: Verdant Robotics, Inc.
- Inventors: Gabriel Thurston Sibley; Lorenzo Ibarria; Curtis Dale Garner; Patrick Christopher Leger; Andre Robert Daniel Michelin; John Phillip Hurliman, II; Wisit Jirattigalochote; Hasan Tafish
- Web page: https://patentgrep.com/patent/US11076589B1

## CPC

- A01M 7/0089: {Regulating or controlling systems (the delivery being related to the movement of a vehicle B05B9/06)}
- G06V 10/764: using classification, e.g. of video objects
- G06F 18/2413: based on distances to training or reference patterns
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- A01G 7/06: Treatment of growing trees or plants, e.g. for preventing decay of wood, for tingeing flowers or wood, for prolonging the life of plants
- A01M 7/0042: {Field sprayers, e.g. self-propelled, drawn or tractor-mounted}
- A01M 21/043: {by chemicals}
- G06V 20/188: {Vegetation}
- A01M: CATCHING, TRAPPING OR SCARING OF ANIMALS (appliances for catching swarms or drone-catching A01K57/00; fishing A01K69/00 - {A01K99/00}; biocides, pest repellants or attractants A01N); APPARATUS FOR THE DESTRUCTION OF NOXIOUS ANIMALS OR NOXIOUS PLANTS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A01G: HORTICULTURE; CULTIVATION OF VEGETABLES, FLOWERS, RICE, FRUIT, VINES, HOPS OR SEAWEED; FORESTRY; WATERING (picking of fruits, vegetables, hops or the like A01D46/00; propagating unicellular algae C12N1/12)

## Abstract

Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to an agricultural observation and treatment system and method of operation. The agricultural treatment system may determine a first real-world geo-spatial location of the treatment system. The system can receive captured images depicting real-world agricultural objects of a geographic scene. The system can associate captured images with the determined geo-spatial location of the treatment system. The treatment system can identify, from a group of mapped and indexed images, images having a second real-word geo-spatial location that is proximate with the first real-world geo-spatial location. The treatment system can compare at least a portion of the identified images with at least a portion of the captured images. The treatment system can determine a target object and emit a fluid projectile at the target object using a treatment device.

## Claims

### Claim 1

A method performed by an agricultural treatment system comprising one or more processors comprising hardware, one or more image sensors, and a treatment unit, the one or more processors configured to perform operations comprising:
storing a group of images on a data storage device, the group of images depicting views of real-world agricultural objects, the one or more stored images having associated real-world geo-spatial locations, and the one or more stored images including pre-identified agricultural objects;
determining a first real-world geo-spatial location of the agricultural treatment system;
receiving one or more captured images depicting real-world agricultural objects of a geographic scene;
associating the one or more captured images with the determined geo-spatial location of the agricultural treatment system;
identifying, from the group of one or more stored images, one or more images having a second real-word geo-spatial location that is proximate with the first real-world geo-spatial location, wherein the one or more captured images are obtained at a time later than when the group of images are stored on the storage device;
comparing at least a portion of the identified images with the one or more captured images;
determining a target object for treatment based on the comparing of the one or more captured images with the portions of the one or more identified images, the target object being determined where a portion of a captured image corresponds to a portion of an image from the identified one or more images; and
emitting a fluid projectile at the target object using the treatment unit.

### Claim 2 (depends on 1)

The method of claim 1, wherein the group of images comprise key frame images and sub-key frame images, the key frame images depicting agriculture objects of the geographical scene, and the sub-key frame images depicting a portion of a key frame image.

### Claim 3 (depends on 2)

The method of claim 2, wherein comparing at least a portion of the identified images comprises:
comparing pixels of the sub-key frame image to a portion of one of the captured images;
determining a confidence level of whether the sub-key frame image matches the portion of the captured image; and
identifying a match where the determined confidence level meets or exceeds a threshold value.

### Claim 4 (depends on 2)

The method of claim 2, wherein the key frame images were previously obtained by the one or more image sensors of the agricultural treatment system, and the captured images are obtained by the same one or more image sensors at a time subsequent to when the key frame images were taken.

### Claim 5 (depends on 1)

The method of claim 1, wherein comparing at least a portion of the identified images comprises:
identifying a landmark object in a captured image; and
determining a portion of the landmark object in the capture image matches a portion of an image from the group of images.

### Claim 6 (depends on 5)

The method of claim 5, wherein the target objects are real-world objects that are intended to be treated with the fluid projectile, and the landmark objects are real-world objects that aid in determining the location of a target object.

### Claim 7 (depends on 1)

The method of claim 1, wherein the one or more captured images are obtained by one or more image sensors and the real-world agricultural objects are located within a distance of 0.5 foot to 100 feet from the camera capturing the images.

### Claim 8

A non-transitory computer storage medium comprising instructions that when executed by one or more process included of an agricultural treatment system, cause the agricultural treatment system to perform the operations comprising:
storing a group of images on a data storage device, the group of images depicting views of real-world agricultural objects, the one or more stored images having associated real-world geo-spatial locations, and the one or more stored images including pre-identified agricultural objects;
determining a first real-world geo-spatial location of the agricultural treatment system;
receiving one or more captured images depicting real-world agricultural objects of a geographic scene;
associating the one or more captured images with the determined geo-spatial location of the agricultural treatment system;
identifying, from the group of one or more stored images, one or more images having a second real-word geo-spatial location that is proximate with the first real-world geo-spatial location, wherein the one or more captured images are obtained at a time later than when the group of images are stored on the storage device;
comparing at least a portion of the identified images with the one or more captured images;
determining a target object for treatment based on the comparing of the one or more captured images with the portions of the one or more identified images, the target object being determined where a portion of a captured image corresponds with a portion of an image from the identified one or more images; and
emitting a fluid projectile at the target object using a treatment unit of the agricultural treatment system.

### Claim 9 (depends on 8)

The non-transitory computer storage medium of claim 8, wherein the group of images comprise key frame images and sub-key frame images, the key frame images depicting agriculture objects of the geographical scene, and the sub-key frame images depicting a portion of a key frame image.

### Claim 10 (depends on 9)

The non-transitory computer storage medium of claim 9, wherein comparing at least a portion of the identified images comprises:
comparing pixels of the sub-key frame image to a portion of one of the captured images;
determining a confidence level of whether the sub-key frame image matches the portion of the captured image; and
identifying a match where the determined confidence level meets or exceeds a predetermined confidence level threshold value.

### Claim 11 (depends on 9)

The non-transitory computer storage medium of claim 9, wherein the key frame images were previously obtained by the one or more image sensors of the agricultural treatment system, and the captured images are obtained by the same one or more image sensors at a time subsequent to when the key frame images were taken.

### Claim 12 (depends on 8)

The non-transitory computer storage medium of claim 8, wherein comparing at least a portion of the identified images comprises:
identifying a landmark object in a captured image; and
determining a portion of the landmark object in the capture image matches a portion of an image from the group of images.

### Claim 13 (depends on 12)

The non-transitory computer storage medium of claim 12, wherein the target objects are real-world objects that are intended to be treated with the fluid projectile, and the landmark objects are real-world objects that aid in determining the location of a target object.

### Claim 14 (depends on 8)

The non-transitory computer storage medium of claim 8, wherein the one or more captured images are obtained by one or more image sensors and the real-world agricultural objects are located within a distance of 0.5 foot to 100 feet from the camera capturing the images.

### Claim 15

An agricultural treatment system, the system comprising:
one or more image sensors for obtaining imagery;
a treatment unit mounted to a gimble, the treatment unit configured to emit a fluid projectile;
one or more tanks, wherein the tanks provide a container for a fluid;
an electronically controlled pump fluidly connected to the one or more tanks and the treatment unit; and
one or more processors, memory, and a non-transitory computer readable storage medium comprising hardware, the one or more processors configured to control operation of the treatment unit, and control actuation of the electronically controlled pump and orientation of the treatment unit for emitting a fluid projectile via the treatment unit, wherein the one or more processors are configured to perform operations comprising:
storing a group of images on a data storage device, the group of images depicting views of real-world agricultural objects, the one or more stored images having associated real-world geo-spatial locations, and the one or more stored images including pre-identified agricultural objects;
determining a first real-world geo-spatial location of the agricultural treatment system;
receiving one or more captured images depicting real-world agricultural objects of a geographic scene;
associating the one or more captured images with the determined geo-spatial location of the agricultural treatment system;
identifying, from the group of one or more stored images, one or more images having a second real-word geo-spatial location that is proximate with the first real-world geo-spatial location, wherein the one or more captured images are obtained at a time later than when the group of images are stored on the storage device;
comparing at least a portion of the identified images with the one or more captured images;
determining a target object for treatment based on the comparing of the one or more captured images with the portions of the one or more identified images, the target object being determined where a portion of a captured image corresponds with a portion of an image from the identified one or more images; and
emitting a fluid projectile at the target object using the treatment unit.

### Claim 16 (depends on 15)

The agricultural treatment system of claim 15, wherein the group of images comprise key frame images and sub-key frame images, the key frame images depicting agriculture objects of the geographical scene, and the sub-key frame images depicting a portion of a key frame image.

### Claim 17 (depends on 16)

The agricultural treatment system of claim 16, wherein comparing at least a portion of the identified images comprises:
comparing pixels of the sub-key frame image to a portion of one of the captured images;
determining a confidence level of whether the sub-key frame image matches the portion of the captured image; and
identifying a match where the determined confidence level meets or exceeds a predetermined confidence level threshold value.

### Claim 18 (depends on 16)

The agricultural treatment system of claim 16, wherein the key frame images were previously obtained by the one or more image sensors of the agricultural treatment system, and the captured images are obtained by the same one or more image sensors at a time subsequent to when the key frame images were taken.

### Claim 19 (depends on 15)

The agricultural treatment system of claim 15, wherein comparing at least a portion of the identified images comprises:
identifying a landmark object in a captured image; and
determining a portion of the landmark object in the capture image matches a portion of an image from the group of images.

### Claim 20 (depends on 15)

The agricultural treatment system of claim 15, wherein the one or more captured images are obtained by one or more image sensors and the real-world agricultural objects are located within a distance of 0.5 foot to 100 feet from the camera capturing the images.

### Claim 21 (depends on 1)

The method of claim 1, wherein the group of images depict views of one or more unique agricultural objects in the real-world.

### Claim 22 (depends on 1)

The method of claim 1, further comprising:
determining whether the target object needs treatment; and
emitting the fluid projectile where the target object has been determined to need treatment.

### Claim 23 (depends on 1)

The method of claim 1, further comprising:
wherein the group of the one or more stored images includes associated information indicating a stage of growth of an agricultural object; and
dispensing a predetermined amount of the fluid based on the stage of growth of the target object.

### Claim 24 (depends on 1)

The method of claim 1, wherein the group of one or more stored images includes associated information indicating an object type of an agricultural object, and wherein emitting the fluid projectile is based on the object type of the target object.

### Claim 25 (depends on 1)

The method of claim 1, wherein the group of one or more stored images includes associated information indicating an object type of an agricultural object; and
wherein the emitting the fluid projectile is based on the treatment history of the target object.

### Claim 26 (depends on 15)

The agricultural treatment system of claim 15, wherein the group of images depict views of one or more unique agricultural objects in the real-world.

### Claim 27 (depends on 15)

The agricultural treatment system of claim 15, further comprising:
determining whether the target object needs treatment; and
emitting the fluid projectile where the target object has been determined to need treatment.

### Claim 28 (depends on 15)

The agricultural treatment system of claim 15, further comprising:
wherein the group of the one or more stored images includes associated information indicating a stage of growth of an agricultural object; and
dispensing a predetermined amount of the fluid based on the stage of growth of the target object.

### Claim 29 (depends on 15)

The agricultural treatment system of claim 15, wherein the group of one or more stored images includes associated information indicating an object type of an agricultural object, and wherein emitting the fluid projectile is based on the object type of the target object.

### Claim 30 (depends on 15)

The agricultural treatment system of claim 15, wherein the group of one or more stored images includes associated information indicating an object type of an agricultural object; and wherein the emitting the fluid projectile is based on the treatment history of the target object.