# US 11,991,940 B2 · Method for autonomous detection of crop location based on tool depth and location

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,991,940 B2
- Document ID: US11991940B2
- Status: Granted
- Kind: B2
- Application number: 17/074,332
- Filed: 2020-10-19
- Published: 2021-02-04
- Granted: 2024-05-28
- Earliest claimed: 2018-08-13
- Assignees: FarmWise Labs, Inc.
- Applicants: FarmWise Labs, Inc.
- Inventors: Arthur Flajolet; Eric Stahl-David; Sébastien Boyer; Thomas Palomares
- Web page: https://patentgrep.com/patent/US11991940B2

## CPC

- A01B 39/18: for weeding
- A01B 63/1112: {using a non-tactile ground distance measurement, e.g. using reflection of waves}
- A01B 69/001: {Steering by means of optical assistance, e.g. television cameras (steering devices for road marking vehicles E01C23/163)}
- A01B 69/004: {automatic}
- A01B 79/005: {Precision agriculture}
- G05D 1/0094: {involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target}
- G05D 1/0219: {ensuring the processing of the whole working surface}
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras
- G05D 1/648: Performing a task within a working area or space, e.g. cleaning
- G05D 1/689: Pointing payloads towards fixed or moving targets (positioning towed, pushed or suspended implements G05D1/672)
- G06T 7/579: from motion
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/188: {Vegetation}
- G06V 20/64: Three-dimensional [3D] objects
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- A01B: SOIL WORKING IN AGRICULTURE OR FORESTRY; PARTS, DETAILS, OR ACCESSORIES OF AGRICULTURAL MACHINES OR IMPLEMENTS, IN GENERAL (making or covering furrows or holes for sowing, planting or manuring A01C5/00; mowers convertible to soil working apparatus or capable of soil working A01D42/04; mowers combined with soil working implements A01D43/12; soil working for engineering purposes E01, E02, E21)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for detecting real lateral locations of target plants includes: recording an image of a ground area at a camera; detecting a target plant in the image; accessing a lateral pixel location of the target plant in the image; for each tool module in a set of tool modules arranged behind the camera and in contact with a plant bed: recording an extension distance of the tool module; and recording a lateral position of the tool module relative to the camera; estimating a depth profile of the plant bed proximal the target plant based on the extension distance and the lateral position of each tool module; estimating a lateral location of the target plant based on the lateral pixel location of the target plant and the depth profile of the plant bed surface proximal the target plant; and driving a tool module to a lateral position aligned with the lateral location of the target plant.

## Claims

### Claim 1

A method comprising, at an autonomous machine:
capturing a first image of a ground area of a plant bed surface via a ground-facing camera arranged on the autonomous machine;
detecting a first target plant in the first image;
accessing a lateral pixel location of the first target plant in the first image;
via a depth sensor arranged on the autonomous machine, estimating a depth of a subregion of the ground area;
for each tool module in a set of tool modules in contact with the plant bed surface and arranged behind the ground-facing camera relative to a direction of forward motion of the autonomous machine:
capturing an extension distance of the tool module; and
capturing a lateral position of the tool module relative to the ground-facing camera;
estimating a surface profile of the plant bed surface based on the extension distance of each tool module in the set of tool modules and the lateral position of each tool module in the set of tool modules;
estimating a depth profile based on the surface profile and the depth of the subregion of the ground area, comprising:
identifying a coincident section of the surface profile laterally coincident with the subregion of the ground area; and
defining the depth profile as a shifted version of the surface profile comprising the coincident section with an average depth equal to the depth of the subregion of the ground area;
estimating a real lateral location of the first target plant relative to the autonomous machine based on the lateral pixel location of the first target plant and the depth profile of the plant bed surface; and
driving a first tool module in the set of tool modules to a lateral position laterally aligned with the real lateral location of the first target plant.

### Claim 2 (depends on 1)

The method of claim 1, wherein detecting the depth of the subregion of the ground area comprises:
at the depth sensor, capturing a depth image of the subregion of the ground area;
identifying a ground region of the depth image; and
calculating an average depth of the ground region based on the ground region of the depth image.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
identifying a plant-occupying region of the depth image; and
calculating a plant height based on the plant-occupying region of the depth image.

### Claim 4 (depends on 3)

The method of claim 3, wherein estimating the real lateral location of the first target plant relative to the ground-facing camera comprises:
accessing a projection in three-dimensional space relative to the ground-facing camera corresponding to the lateral pixel location of the first target plant;
subtracting the plant height from the depth profile to calculate an approximate depth of the first target plant; and
estimating the real lateral location along the projection, the real lateral location corresponding to the approximate depth of the first target plant.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
at the second time, capturing an extension distance of a first weeding module in the set of tool modules; and
estimating a first depth of blades of the first weeding module based on a difference between the depth of the subregion of the ground area and the extension distance of the first weeding module.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
receiving a set depth for the blades of the first weeding module; and
actuating a suspension of the first weeding module according to feedback control based on the first depth of the blades of the first weeding module and the set depth of the blades of the first weeding module.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
estimating a maturity of the first target plant based on the first image; and
modifying the set depth for the blades of the first weeding module based on the maturity of the first target plant.

### Claim 8 (depends on 5)

The method of claim 5, further comprising:
in response to the difference between the depth of the subregion of the ground area and the extension distance of the first weeding module exceeding a threshold difference:
identifying malfunction of the depth sensor;
estimating a second depth profile of the plant bed surface proximal the first target plant based on the first extension distance of each tool module in the set of tool modules and the first lateral position of each tool module in the set of tool modules; and
updating the real lateral location of the first target plant relative to the ground-facing camera based on the lateral pixel location of the first target plant and the second depth profile.

### Claim 9 (depends on 8)

The method of claim 8, wherein estimating the second depth profile comprises:
for each tool module in the set of tool modules, defining a contact point of the tool module at an intersection of the first extension distance of the tool module and the first lateral position of the tool module to generate a set of contact points; and
interpolating between the set of contact points to generate the second depth profile.

### Claim 10

A method comprising, at an autonomous machine:
capturing a first image of a ground area of a plant bed surface via a ground-facing camera arranged on the autonomous machine;
detecting a first target plant in the first image;
accessing a lateral pixel location of the first target plant in the first image;
via a depth sensor arranged on the autonomous machine, estimating a depth of a subregion of the ground area, comprising:
at the depth sensor, capturing a depth image of the subregion of the ground area;
identifying a ground region of the depth image; and
calculating an average depth of the ground region based on the ground region of the depth image;
for each tool module in a set of tool modules in contact with the plant bed surface and arranged behind the ground-facing camera relative to a direction of forward motion of the autonomous machine:
capturing an extension distance of the tool module; and
capturing a lateral position of the tool module relative to the ground-facing camera;
estimating a surface profile of the plant bed surface based on the extension distance of each tool module in the set of tool modules and the lateral position of each tool module in the set of tool modules;
estimating a depth profile based on the surface profile and the depth of the subregion of the ground area;
estimating a real lateral location of the first target plant relative to the autonomous machine based on the lateral pixel location of the first target plant and the depth profile of the plant bed surface; and
driving a first tool module in the set of tool modules to a lateral position laterally aligned with the real lateral location of the first target plant.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
identifying a plant-occupying region of the depth image; and
calculating a plant height based on the plant-occupying region of the depth image.

### Claim 12 (depends on 11)

The method of claim 11, wherein estimating the real lateral location of the first target plant relative to the ground-facing camera comprises:
accessing a projection in three-dimensional space relative to the ground-facing camera corresponding to the lateral pixel location of the first target plant;
subtracting the plant height from the depth profile to calculate an approximate depth of the first target plant; and
estimating the real lateral location along the projection, the real lateral location corresponding to the approximate depth of the first target plant.

### Claim 13

A method comprising, at an autonomous machine:
capturing a first image of a ground area of a plant bed surface via a ground-facing camera arranged on the autonomous machine;
detecting a first target plant in the first image;
accessing a lateral pixel location of the first target plant in the first image;
via a depth sensor arranged on the autonomous machine, estimating a depth of a subregion of the ground area;
for each tool module in a set of tool modules in contact with the plant bed surface and arranged behind the ground-facing camera relative to a direction of forward motion of the autonomous machine:
capturing an extension distance of the tool module; and
capturing a lateral position of the tool module relative to the ground-facing camera;
estimating a surface profile of the plant bed surface based on the extension distance of each tool module in the set of tool modules and the lateral position of each tool module in the set of tool modules;
estimating a depth profile based on the surface profile and the depth of the subregion of the ground area;
estimating a real lateral location of the first target plant relative to the autonomous machine based on the lateral pixel location of the first target plant and the depth profile of the plant bed surface; and
driving a first tool module in the set of tool modules to a lateral position laterally aligned with the real lateral location of the first target plant;
at a second time, capturing an extension distance of a first weeding module in the set of tool modules; and
estimating a first depth of blades of the first weeding module based on a difference between the depth of the subregion of the ground area and the extension distance of the first weeding module.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
receiving a set depth for the blades of the first weeding module; and
actuating a suspension of the first weeding module according to feedback control based on the first depth of the blades of the first weeding module and the set depth of the blades of the first weeding module.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
estimating a maturity of the first target plant based on the first image; and
modifying the set depth for the blades of the first weeding module based on the maturity of the first target plant.

### Claim 16 (depends on 13)

The method of claim 13, further comprising:
in response to the difference between the depth of the subregion of the ground area and the extension distance of the first weeding module exceeding a threshold difference:
identifying malfunction of the depth sensor;
estimating a second depth profile of the plant bed surface proximal the first target plant based on the first extension distance of each tool module in the set of tool modules and the first lateral position of each tool module in the set of tool modules; and
updating the real lateral location of the first target plant relative to the ground-facing camera based on the lateral pixel location of the first target plant and the second depth profile.

### Claim 17 (depends on 16)

The method of claim 16, wherein estimating the second depth profile comprises:
for each tool module in the set of tool modules, defining a contact point of the tool module at an intersection of the first extension distance of the tool module and the first lateral position of the tool module to generate a set of contact points; and
interpolating between the set of contact points to generate the second depth profile.