# US 2022/0264801 A1 · Method of controlling a robotic harvesting device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0264801 A1
- Document ID: US20220264801A1
- Status: Published application
- Kind: A1
- Application number: 17/631,785
- Filed: 2020-07-27
- Published: 2022-08-25
- Earliest claimed: 2019-07-31
- Applicants: Noronn AS
- Inventors: Ya Xiong; Yuanyue Ge; Pål Johan From
- Web page: https://patentgrep.com/patent/US20220264801A1

## CPC

- A01D 46/30: Robotic devices for individually picking crops
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/30188: Vegetation; Agriculture
- G06T 2207/30261: Obstacle
- A01D: HARVESTING; MOWING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method of controlling a robotic harvesting device to pick a target piece of fruit from a plant includes determining a position of the target piece of fruit, determining a position of one or more obstacles relative to the target piece of fruit , and plotting a path for the robotic harvesting device to be moved along from a starting position to a picking position. The path is plotted such that the movement of the robotic harvesting device along the path serves to move at least some of the obstacles relative to the target piece of fruit so that the obstacles do not impede the picking of the target piece of fruit .

## Claims

### Claim 1

A method of controlling a robotic harvesting device to pick a target piece of fruit from a plant, the method comprising:
determining a position of the target piece of fruit to be harvested from the plant;
determining a position of one or more obstacles relative to the target piece of fruit; and
plotting a path for the robotic harvesting device to be moved along from a starting position to a picking position, wherein the path is plotted such that the movement of the robotic harvesting device along the path serves to move at least some of the one or more obstacles relative to the target piece of fruit such that the one or more obstacles do not impede the picking of the target piece of fruit by the robotic harvesting device.

### Claim at position 2

-4. (canceled)

### Claim 5 (depends on 1)

The method as claimed in claim 1, wherein:
said determining of a position of the target piece of fruit and/or a position of the one or more obstacles comprises capturing image data of the target piece of fruit and/or the one or more obstacles to generate an image or representation of the target piece of fruit and its surroundings;
the image or representation of the target piece of fruit is divided into a plurality of sub-volumes corresponding to a plurality of locations around the target piece of fruit, the plurality of sub-volumes comprising a plurality of columns and/or a plurality of rows; and
the plurality of rows defines one or more of: a sub-volume of the image or representation that is located above the target piece of fruit, a sub-volume of the image or representation that is located below the target piece of fruit and a sub-volume of the image or representation that is located around the target piece of fruit.

### Claim 6 (depends on 5)

The method as claimed in claim 5, further comprising determining which of the plurality of sub-volumes contains an obstacle.

### Claim 7

(canceled)

### Claim 8 (depends on 5)

The method as claimed in claim 5, wherein each sub-volume of the plurality of sub-volumes is assigned one sub-vector of a plurality of sub-vectors.

### Claim 9 (depends on 8)

The method as claimed in claim 8, wherein each sub-vector of the plurality of sub-vectors is directed from a center of the corresponding sub-volume to a center of the target piece of fruit.

### Claim 10 (depends on 1)

The method as claimed in claim 1, wherein plotting the path for the robotic harvesting device to be moved along comprises determining a set of one or more path vectors.

### Claim 11 (depends on 10)

The method as claimed in claim 10, wherein the one or more path vectors are determined using the determined position of the one or more obstacles.

### Claim 12 (depends on 11)

The method as claimed in claim 11, wherein the directions of each path vector of the one or more path vectors is calculated from one or more of a plurality of subsets of sub-vectors that correspond to one or more of a plurality of subsets of sub-volumes.

### Claim 13 (depends on 12)

The method as claimed in claim 12, wherein each subset of sub-volumes is determined using the determined positions of the one or more obstacles within the image or representation.

### Claim 14 (depends on 12)

The method as claimed in claim 12, wherein each subset of sub-volumes comprises a plurality of adjacent sub-volumes all having the same characteristic representative of whether they are occupied or non-occupied by obstacles of the one or more obstacles.

### Claim 15 (depends on 14)

The method as claimed in claim 14, wherein a subset of sub-volumes comprises a largest group of adjacent sub-volumes that all have the same characteristic representative of whether they are occupied or non-occupied by obstacles of the one or more obstacles.

### Claim 16 (depends on 1)

The method as claimed in claim 1, wherein the path for the robotic harvesting device to be moved along comprises a plurality of sub-paths, optionally wherein each of the plurality of sub-paths is plotted substantially within a different region around the target piece of fruit.

### Claim 17 (depends on 16)

The method as claimed in claim 16, wherein at least one sub-path of the plurality of sub-paths comprises a single linear movement or an oscillating movement of the robotic harvesting device within a region below the target piece of fruit.

### Claim 18 (depends on 16)

The method as claimed in claim 16, wherein at least one sub-path of the plurality of sub-paths comprises a combined upwards and oscillating movement of the robotic harvesting device within a region proximal to the target piece of fruit.

### Claim 19 (depends on 16)

The method as claimed in claim 16, wherein at least one sub-path of the plurality of sub-paths comprises a dragging movement of the robotic harvesting device that moves the target piece of fruit from a first position immediately below at least one of the one or more obstacles to a second, higher, position that is not immediately below at least one of the one or more obstacles.

### Claim 20

(canceled)

### Claim 21 (depends on 1)

The method as claimed in claim 1, further comprising identifying whether the one or more obstacles are movable or immovable.

### Claim 22 (depends on 1)

The method as claimed in claim 1, further comprising moving the robotic harvesting device along the plotted path from the starting position to the picking position and controlling the robotic harvesting device to pick the target piece of fruit.

### Claim 23 (depends on 1)

The method as claimed in claim 1, wherein the starting position is determined by determining the largest opening angle between two obstacles, having no obstacles therebetween, and positioning the starting position at the midpoint of the opening angle.

### Claim 24 (depends on 1)

A non-transitory computer readable storage medium storing computer software code which when executing on a data processing system performs a method as claimed in claim 1.

### Claim 25

An apparatus for picking a target piece of fruit from a plant, the apparatus comprising:
a picking device for detaching the target piece of fruit from the plant;
a maneuvering mechanism for moving the picking device along a path towards the target piece of fruit;
one or more sensors for capturing data representative of positions of the target piece of fruit and one or more obstacles; and
processing circuitry operable to:
determine a position of the target piece of fruit to be harvested from the plant using the data captured by the one or more sensors;
determine a position of the one or more obstacles relative to the target piece of fruit using the data captured by the one or more sensors; and
plot the path for the picking device to be moved along from a starting position to a picking position by the maneuvering mechanism, wherein the path is plotted such that the movement of the picking device along the path serves to move at least some of the one or more obstacles relative to the target piece of fruit such that the one or more obstacles do not impede the picking of the target piece of fruit by the picking device.
