# US 2023/0266473 A1 · Method and system for object detection for a mobile robot with time-of-flight camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0266473 A1
- Document ID: US20230266473A1
- Status: Published application
- Kind: A1
- Application number: 17/768,320
- Filed: 2020-10-21
- Published: 2023-08-24
- Earliest claimed: 2019-10-25
- Assignees: Starship Technologies Oü
- Applicants: Starship Technologies Oü
- Inventors: Ardi LOOT; Risto REINPÕLD; Sergii KHARAGORGIIEV; Tommi TYKKÄLÄ
- Web page: https://patentgrep.com/patent/US20230266473A1

## CPC

- G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/931: of land vehicles
- G05D 1/0242: {using non-visible light signals, e.g. IR or UV signals}
- G05D 1/0248: {in combination with a laser (lasers per se H01S)}
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES

## Abstract

The present invention provides a system and method for operating a mobile robot. Firstly, a mobile robot can be equipped with at least one time-of-flight (ToF) sensor and the mobile robot can travel in an outdoor setting. At least one ToF sensor image related to the outdoor setting can be captured via the at least one ToF sensor. A data processing unit can process the at least one ToF sensor image to identify at least one cluster of pixels on the at least one ToF sensor image based on at least one pixel-clustering parameter. It can be determined whether at least one cluster of pixels corresponds to a hazardous object in the outdoor setting.

## Claims

### Claim 1

A method for operating a mobile robot comprising at least one time-of-flight, ToF, sensor, the method comprising:
the mobile robot travelling in an outdoor setting; and
capturing at least one ToF sensor image related to the outdoor setting via the at least one ToF sensor; and
a data processing unit processing the at least one ToF sensor image to identify at least one cluster of pixels on the at least one ToF sensor image based on at least one pixel-clustering parameter; and
determining whether at least one cluster of pixels corresponds to a hazardous object in the outdoor setting.

### Claim at position 2

2-15. (canceled)

### Claim 16 (depends on 1)

The method according to claim 1, wherein capturing the at least one ToF sensor image comprises
capturing at least one distance image wherein each pixel comprises data indicating a distance between the at least one ToF sensor and a corresponding segment in the outdoor setting.

### Claim 17 (depends on 1)

The method according to claim 1, wherein capturing the at least one ToF sensor image comprises
capturing at least one brightness image wherein each pixel comprises data indicating an amount of light received by the at least one ToF sensor.

### Claim 18 (depends on 1)

The method according to claim 1, wherein the at least one pixel-clustering parameter is a plurality of pixel-clustering parameters, and
wherein each of the pixels is associated with a respective pixel-clustering parameter.

### Claim 19 (depends on 18)

The method according to claim 18, wherein processing the at least one ToF sensor image to identify at least one cluster of pixels comprises identifying a plurality of pixels on the at least one ToF sensor image that are positioned within a predetermined distance from each other on the at least one ToF sensor image, and
wherein the difference between their respective pixel-clustering parameters is less than a pre-determined pixel-clustering parameter similarity threshold.

### Claim 20 (depends on 1)

The method according to claim 1, wherein processing the at least one ToF sensor image to identify at least one cluster of pixels is based on a two-phase clustering process,
wherein in a first phase of the two-phase clustering process, pixels in at least some of a plurality of first portions of the at least one ToF sensor image are grouped into clusters based on respective first pixel-clustering parameters corresponding to the pixels;
wherein in a second phase of the two-phase clustering process, clusters in at least some of a plurality of second portions of the at least one ToF sensor image are grouped into composite clusters based on respective second-pixel clustering parameters corresponding to the clusters;
wherein the second phase is performed after the first phase and
wherein each of the second portions of the at least one ToF sensor image is composed of at least two of the plurality of the first portions of the at least one ToF sensor image.

### Claim 21 (depends on 20)

The method according to claim 20, wherein the second portion of the at least one ToF sensor image is composed of a plurality of first portions of the at least one ToF sensor image that are vertically aligned.

### Claim 22 (depends on 20)

The method according to claim 20, wherein for each of at least some of the clusters, the respective second-pixel clustering parameter is calculated based on the first pixel-clustering parameters corresponding to the pixels in that cluster.

### Claim 23 (depends on 1)

The method according to claim 1, wherein determining whether at least one cluster of pixels corresponds to a hazardous object in the outdoor setting comprises
determining whether at least one cluster of pixels corresponds to an object or obstacle in the outdoor setting that obstructs mobile robot’s travelling.

### Claim 24 (depends on 1)

The method according to claim 1, further comprising
the data processing unit generating a computerized view of the outdoor setting and projecting the at least one cluster of pixels and/or the detected hazardous object on the computerized view.

### Claim 25 (depends on 1)

The method according to claim 1, further comprising:
capturing at least two ToF sensor images at different times via the at least one ToF sensor, and
wherein determining whether at least one cluster of pixels corresponds to a hazardous object in the outdoor setting comprises determining whether the at least one cluster of pixels corresponds to a moving object in the outdoor setting.

### Claim 26 (depends on 1)

The method according to claim 1, wherein the method comprises operating the mobile robot based on the determining whether at least one cluster of pixels corresponds to a hazardous object in the outdoor setting.

### Claim 27

A system configured to operate a mobile robot comprising:
a mobile robot configured to travel in an outdoor setting and comprising at least one time-of-flight, ToF, sensor configured to capture at least one ToF sensor image, wherein each of the at least one ToF sensor image comprises a plurality of pixels; and
a data processing unit configured to:
process the at least one ToF sensor image to identify at least one cluster of pixels on the at least one ToF sensor image based on at least one pixel-clustering parameter; and
determine whether at least one cluster of pixels corresponds to a hazardous object in the outdoor setting.

### Claim 28 (depends on 27)

The system according to claim 27, wherein the at least one ToF sensor is configured to capture at least one distance image wherein each pixel comprises data indicating a distance between the at least one ToF sensor and a corresponding segment in the outdoor setting.

### Claim 29 (depends on 27)

The system according to claim 27, wherein the at least one ToF sensor is configured to capture at least one brightness image wherein each pixel comprises data indicating an amount of light received by the at least one ToF sensor.

### Claim 30 (depends on 27)

The system according to claim 27, wherein the at least one pixel-clustering parameter is a plurality of pixel-clustering parameters, and wherein each of the pixels is associated with a respective pixel-clustering parameter.

### Claim 31 (depends on 30)

The system according to claim 30, wherein data processing unit is configured to process the at least one ToF sensor image to identify at least one cluster of pixels by identifying a plurality of pixels on the at least one ToF sensor image that are positioned within a predetermined distance from each other on the at least one ToF sensor image, and
wherein the difference between their respective pixel-clustering parameters is less than a pre-determined pixel-clustering parameter similarity threshold.

### Claim 32 (depends on 27)

The system according to claim 27, wherein the data processing unit is configured to execute a two-phase clustering process to identify at least one cluster of pixels on the at least one ToF sensor image based on at least one pixel-clustering parameter,
wherein in a first phase of the two-phase clustering process, the data processing unit is configured to group pixels in at least some of a plurality of first portions of the at least one ToF sensor image into clusters based on respective first pixel-clustering parameters corresponding to the pixels,
wherein in a second phase of the two-phase clustering process, the data processing unit is configured to group clusters in at least some of a plurality of second portions of the at least one ToF sensor image into composite clusters based on respective second-pixel clustering parameters corresponding to the clusters, and
wherein the data processing unit is configured to perform the second phase after the first phase and
wherein each of the second portions of the at least one ToF sensor image is composed of at least two of the plurality of the first portions of the at least one ToF sensor image.

### Claim 33 (depends on 27)

The system according to claim 27, wherein the at least one ToF sensor is configured to capture at least two ToF sensor images at different times, and
wherein the data processing device is configured to determine whether the at least one cluster of pixels corresponds to a moving object in the outdoor setting, and,
based thereon, to determine whether at least one cluster of pixels corresponds to a hazardous object in the outdoor setting.

### Claim 34 (depends on 27)

The system according to claim 27, wherein the data processing unit is configured to operate the mobile robot based on the determining whether at least one cluster of pixels corresponds to a hazardous object in the outdoor setting.
