# US 11,443,147 B2 · Systems and methods for object detection using stereovision information

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,147 B2
- Document ID: US11443147B2
- Status: Granted
- Kind: B2
- Application number: 17/118,705
- Filed: 2020-12-11
- Published: 2022-06-16
- Granted: 2022-09-13
- Earliest claimed: 2020-12-11
- Assignees: Argo AI, LLC
- Applicants: Argo AI, LLC
- Inventors: Xiaoyan Hu; Lingyuan Wang; Michael Happold; Jason Ziglar
- Web page: https://patentgrep.com/patent/US11443147B2

## CPC

- G06F 18/2193: {based on specific statistical tests}
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- G06F 18/24323: {Tree-organised classifiers}
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597)
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for operating a vehicle. The methods comprise, by a processor: obtaining a pair of stereo images captured by a stereo camera; processing the pair of stereo images to generate a disparity map comprising a plurality of pixels defined by intensity values; converting each intensity value to a 3D position in a map (each 3D position defining a location of a data point in a point cloud); performing a hierarchical decision tree classification to determine a classification for each data point in the point cloud (the classification being a foreground classification or a background classification); and using the classifications to facilitate autonomous control of the vehicle.

## Claims

### Claim 1

A method for operating a vehicle, comprising:
obtaining, by a processor, a pair of stereo images captured by a stereo camera;
processing, by the processor, the pair of stereo images to generate a disparity map comprising a plurality of pixels defined by intensity values, each said intensity value denoting a position difference of a pixel in the stereo images and a distance between the pixel and the stereo camera;
converting, by the processor, each said intensity value to a 3D position in a map, each 3D position defining a location of a data point in a point cloud;
performing, by the processor, a hierarchical decision tree classification based on at least the intensity value of a given pixel of the plurality of pixels to determine a classification for each said data point in the point cloud, the classification being a foreground classification or a background classification; and
using the classifications by the processor to facilitate autonomous control of the vehicle.

### Claim 2 (depends on 1)

The method according to claim 1, wherein each said intensity value is converted into the 3D position based on at least one of a known time of image capture, a known location of the vehicle at the time of image capture, known positions of the stereo cameras relative to the vehicle, known pixel pointing directions, and known distances of the pixels to the stereo cameras.

### Claim 3 (depends on 1)

The method according to claim 1, further comprising performing, by the processor, ground segmentation to reduce a total number of data points in the point cloud prior to performance of the hierarchical decision tree classification.

### Claim 4 (depends on 1)

The method according to claim 1, further comprising computing, by the processor, a delta value for each said data point that represents a difference between a ground height value and a z-coordinate value partially defining a respective 3D position in the map.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the hierarchical decision tree classification is further based on the delta value.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the hierarchical decision tree classification is based further on at least one of said 3D position, a color value for a first pixel in the pair of stereo images, and at least one feature of a second pixel in the pair of stereo images.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the second pixel resides proximate to the first pixel in the pair of stereo images.

### Claim 8 (depends on 6)

The method according to claim 6, wherein the at least one feature of the second pixel comprises an intensity value, a 3D position, a color value, and a location of the second pixel relative to the first pixel in the pair of stereo images.

### Claim 9 (depends on 1)

The method according to claim 1, wherein the hierarchical decision tree classification is facilitated by machine learned models that each operate using a set of decision trees, where each said decision tree produces a class prediction for each said data point in said point cloud and a confidence value for the class prediction.

### Claim 10 (depends on 9)

The method according to claim 9, wherein the class predictions produced by the decision trees in the set are respectively combined together to determine a final class prediction for each said data point.

### Claim 11 (depends on 10)

The method according to claim 10, wherein the final class predictions is determined using a weighted function in which the confidence values are used as weights.

### Claim 12 (depends on 1)

The method according to claim 1, wherein the hierarchical decision tree classification comprises a Random Forest classification.

### Claim 13

A system, comprising:
a processor; and
a non-transitory computer-readable storage medium comprising programming instructions that are configured to cause the processor to implement a method for operating a vehicle, wherein the programming instructions comprise instructions to:
obtain a pair of stereo images captured by a stereo camera;
process the pair of stereo images to generate a disparity map comprising a plurality of pixels defined by intensity values, each said intensity value denoting a position difference of a pixel in the stereo images and a distance between the pixel and the stereo camera;
convert each said intensity value to a 3D position in a map, each 3D position defining a location of a data point in a point cloud;
perform a hierarchical decision tree classification based on at least the intensity value of a given pixel of the plurality of pixels to determine a classification for each said data point in the point cloud, the classification being a foreground classification or a background classification; and
use the classifications to facilitate autonomous control of the vehicle.

### Claim 14 (depends on 13)

The system according to claim 13, wherein each said intensity value is converted into the 3D position based on at least one of a known time of image capture, a known location of the vehicle at the time of image capture, known positions of the stereo cameras relative to the vehicle, known pixel pointing directions, and known distances of the pixels to the stereo cameras.

### Claim 15 (depends on 13)

The system according to claim 13, wherein the programming instructions further comprise instructions to reduce a total number of data points in the point cloud based on ground height, prior to performance of the hierarchical decision tree classification.

### Claim 16 (depends on 13)

The system according to claim 13, wherein the programming instructions further comprise instructions to compute a delta value for each said data point that represents a difference between a ground height value and a z-coordinate value partially defining a respective 3D position in the map.

### Claim 17 (depends on 16)

The system according to claim 16, wherein the hierarchical decision tree classification is based further on the delta value.

### Claim 18 (depends on 13)

The system according to claim 13, wherein the hierarchical decision tree classification is based on further at least one of said 3D position, a color value for a first pixel in the pair of stereo images, and at least one feature of a second pixel in the pair of stereo images.

### Claim 19 (depends on 18)

The system according to claim 18, wherein the second pixel resides proximate to the first pixel in the pair of stereo images.

### Claim 20 (depends on 18)

The system according to claim 18, wherein the at least one feature of the second pixel comprises an intensity value, a 3D position, a color value, and a location of the second pixel relative to the first pixel in the pair of stereo images.

### Claim 21 (depends on 13)

The system according to claim 13, wherein the hierarchical decision tree classification is facilitated by machine learned models that each operate using a set of decision trees, where each said decision tree produces a class prediction for each said data point in said point cloud and a confidence value for the class prediction.

### Claim 22 (depends on 21)

The system according to claim 21, wherein the class predictions produced by the decision trees in the set are respectively combined together to determine a final class prediction for each said data point.

### Claim 23 (depends on 22)

The system according to claim 22, wherein the final class predictions is determined using a weighted function in which the confidence values are used as weights.

### Claim 24 (depends on 13)

The system according to claim 13, wherein the hierarchical decision tree classification comprises a Random Forest classification.

### Claim 25

A non-transitory computer-readable medium that stores instructions that is configured to, when executed by at least one computing device, cause the at least one computing device to perform operations comprising:
obtaining a pair of stereo images captured by a stereo camera;
processing the pair of stereo images to generate a disparity map comprising a plurality of pixels defined by intensity values, each said intensity value denoting a position difference of a pixel in the stereo images and a distance between the pixel and the stereo camera;
converting each said intensity value to a 3D position in a map, each 3D position defining a location of a data point in a point cloud;
performing a hierarchical decision tree classification based on at least the intensity value of a given pixel of the plurality of pixels to determine a classification for each said data point in the point cloud, the classification being a foreground classification or a background classification; and
using the classifications to facilitate autonomous control of the vehicle.