# US 12,008,818 B2 · Systems and methods to train a prediction system for depth perception

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,818 B2
- Document ID: US12008818B2
- Status: Granted
- Kind: B2
- Application number: 17/384,121
- Filed: 2021-07-23
- Published: 2022-09-22
- Granted: 2024-06-11
- Earliest claimed: 2021-03-16
- Assignees: Toyota Research Institute, Inc.
- Applicants: Toyota Research Institute, Inc.
- Inventors: Rares A. Ambrus; Dennis Park; Vitor Guizilini; Jie Li; Adrien David Gaidon
- Web page: https://patentgrep.com/patent/US12008818B2

## CPC

- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G01S 7/4802: {using analysis of echo signal for target characterisation; Target signature; Target cross-section}
- G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46)
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/89: for mapping or imaging
- G01S 17/931: of land vehicles
- G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/251: {of input or preprocessed data}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06T 7/11: Region-based segmentation
- G06T 7/50: Depth or shape recovery
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 10/757: {Matching configurations of points or features}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06N 3/045: Combinations of networks
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30248: Vehicle exterior or interior
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

System, methods, and other embodiments described herein relate to a manner of training a depth prediction system using bounding boxes. In one embodiment, a method includes segmenting an image to mask areas beyond bounding boxes and identify unmasked areas within the bounding boxes. The method also includes training a depth model using depth losses from comparing weighted points associated with pixels of the image within the unmasked areas to ground-truth depth. The method also includes providing the depth model for object detection.

## Claims

### Claim 1

A prediction system for training a depth model, comprising:
a processor; and
a memory storing instructions that, when executed by the processor, cause the processor to:
segment an image to mask areas beyond bounding boxes and identify unmasked areas within the bounding boxes;
train the depth model using depth losses from comparing weighted points associated with ranked pixels and a size of objects associated with a pixel count within the unmasked areas to ground-truth depth; and
provide the depth model for object detection.

### Claim 2 (depends on 1)

The prediction system of claim 1, wherein the instructions to train the depth model further include instructions to calculate the depth losses by adaptively decaying the ranked pixels relative to centers of the bounding boxes and selecting the weighted points according to the decaying.

### Claim 3 (depends on 1)

The prediction system of claim 1, wherein the instructions to train the depth model further include instructions to calculate the depth losses for the ranked pixels within the bounding boxes according to a statistical distribution for selecting the weighted points.

### Claim 4 (depends on 1)

The prediction system of claim 1, wherein the weighted points are over-weighted for the ranked pixels that overlap between the bounding boxes and the bounding boxes represent the objects identified within the image.

### Claim 5 (depends on 1)

The prediction system of claim 1, wherein the instructions to train the depth model further include instructions to calculate the depth losses by selecting the weighted points according to the size identified within the unmasked areas.

### Claim 6 (depends on 1)

The prediction system of claim 1, wherein the weighted points are proximate to focal points of the objects within the unmasked areas.

### Claim 7 (depends on 1)

The prediction system of claim 1, wherein the bounding boxes are locations within the image having the objects in a scene and the weighted points are associated with a point cloud generated from the image.

### Claim 8

A non-transitory computer-readable medium for training a depth model including instructions that, when executed by a processor, cause the processor to:
segment an image to mask areas beyond bounding boxes and identify unmasked areas within the bounding boxes;
train the depth model using depth losses from comparing weighted points associated with ranked pixels and a size of objects associated with a pixel count within the unmasked areas to ground-truth depth; and
provide the depth model for object detection.

### Claim 9 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the instructions to train the depth model further include instructions to calculate the depth losses by adaptively decaying the ranked pixels relative to centers of the bounding boxes and selecting the weighted points according to the decaying.

### Claim 10 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the instructions to train the depth model further include instructions to calculate the depth losses the ranked pixels within the bounding boxes according to a statistical distribution for selecting the weighted points.

### Claim 11 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the weighted points are over-weighted for the ranked pixels that overlap between the bounding boxes and the bounding boxes represent the objects identified within the image.

### Claim 12 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the instructions to train the depth model further include instructions to calculate the depth losses by selecting the weighted points according to the sizes identified within the unmasked areas.

### Claim 13 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the weighted points are proximate to focal points of the objects within the unmasked areas.

### Claim 14

A method, comprising:
segmenting an image to mask areas beyond bounding boxes and identify unmasked areas within the bounding boxes;
training a depth model using depth losses from comparing weighted points associated with ranked pixels and a size of objects associated with a pixel count within the unmasked areas to ground-truth depth; and
providing the depth model for object detection.

### Claim 15 (depends on 14)

The method of claim 14, wherein training the depth model further includes calculating the depth losses by adaptively decaying the ranked pixels relative to centers of the bounding boxes and selecting the weighted points according to the decaying.

### Claim 16 (depends on 14)

The method of claim 14, wherein training the depth model further includes calculating the depth losses for the ranked pixels within the bounding boxes according to a statistical distribution for selecting the weighted points.

### Claim 17 (depends on 14)

The method of claim 14, wherein the weighted points are over-weighted for the ranked pixels that overlap between the bounding boxes and the bounding boxes represent the objects identified within the image.

### Claim 18 (depends on 14)

The method of claim 14, wherein training the depth model further includes calculating the depth losses by selecting the weighted points according to the size identified within the unmasked areas.

### Claim 19 (depends on 14)

The method of claim 14, wherein the weighted points are proximate to focal points of the objects within the unmasked areas.

### Claim 20 (depends on 14)

The method of claim 14, wherein the bounding boxes are locations within the image having the objects in a scene and the weighted points are associated with a point cloud generated from the image.