# US 12,602,936 B2 · Long-range 3D object detection using 2D bounding boxes

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,936 B2
- Document ID: US12602936B2
- Status: Granted
- Kind: B2
- Application number: 18/223,473
- Filed: 2023-07-18
- Published: 2024-07-25
- Granted: 2026-04-14
- Earliest claimed: 2023-01-20
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Zetong Yang; Zhiding Yu; Ren Hao Wang; Chris Choy; Anima Anandkumar; Jose M. Alvarez Lopez
- Web page: https://patentgrep.com/patent/US12602936B2

## CPC

- G06V 20/64: Three-dimensional [3D] objects
- G06T 7/50: Depth or shape recovery
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06V 10/225: {based on a marking or identifier characterising the area}
- G06V 10/82: using neural networks
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06V 2201/07: Target detection
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

3D object detection is a computer vision task that generally detects (e.g. classifies and localizes) objects in 3D space from the 2D images or videos that capture the objects. Current techniques used for 3D object detection rely on machine learning processes that learn to detect 3D objects from existing images annotated with high-quality 3D information including depth information generally obtained using lidar technology. However, due to lidar's limited measurable range, current machine learning solutions to 3D object detection do not support detection of 3D objects beyond the lidar range, which is needed for numerous applications, including autonomous driving applications where existing close or midrange 3D object detection does not always meet the safety-critical requirement of autonomous driving. The present disclosure provides for 3D object detection using a technique that supports long-range detection (i.e. detection beyond the lidar range).

## Claims

### Claim 1

A method, comprising:
at a device:
computing a two-dimensional (2D) bounding box for an object in an image of a scene;
processing the 2D bounding box, using a neural network, to predict a three-dimensional (3D) bounding box for the object in the scene;
output the 3D bounding box for use as a ground truth in a dataset on which a second neural network is trained to predict 3D bounding boxes for objects in a given scene; and
train the second neural network to predict 3D bounding boxes for objects in a given scene, using the dataset;
wherein the second neural network is evaluated based on a dynamic detection metric that allows for depth accuracy to be relaxed as object depth increases.

### Claim 2 (depends on 1)

The method of claim 1, wherein the image of the scene is captured by a camera.

### Claim 3 (depends on 1)

The method of claim 1, wherein a camera calibration matrix is used to compute the 2D bounding box for the object.

### Claim 4 (depends on 1)

The method of claim 1, wherein the 2D bounding box is defined by a width value and a height value.

### Claim 5 (depends on 1)

The method of claim 1, wherein the neural network further processes a feature vector representing the object together with the 2D bounding box to predict the 3D bounding box for the object in the scene.

### Claim 6 (depends on 5)

The method of claim 5, wherein the feature vector is determined from the image of the scene.

### Claim 7 (depends on 6)

The method of claim 6, wherein the feature vector includes data associated with an orientation of the object.

### Claim 8 (depends on 6)

The method of claim 6, wherein the feature vector includes data associated with a size of the object.

### Claim 9 (depends on 6)

The method of claim 6, wherein the feature vector includes data associated with a classification of the object.

### Claim 10 (depends on 1)

The method of claim 1, wherein the neural network is trained on a dataset that includes labeled objects having a depth less than a defined depth threshold, wherein the label for each of the labeled objects indicates:
one or more features of the labeled object,
a 3D bounding box for the labeled object.

### Claim 11 (depends on 10)

The method of claim 10, wherein the neural network learns an inverse function for determining object depth from a given object 2D bounding box.

### Claim 12 (depends on 1)

The method of claim 1, wherein a depth of the object is greater than a defined depth threshold.

### Claim 13 (depends on 1)

The method of claim 1, wherein the 3D bounding box indicates a depth of the object.

### Claim 14 (depends on 1)

The method of claim 1, wherein the second neural network is a camera-based 3D detector.

### Claim 15 (depends on 14)

The method of claim 14, wherein the device further:
uses the camera-based 3D detector for an autonomous navigation application.

### Claim 16 (depends on 15)

The method of claim 15, wherein autonomous navigation application is an autonomous driving application.

### Claim 17 (depends on 14)

The method of claim 14, wherein the device further:
uses the camera-based 3D detector for a robotics application.

### Claim 18

A system, comprising:
a non-transitory memory storage comprising instructions; and
one or more processors in communication with the memory, wherein the one or more processors execute the instructions to:
compute a two-dimensional (2D) bounding box for an object in an image of a scene;
process the 2D bounding box, using a neural network, to predict a three-dimensional (3D) bounding box for the object in the scene;
output the 3D bounding box for use as a ground truth in a dataset on which a second neural network is trained to predict 3D bounding boxes for objects in a given scene; and
train the second neural network to predict 3D bounding boxes for objects in a given scene, using the dataset;
wherein the second neural network is evaluated based on a dynamic detection metric that allows for depth accuracy to be relaxed as object depth increases.

### Claim 19

A non-transitory computer-readable media storing computer instructions which when executed by one or more processors of a device cause the device to:
compute a two-dimensional (2D) bounding box for an object in an image of a scene;
process the 2D bounding box, using a neural network, to predict a three-dimensional (3D) bounding box for the object in the scene;
output the 3D bounding box for use as a ground truth in a dataset on which a second neural network is trained to predict 3D bounding boxes for objects in a given scene; and
train the second neural network to predict 3D bounding boxes for objects in a given scene, using the dataset;
wherein the second neural network is evaluated based on a dynamic detection metric that allows for depth accuracy to be relaxed as object depth increases.