# US 2022/0284623 A1 · Framework For 3D Object Detection And Depth Prediction From 2D Images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284623 A1
- Document ID: US20220284623A1
- Status: Published application
- Kind: A1
- Application number: 17/689,530
- Filed: 2022-03-08
- Published: 2022-09-08
- Earliest claimed: 2021-03-08
- Assignees: Ridecell, Inc.
- Applicants: Ridecell, Inc.
- Inventors: Arun Kumar Chockalingam Santha Kumar; Paridhi Singh; Gaurav Singh
- Web page: https://patentgrep.com/patent/US20220284623A1

## CPC

- G06T 7/73: using feature-based methods
- G06T 7/74: {involving reference images or patches}
- B60W 60/0015: {specially adapted for safety}
- G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46)
- G01S 17/89: for mapping or imaging
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 5/027: {Frames}
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/7792: {the supervisor being an automated module, e.g. "intelligent oracle"}
- G06V 10/82: using neural networks
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/64: Three-dimensional [3D] objects
- B60W 2420/408: Radar; Laser, e.g. lidar
- B60W 2554/4029: Pedestrians
- B60W 2554/4041: Position
- G01S 7/4808: {Evaluating distance, position or velocity data}
- G01S 17/931: of land vehicles
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T 2207/30261: Obstacle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Multi-object tracking in autonomous vehicles uses both camera data and LiDAR data for training, but not LiDAR data at query time. Thus, no LiDAR sensor is on a piloted autonomous vehicle. Example systems and methods rely on camera 2D object detections alone, rather than 3D annotations. Example systems/methods utilize a single network that is given a camera image as input and can learn both object detection and dense depth in a multimodal regression setting, where the ground truth LiDAR data is used only at training time to compute depth regression loss. The network uses the camera image alone as input at test time (i.e., when deployed for piloting an autonomous vehicle) and can predict both object detections and dense depth of the scene. LiDAR is only used for data acquisition and is not required for drawing 3D annotations or for piloting the vehicle.

## Claims

### Claim 1

A method for estimating a three-dimensional (3-D) location of an object captured in a two-dimensional (2-D) image, said method comprising:
acquiring position data indicative of 3-D positions of objects in a space;
acquiring a 2-D image of said space, said 2-D image including 2-D representations of said objects in said space;
associating said position data and said 2-D image to create training data;
providing a machine learning framework;
using said training data to train said machine learning framework to create a trained machine learning framework capable of determining 3-D positions of target objects represented in 2-D images of target spaces including said target objects;
obtaining a subsequent 2-D image of a subsequent space; and
utilizing said trained machine learning framework to determine a 3-D position of an object in said subsequent space.

### Claim 2 (depends on 1)

The method of claim 1, wherein said step of utilizing said trained machine learning framework to determine a 3-D position of an object in said subsequent space includes:
providing said subsequent 2-D image to a first portion of said trained machine learning framework configured to encode said subsequent 2-D image to generate an encoded 2-D image;
providing said encoded 2-D image to a second portion of said trained machine learning framework configured to determine a 2-D position of said object in said subsequent space;
providing said encoded 2-D image to a third portion of said trained machine learning framework configured to estimate a depth of said object within said subsequent space; and
combining said 2-D position of said object and said depth of said object to estimate a 3-D position of said object within said subsequent space.

### Claim 3 (depends on 2)

The method of claim 2, wherein:
said first portion of said trained machine learning framework is configured to generate an encoded tensor based at least in part on said subsequent 2-D image;
said step of providing said encoded 2-D image to a second portion of said trained machine learning framework includes providing said encoded tensor to said second portion of said trained machine learning framework; and
said step of providing said encoded 2-D image to a third portion of said trained machine learning framework includes providing said encoded tensor to said third portion of said trained machine learning framework.

### Claim 4 (depends on 3)

The method of claim 3, wherein said first portion of said trained machine learning framework is a deep learning aggregation network configured to encode image features of said subsequent 2-D image to create encoded image features, said encoded image features existing at varying scales.

### Claim 5 (depends on 2)

The method of claim 2, wherein said steps of providing said encoded 2-D image to a second portion of said trained machine learning framework and providing said encoded 2-D image to a third portion of said trained machine learning framework occur in parallel.

### Claim 6 (depends on 2)

The method of claim 2, further comprising determining a 2-D, real-world position of said object in said subsequent space, and wherein:
said step of obtaining a subsequent 2-D image of a subsequent space includes capturing said subsequent 2-D image with an image capture device, said image capture device being associated with an intrinsic matrix;
said intrinsic matrix represents a relationship between points in an image space of said image capture device and locations in a 3-D world space corresponding to said subsequent space; and
said step of determining a 2-D, real-world position of said object in said subsequent space includes associating points in said image space of said image capture device with locations in said 3-D world space based at least in part on said intrinsic matrix.

### Claim 7 (depends on 6)

The method of claim 6, wherein:
said step of combining said 2-D position of said object and said depth of said object to estimate a 3-D position of said object within said subsequent space includes utilizing the Pythagorean theorem to relate said estimated depth of said object, a coordinate of said 2-D position of said object, and a corrected depth of said object;
said estimated depth is a distance between said image capture device and said object as estimated by said third portion of said trained machine learning framework;
said corrected depth is a distance between a first plane perpendicular to an optical axis of said image capture device and intersecting said image capture device and a second plane perpendicular to said optical axis and intersecting said object, said depth of said object representing an estimate of the distance between said image capture device and said object; and
said step of combining said 2-D position of said object and said depth of said object includes calculating said corrected depth based at least in part on said estimated depth, said coordinate, and the Pythagorean theorem.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
said step of obtaining a subsequent 2-D image of a subsequent space includes capturing said subsequent image with an image capture device;
said image capture device is coupled to a vehicle;
said subsequent 2-D image is captured at a particular time; and
said subsequent 2-D image is at least partially representative of the surroundings of said vehicle at said particular time.

### Claim 9 (depends on 8)

The method of claim 8, further comprising providing said trained machine learning framework to said vehicle, and wherein:
said vehicle is an autonomous vehicle; and
movements of said autonomous vehicle are informed at least in part by said 3-D position of said object.

### Claim 10 (depends on 1)

The method of claim 1, wherein said step of using said training data to train said machine learning framework includes:
utilizing said machine learning framework to estimate depths of said objects in said space;
utilizing said machine learning framework to estimate 2-D positions of said objects in said space;
comparing said estimated depths of said objects to observed depths of said objects obtained from said position data;
comparing said estimated 2-D positions of said objects to observed 2-D positions of said objects obtained from position data;
generating a loss function based at least in part on said comparison between said estimated depths of said objects and said observed depths of said objects and based at least in part on said comparison between said estimated 2-D positions of said objects and said observed 2-D positions of said objects; and
altering said machine learning framework based at least in part on said loss function.

### Claim 11 (depends on 10)

The method of claim 10, wherein said step of altering said machine learning framework based at least in part on said loss function includes:
calculating a contribution of a node of said machine learning framework to said loss function; and
altering at least one value corresponding to said node of said machine learning framework based at least in part on said contribution of said node to said loss function.

### Claim 12 (depends on 1)

The method of claim 1, wherein said position data is light detection and ranging (LiDAR) data.

### Claim 13

A system configured to estimate a three-dimensional (3-D) location of an object captured in a two-dimensional (2-D) image, said system comprising:
a hardware processor configured to execute code, said code including a set of native instructions for causing said hardware processor to perform a corresponding set of operations when executed by said hardware processor; and
memory for storing data and said code, said data including
position data indicative of 3-D positions of objects in a space,
a 2-D image of said space, said 2-D image including 2-D representations of said objects in said space, and
a subsequent 2-D image of a subsequent space; and wherein
said code includes
a machine learning framework,
a first subset of said set of native instructions configured to associate said position data and said 2-D image to create training data,
a second subset of said set of native instructions configured to use said training data to train said machine learning framework to create a trained machine learning framework capable of determining 3-D positions of target objects represented in 2-D images of target spaces including said target objects, and
a third subset of said set of native instructions configured to cause said trained machine learning framework to determine a 3-D position of an object in said subsequent space.

### Claim 14 (depends on 13)

The system of claim 13, wherein said third subset of said set of native instructions is additionally configured to:
provide said subsequent 2-D image to a first portion of said trained machine learning framework configured to encode said subsequent 2-D image to generate an encoded 2-D image;
provide said encoded 2-D image to a second portion of said trained machine learning framework configured to determine a 2-D position of said object in said subsequent space;
provide said encoded 2-D image to a third portion of said trained machine learning framework configured to estimate a depth of said object within said subsequent space; and
combine said 2-D position of said object and said depth of said object to estimate a 3-D position of said object within said subsequent space.

### Claim 15 (depends on 14)

The system of claim 14, wherein:
said first portion of said trained machine learning framework is configured to generate an encoded tensor based at least in part on said subsequent 2-D image;
said third subset of said set of native instructions is additionally configured to provide said encoded tensor to said second portion of said trained machine learning framework; and
said third subset of said set of native instructions is additionally configured to provide said encoded tensor to said third portion of said trained machine learning framework.

### Claim 16 (depends on 15)

The system of claim 15, wherein said first portion of said trained machine learning framework is a deep learning aggregation network configured to encode image features of said subsequent 2-D image to create encoded image features, said encoded image features existing at varying scales.

### Claim 17 (depends on 14)

The system of claim 14, wherein said third subset of said set of native instructions is additionally configured to provide said encoded 2-D image to said second portion of said trained machine learning framework and to said third portion of said trained machine learning framework in parallel.

### Claim 18 (depends on 14)

The system of claim 14, further comprising an image capture device associated with an intrinsic matrix, and wherein:
said code includes a fourth subset of said set of native instructions configured to determine a 2-D, real-world position of said object in said subsequent space;
said subsequent 2-D image is captured by said image capture device;
said intrinsic matrix represents a relationship between points in an image space of said image capture device and locations in a 3-D world space corresponding to said subsequent space; and
said fourth subset of said set of native instructions is additionally configured to associate points in said image space of said image capture device with locations in said 3-D world space based at least in part on said intrinsic matrix.

### Claim 19 (depends on 18)

The system of claim 18, wherein:
said third subset of said set of native instructions is additionally configured to utilize the Pythagorean theorem to relate said estimated depth of said object, a coordinate of said 2-D position of said object, and a corrected depth of said object;
said estimated depth is a distance between said image capture device and said object as estimated by said third portion of said trained machine learning framework;
said corrected depth is a distance between a first plane perpendicular to an optical axis of said image capture device and intersecting said image capture device and a second plane perpendicular to said optical axis and intersecting said object, said depth of said object representing an estimate of the distance between said image capture device and said object; and
said third subset of said set of native instructions is additionally configured to calculate said corrected depth based at least in part on said estimated depth, said coordinate, and the Pythagorean theorem.

### Claim 20 (depends on 13)

The system of claim 13, further comprising:
a vehicle; and
an image capture device coupled to said vehicle; and wherein
said subsequent 2-D image is captured by said image capture device at a particular time; and
said subsequent 2-D image is at least partially representative of the surroundings of said vehicle at said particular time.

### Claim 21 (depends on 20)

The system of claim 20, further comprising a network adapter configured to establish a data connection between said hardware processor and said vehicle, and wherein:
said code includes a fourth subset of said set of native instructions configured to provide said trained machine learning framework to said vehicle;
said vehicle is an autonomous vehicle; and
movements of said autonomous vehicle are informed at least in part by said 3-D position of said object.

### Claim 22 (depends on 13)

The system of claim 13, wherein said second subset of said set of native instructions is additionally configured to:
utilize said machine learning framework to estimate depths of said objects in said space;
utilize said machine learning framework to estimate 2-D positions of said objects in said space;
compare said estimated depths of said objects to observed depths of said objects obtained from said position data;
compare said estimated 2-D positions of said objects to observed 2-D positions of said objects obtained from position data;
generate a loss function based at least in part on said comparison between said estimated depths of said objects and said observed depths of said objects and at least in part on said comparison between said estimated 2-D positions of said objects and said observed 2-D positions of said objects; and
alter said machine learning framework based at least in part on said loss function.

### Claim 23 (depends on 22)

The system of claim 22, wherein said second subset of said set of native instructions is additionally configured to:
calculate a contribution of a node of said machine learning framework to said loss function; and
alter at least one value corresponding to said node of said machine learning framework based at least in part on said contribution of said node to said loss function.

### Claim 24 (depends on 13)

The method of claim 13, wherein said position data is light detection and ranging (LiDAR) data.
