# US 12,111,177 B2 · Generating training data for deep learning models for building high definition maps

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,111,177 B2
- Document ID: US12111177B2
- Status: Granted
- Kind: B2
- Application number: 16/919,131
- Filed: 2020-07-02
- Published: 2021-01-07
- Granted: 2024-10-08
- Earliest claimed: 2019-07-05
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Yu Zhang; Lin Yang
- Web page: https://patentgrep.com/patent/US12111177B2

## CPC

- G01C 21/3841: {Data obtained from two or more sources, e.g. probe vehicles}
- G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)}
- G05D 1/246: using environment maps, e.g. simultaneous localisation and mapping [SLAM]
- G06T 7/74: {involving reference images or patches}
- G06T 7/75: {involving models}
- G06T 11/20: Drawing from basic elements
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T 2210/56: Particle system, point based geometry or rendering
- G06T 2219/004: Annotating, labelling
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

According to an aspect of an embodiment, operations may comprise receiving sensor data from one or more vehicles, determining, by combining the received sensor data, a high definition map comprising a point cloud, and labeling one or more objects in the point cloud. The operations may also comprise generating training data by receiving a new image captured by one of the vehicles, receiving a pose of the vehicle when the new image was captured, determining an object having a label in the point cloud that is observable from the pose of the vehicle, determining a position of the object in the new image, and labeling the new image by assigning the label of the object to the new image, the labeled new image comprising the training data. The operations may also comprise training a deep learning model using the training data.

## Claims

### Claim 1

A method, comprising:
receiving occupancy map data corresponding to an occupancy map of an area, the occupancy map being generated based at least on sensor data previously obtained using a plurality of vehicles and including a point cloud and a label corresponding to one or more points of the point cloud, the label including a classification and a first location corresponding to one or more features associated with the one or more points;
performing localization, with respect to the occupancy map, of a vehicle different from the plurality of vehicles used to obtain the sensor data used to generate the occupancy map by, at least, comparing the occupancy map data with sensor data obtained using one or more sensors corresponding to the vehicle;
capturing a camera image while the vehicle is localized with respect to the occupancy map;
aligning the camera image with the occupancy map based at least on the localization of the vehicle with respect to the occupancy map;
projecting the label corresponding to the one or more points from the occupancy map to a portion of the camera image that depicts an object and that is aligned, based at least on the aligning of the camera image with the occupancy map, with the one or more points of the occupancy map data that correspond to the object, the projecting of the label to the portion of the camera image indicating a second location of the one or more features in the camera image;
computing, using a deep learning model, an estimated location and an estimated classification of the object as depicted in the camera image;
comparing the estimated location with the second location and the estimated classification with the label of the object corresponding to the camera image; and
updating one or more parameters of a deep learning model based at least on the comparing.

### Claim 2 (depends on 1)

The method of claim 1, wherein the camera image is captured with respect to one or more different conditions than one or more occupancy map camera images corresponding to determining the label as included in the occupancy map.

### Claim 3 (depends on 2)

The method of claim 2, wherein the one or more different conditions include one or more of: a time of day, a weather condition, a time of year, or a lighting condition.

### Claim 4 (depends on 1)

The method of claim 1, wherein:
the localization includes determining a pose of the vehicle at a time of capture of the camera image.

### Claim 5 (depends on 1)

The method of claim 1, wherein the assigning of the label to the portion of the camera image is selected for review based at least on one or more filters.

### Claim 6 (depends on 5)

The method of claim 5, wherein the reviewing of the label includes verifying an accuracy of the label.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
the object is selected for labeling with respect to the camera image based at least on a determination that the object is observable from a pose of the vehicle at a time of capture of the camera image.

### Claim 8

A processor comprising processing circuitry to cause performance of operations, the operations comprising:
receiving occupancy map data corresponding to an occupancy map of an area, the occupancy map being generated based at least on sensor data previously obtained using a plurality of vehicles and including a point cloud and a label corresponding to one or more points of the point cloud;
performing localization, with respect to the occupancy map, of a vehicle different from the plurality of vehicles used to obtain the sensor data used to generate the occupancy map by, at least, comparing the occupancy map data with sensor data obtained using one or more sensors corresponding to the vehicle;
aligning an image obtained using a camera with the occupancy map based at least on the localization of the vehicle with respect to the occupancy map;
assigning the label to a portion of the image that depicts an object and that is aligned with the one or more points of the occupancy map that correspond to the object; and
using the image and the label as training data for a deep learning model, wherein using the image and the label for the training includes:
computing, using the deep learning model, an estimated classification of the object as depicted in the image;
comparing the estimated classification with the label of the object corresponding to the image; and
updating one or more parameters of the deep learning model based at least on the comparing.

### Claim 9 (depends on 8)

The processor of claim 8, wherein the image is captured with respect to one or more different conditions than one or more occupancy map camera images corresponding to determining the label as included in the occupancy map.

### Claim 10 (depends on 9)

The processor of claim 9, wherein the one or more different conditions include one or more of: a time of day, a weather condition, a time of year, or a lighting condition.

### Claim 11 (depends on 8)

The processor of claim 8, wherein:
the localization includes determining a pose of the vehicle at a time of capture of the image.

### Claim 12 (depends on 8)

The processor of claim 8, wherein the assigning of the label to the portion of the image is selected for review based at least on one or more filters.

### Claim 13 (depends on 12)

The processor of claim 12, wherein the reviewing of the label includes verifying an accuracy of the label.

### Claim 14 (depends on 8)

The processor of claim 8, wherein:
the object is selected for labeling based at least on a determination that the object is observable from a pose of the vehicle at a time of capture of the image.

### Claim 15

A system comprising:
one or more processing units to cause the system to perform operations, the operations comprising:
generating training data, at least, by:
receiving occupancy map data corresponding to an occupancy map of an area, the occupancy map being generated based at least on sensor data previously obtained using a plurality of vehicles and including a point cloud and a label corresponding to one or more points of the point cloud;
performing localization, with respect to the occupancy map, of a vehicle different from the plurality of vehicles used to obtain the sensor data used to generate the occupancy map by, at least, comparing the occupancy map data with sensor data obtained using one or more sensors corresponding to the vehicle;
labeling at least a portion of an image captured using a camera and subsequent the localization, the labeling including assigning a label to an object depicted in the image, the assigning of the label including:
aligning the object, as depicted in the image, with the one or more points of the occupancy map based at least on the localization of the vehicle with respect to the occupancy map;
identifying that the one or more points of the point cloud are labeled with the label; and
assigning the label to the object in the image based at least on the one or more points of the occupancy map being aligned with the object as depicted in the image and based at least on the one or more points being labeled with the label; and
training a deep learning model using the training data, the training of the deep learning model including:
computing, using the deep learning model, an estimated classification of the object as depicted in the image;
comparing the estimated classification with the label of the object corresponding to the image; and
updating one or more parameters of the deep learning model based at least on the comparing.

### Claim 16 (depends on 15)

The system of claim 15, wherein the image is captured with respect to one or more different conditions than one or more occupancy map camera images corresponding to determining the label as included in the occupancy map.

### Claim 17 (depends on 16)

The system of claim 16, wherein the one or more different conditions include one or more of: a time of day, a weather condition, a time of year, or a lighting condition.

### Claim 18 (depends on 15)

The system of claim 15, wherein:
the localization includes determining a pose of the vehicle at a time of capture of the image.

### Claim 19 (depends on 15)

The system of claim 15, wherein the assigning of the label to the object in the image is selected for review based at least on one or more filters.

### Claim 20 (depends on 19)

The system of claim 19, wherein the reviewing of the label includes verifying an accuracy of the label.

### Claim 21 (depends on 15)

The system of claim 15, wherein:
the object is selected for labeling based at least on a determination that the object is observable from a pose of the vehicle at a time of capture of the image.