# US 12,000,965 B2 · Creating multi-return map data from single return lidar data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,000,965 B2
- Document ID: US12000965B2
- Status: Granted
- Kind: B2
- Application number: 17/497,314
- Filed: 2021-10-08
- Published: 2023-04-13
- Granted: 2024-06-04
- Earliest claimed: 2021-10-08
- Assignees: HERE Global B.V.
- Applicants: HERE Global B.V.
- Inventors: Deekshant Saxena; Senjuti Sen
- Web page: https://patentgrep.com/patent/US12000965B2

## CPC

- G01S 7/497: Means for monitoring or calibrating
- G01S 7/486: Receivers
- G01S 7/487: Extracting wanted echo signals {, e.g. pulse detection}
- G01S 17/10: using transmission of interrupted, pulse-modulated waves (determination of distance by phase measurements G01S17/32)
- G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46)
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/89: for mapping or imaging
- G01S 17/931: of land vehicles
- G06N 3/02: Neural networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/094: Adversarial learning
- G01S 7/4817: {relating to scanning}
- G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar
- G06F 16/29: Geographical information databases
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

System and methods for creating multi-return map data using single return Lidar data. The systems and methods use a long short-term memory (LSTM) model in combination with a Generative Adversarial Network (GAN) model. The systems and method use a single (1st) return of Lidar at a time stamp and create multiple unseen samples of 2nd and 3rd returns. The LSTM model is used to create a sequential calibration based on incidence angle to choose the optimized 2nd and 3rd return at the same instance of the time stamp. This creates a localized model of three returns from a single return of Lidar and thus provides additional data to generate an HD map.

## Claims

### Claim 1

A system comprising:
a memory configured to store single return Lidar data and real multi-return Lidar data;
a generative network configured to input the single return Lidar data and generate synthetic multi-return Lidar data; and
a discriminator network configured to distinguish the synthetic multi-return Lidar data from the real multi-return Lidar data;
wherein the generative network and discriminator network are iteratively trained using an adversarial approach where the generative network learns to generate more plausible synthetic multi-return Lidar data, while the discriminative network attempts to distinguish synthetic multi-return Lidar data produced by the generative network from the real multi-return Lidar data.

### Claim 2 (depends on 1)

The system of claim 1, wherein the real multi-return Lidar data comprises at least three levels of returns.

### Claim 3 (depends on 1)

The system of claim 1, wherein the generative network comprises a generator configured to generate 2nd and 3rd returns and a LSTM model configured to create a sequential calibration to select optimized 2nd and 3rd return for a time stamp.

### Claim 4 (depends on 3)

The system of claim 3, wherein the sequential calibration comprises a time, an intensity, and a derived distance.

### Claim 5 (depends on 1)

The system of claim 1, wherein the generative network is further configured to input noise with the single return Lidar data.

### Claim 6 (depends on 1)

The system of claim 1, wherein the discriminator network comprises a support vector machine.

### Claim 7 (depends on 1)

The system of claim 1, further comprising:
a vehicle comprising a single return Lidar device configured to acquire the single return Lidar data as the vehicle traverses a roadway.

### Claim 8 (depends on 7)

The system of claim 7, wherein the single return Lidar data is previously collected data from the roadway, wherein the synthetic multi-return Lidar data is used to generate a high-definition map without redriving the roadway.

### Claim 9 (depends on 1)

The system of claim 1, wherein the single return Lidar data comprises data for a location on a roadway for which real multi-return Lidar data is not stored in the memory.

### Claim 10

A method for training a generative network, the method comprising:
acquiring single return Lidar data and multi-return Lidar data;
inputting the single return Lidar data into the generative network configured to output synthetic multi-return Lidar data;
inputting, randomly, the multi-return Lidar data or the synthetic multi-return Lidar data into a discriminator network configured to attempt to classify the input as the multi-return Lidar data or the synthetic multi-return Lidar data;
adjusting the generative network and the discriminative network based on the classification provided by the discriminator network;
repeating inputting, inputting, and adjusting until convergence; and
outputting a trained generative network.

### Claim 11 (depends on 10)

The method of claim 10, wherein the synthetic multi-return Lidar data comprises at least three levels of returns.

### Claim 12 (depends on 10)

The method of claim 10, wherein the generative network comprises a generator configured to generate 2nd and 3rd returns and a LSTM model configured to create a sequential calibration to select optimized 2nd and 3rd return for a time stamp.

### Claim 13 (depends on 10)

The method of claim 10, wherein the generative network is further configured to input noise with the single return Lidar data.

### Claim 14 (depends on 10)

The method of claim 10, wherein the single return lidar data is acquired by a vehicle including at least a single return Lidar device as the vehicle traverses a roadway.

### Claim 15

A method for creating multi-return Lidar data from single return Lidar data, the method comprising:
acquiring single return Lidar data for a location;
inputting the single return Lidar data into a trained generative network configured using an adversarial process to generate synthetic multi-return Lidar data;
outputting, by the trained generative network, synthetic multi-return Lidar data for the location; and
updating a high-definition map with the synthetic multi-return Lidar data for the location.

### Claim 16 (depends on 15)

The method of claim 15, wherein the synthetic multi-return Lidar data comprises at least three levels of returns.

### Claim 17 (depends on 15)

The method of claim 15, wherein the single return lidar data is acquired by a vehicle including at least a single return Lidar device as the vehicle traverses the location.

### Claim 18 (depends on 15)

The method of claim 15, further comprising:
providing the high-definition map to one or more vehicles.

### Claim 19 (depends on 15)

The method of claim 15, wherein the generative network comprises a generator configured to generate 2nd and 3rd return data and a LSTM model configured to create a sequential calibration to select optimized 2nd and 3rd return data for a respective time stamp.

### Claim 20 (depends on 15)

The method of claim 15, wherein the single return Lidar data is acquired from a geographic database.