# US 2022/0091266 A1 · Systems and methods for enhancing outputs of a lidar

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0091266 A1
- Document ID: US20220091266A1
- Status: Published application
- Kind: A1
- Application number: 17/025,730
- Filed: 2020-09-18
- Published: 2022-03-24
- Earliest claimed: 2020-09-18
- Assignees: DENSO International America, Inc.; Denso Corporation
- Applicants: Denso International America, Inc.; DENSO Corporation
- Inventors: Prasanna Kumar Sivakumar
- Web page: https://patentgrep.com/patent/US20220091266A1

## CPC

- G01S 17/89: for mapping or imaging
- B60W 60/001: {Planning or execution of driving tasks}
- G01S 7/4808: {Evaluating distance, position or velocity data}
- G01S 7/4816: {of receivers alone}
- G01S 17/931: of land vehicles
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06T 3/4046: using neural networks
- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/80: Geometric correction
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2420/408: Radar; Laser, e.g. lidar
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems, methods, and other embodiments described herein relate to enhancing an output of a LiDAR sensor. In one embodiment, a method includes acquiring, using the LiDAR sensor, a point cloud and an ambient image. The method includes processing the point cloud and the ambient image to generate enhanced data according to an enhancer pipeline that includes a set of processing routines. The method includes providing the enhanced data to at least one module to facilitate generating determinations about a surrounding environment of the LiDAR sensor from the enhanced data.

## Claims

### Claim 1

A refinement system for enhancing an output of a LiDAR sensor, comprising:
one or more processors;
a memory communicably coupled to the one or more processors and storing:
a sensor module including instructions that when executed by the one or more processors cause the one or more processors to acquire, using the LiDAR sensor, a point cloud and an ambient image; and
a pipeline module including instructions that when executed by the one or more processors cause the one or more processors to process the point cloud and the ambient image to generate enhanced data according to an enhancer pipeline that includes a set of processing routines,
wherein the pipeline module includes instructions to provide the enhanced data to at least one module to facilitate generating determinations about a surrounding environment of the LiDAR sensor from the enhanced data.

### Claim 2 (depends on 1)

The refinement system of claim 1, wherein the pipeline module includes instructions to provide the enhanced data to the at least one module including instructions to provide the enhanced data to an automated driving module to facilitate control of a vehicle, and
wherein the pipeline module includes instructions to process the point cloud and the ambient image according to the enhancer pipeline including instructions to at least correct aberrations in the ambient image to generate the enhanced data with a visible light image derived from at least the ambient image.

### Claim 3 (depends on 1)

The refinement system of claim 1, wherein the pipeline module includes instructions to process the point cloud and the ambient image including instructions to super-resolve the ambient image from a low-resolution image into a high-resolution image according to a super-resolution neural network.

### Claim 4 (depends on 1)

The refinement system of claim 1, wherein the pipeline module includes instructions to process the point cloud and the ambient image including instructions to predict portions of the ambient image subject to aberrations using a high-resolution image derived from the ambient image and a generative adversarial network (GAN), and
wherein the pipeline module includes instructions to predict the portions including instructions to predict sections of the ambient image that are missing as a result of the aberrations.

### Claim 5 (depends on 1)

The refinement system of claim 1, wherein the pipeline module includes instructions to process the point cloud and the ambient image including instructions to colorize the ambient image by converting a high-resolution image derived from the ambient image into an RGB image using a convolutional neural network (CNN).

### Claim 6 (depends on 1)

The refinement system of claim 1, wherein the pipeline module includes instructions to process the point cloud and the ambient image including instructions to densify the point cloud using a high-resolution image that is colorized and derived from the ambient image according to a deep neural network (DNN).

### Claim 7 (depends on 1)

The refinement system of claim 1, wherein the sensor module includes instructions to acquire and process the point cloud and the ambient image including instructions to acquire a batch of point clouds and ambient images from multiple sensors and batch processing the batch to generate the enhanced data together for the batch.

### Claim 8 (depends on 1)

The refinement system of claim 1, wherein the pipeline module includes instructions to process the point cloud and the ambient image according to the enhancer pipeline including instructions to super-resolve the ambient image into a high-resolution image, colorize the high-resolution image, complete the high-resolution image to fill-in missing portions, and densify the point cloud,
wherein the sensor module includes instructions to acquire the point cloud and the ambient image including instructions to control the LiDAR sensor to acquire the point cloud and the ambient image together, and wherein the LiDAR sensor is a single-photon avalanche diode (SPAD) LiDAR.

### Claim 9

A non-transitory computer-readable medium storing instructions for enhancing an output of a LiDAR sensor and that when executed by one or more processors, cause the one or more processors to:
acquire, using the LiDAR sensor, a point cloud and an ambient image;
process the point cloud and the ambient image to generate enhanced data according to an enhancer pipeline that includes a set of processing routines; and
provide the enhanced data to at least one module to facilitate generating determinations about a surrounding environment of the LiDAR sensor from the enhanced data.

### Claim 10 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the instructions to provide the enhanced data to the at least one module include instructions to provide the enhanced data to an automated driving module to facilitate control of a vehicle, and
wherein the instructions to process the point cloud and the ambient image according to the enhancer pipeline include instructions to at least correct aberrations in the ambient image to generate the enhanced data with a visible light image derived from at least the ambient image.

### Claim 11 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the instructions include instructions to process the point cloud and the ambient image according to the enhancer pipeline include instructions to super-resolve the ambient image into a high-resolution image, colorize the high-resolution image, complete the high-resolution image to fill-in missing portions, and densify the point cloud.

### Claim 12 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the instructions to acquire the point cloud and the ambient image include instructions to control the LiDAR sensor to acquire the point cloud and the ambient image together, and wherein the LiDAR sensor is a single-photon avalanche diode (SPAD) LiDAR.

### Claim 13

A method of enhancing an output of a LiDAR sensor, comprising:
acquiring, using the LiDAR sensor, a point cloud and an ambient image;
processing the point cloud and the ambient image to generate enhanced data according to an enhancer pipeline that includes a set of processing routines; and
providing the enhanced data to at least one module to facilitate generating determinations about a surrounding environment of the LiDAR sensor from the enhanced data.

### Claim 14 (depends on 13)

The method of claim 13, wherein providing the enhanced data to the at least one module includes providing the enhanced data to an automated driving module to facilitate control of a vehicle, and
wherein processing the point cloud and the ambient image according to the enhancer pipeline includes at least correcting aberrations in the ambient image to generate the enhanced data with a visible light image derived from at least the ambient image.

### Claim 15 (depends on 13)

The method of claim 13, wherein processing the point cloud and the ambient image includes super resolving the ambient image from a low-resolution image into a high-resolution image according to a super-resolution neural network.

### Claim 16 (depends on 13)

The method of claim 13, wherein processing the point cloud and the ambient image includes predicting portions of the ambient image subject to aberrations using a high-resolution image derived from the ambient image and a generative adversarial network (GAN), wherein predicting the portions includes predicting sections of the ambient image that are missing as a result of the aberrations.

### Claim 17 (depends on 13)

The method of claim 13, wherein processing the point cloud and the ambient image includes colorizing the ambient image by converting a high-resolution image derived from the ambient image into an RGB image using a convolutional neural network (CNN).

### Claim 18 (depends on 13)

The method of claim 13, wherein processing the point cloud and the ambient image includes densifying the point cloud using a high-resolution image that is colorized and derived from the ambient image according to a deep neural network (DNN).

### Claim 19 (depends on 13)

The method of claim 13, wherein acquiring and processing the point cloud and the ambient image includes acquiring a batch of point clouds and ambient images from multiple sensors and batch processing the batch to generate the enhanced data together for the batch.

### Claim 20 (depends on 13)

The method of claim 13, wherein processing the point cloud and the ambient image according to the enhancer pipeline includes super-resolving the ambient image into a high-resolution image, colorizing the high-resolution image, completing the high-resolution image to fill-in missing portions, and densifying the point cloud,
wherein acquiring the point cloud and the ambient image includes controlling the LiDAR sensor to acquire the point cloud and the ambient image together, and wherein the LiDAR sensor is a single-photon avalanche diode (SPAD) LiDAR.
