# US 2021/0080952 A1 · Obstacle detection and vehicle navigation using resolution-adaptive fusion of point clouds

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0080952 A1
- Document ID: US20210080952A1
- Status: Published application
- Kind: A1
- Application number: 16/570,139
- Filed: 2019-09-13
- Published: 2021-03-18
- Earliest claimed: 2019-09-13
- Assignees: The Boeing Company
- Applicants: The Boeing Company
- Inventors: Yuri Owechko
- Web page: https://patentgrep.com/patent/US20210080952A1

## CPC

- G01S 13/426: {Scanning radar, e.g. 3D radar (G01S13/66 takes precedence)}
- G01S 13/726: {Multiple target tracking}
- G01S 13/865: {Combination of radar systems with lidar systems}
- G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46)
- G01S 17/66: Tracking systems using electromagnetic waves other than radio waves
- G01S 17/87: Combinations of systems using electromagnetic waves other than radio waves
- G01S 17/931: of land vehicles
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G05D 1/0231: {using optical position detecting means (position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16)}
- G06F 18/251: {of input or preprocessed data}
- G06T 5/73: Deblurring; Sharpening
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G01C 3/08: Use of electric radiation detectors
- G01S 13/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S13/46)
- G01S 13/86: Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
- G01S 13/931: of land vehicles
- G01S 2013/9323: {Alternative operation using light waves}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30261: Obstacle
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)

## Abstract

A method for obstacle detection and navigation of a vehicle using resolution-adaptive fusion includes performing, by a processor, a resolution-adaptive fusion of at least a first three-dimensional (3D) point cloud and a second 3D point cloud to generate a fused, denoised, and resolution-optimized 3D point cloud that represents an environment associated with the vehicle. The first 3D point cloud is generated by a first-type 3D scanning sensor, and the second 3D point cloud is generated by a second-type 3D scanning sensor. The second-type 3D scanning sensor includes a different resolution in each of a plurality of different measurement dimensions relative to the first-type 3D scanning sensor. The method also includes detecting obstacles and navigating the vehicle using the fused, denoised, and resolution-optimized 3D point cloud.

## Claims

### Claim 1

A method for obstacle detection and navigation of a vehicle using resolution-adaptive fusion, the method comprising:
performing, by a processor, a resolution-adaptive fusion of at least a first three-dimensional (3D) point cloud and a second 3D point cloud to generate a fused, denoised, and resolution-optimized 3D point cloud that represents an environment associated with the vehicle, wherein the first 3D point cloud is generated by a first-type 3D scanning sensor and the second 3D point cloud is generated by a second-type 3D scanning sensor, and wherein the second-type 3D scanning sensor includes a different resolution in each of a plurality of different measurement dimensions relative to the first-type 3D scanning sensor; and
detecting obstacles and navigating the vehicle using the fused, denoised, and resolution-optimized 3D point cloud.

### Claim 2 (depends on 1)

The method of claim 1, wherein said performing the resolution-adaptive fusion comprises:
generating a first volumetric surface function by performing a 3D convolution of each measured point from a plurality of measured points of the first 3D point cloud with an associated 3D point spread function of the first-type 3D scanning sensor for representing an uncertainty in a spatial location of each measured point, wherein the first volumetric surface function incorporates a resolution of the first-type 3D scanning sensor; and
generating a second volumetric surface function by performing a 3D convolution of each measured point from a plurality of measured points of the second 3D point cloud with an associated 3D points spread function of the second-type 3D scanning sensor for representing an uncertainty in a spatial location of each measured point, wherein the second volumetric surface function incorporates a resolution of the second-type 3D scanning sensor.

### Claim 3 (depends on 2)

The method of claim 2, wherein said performing the resolution-adaptive fusion further comprises forming a 3D composite volumetric surface function by multiplying or adding the first volumetric surface function for the first-type 3D scanning sensor and the second volumetric surface function for the second-type 3D scanning sensor, wherein inaccurate point cloud data from one type 3D scanning sensor will be compensated by accurate point cloud data from the other type scanning sensor by said forming the 3D composite volumetric surface function.

### Claim 4 (depends on 3)

The method of claim 3, wherein said forming the 3D composite volumetric surface function comprises adding the first volumetric surface function and the second volumetric surface function in response to a condition that causes one of the 3D scanning sensors to be ineffective.

### Claim 5 (depends on 3)

The method of claim 3, wherein said forming the 3D composite volumetric surface function comprises multiplying the first volumetric surface function and the second volumetric surface function to enhance a resolution for detecting an obstacle in the environment associated with the vehicle compared to using a volumetric surface function of only one of the 3D scanning sensors.

### Claim 6 (depends on 3)

The method of claim 3, wherein said performing the resolution-adaptive fusion further comprises generating an iso-contour of the 3D composite volumetric surface function by performing automated edge-based thresholding to find a best resolution-adaptive iso-contour of the 3D composite volumetric surface function, wherein the automated edge-based thresholding is based on volumetric surface function edge map optimization.

### Claim 7 (depends on 6)

The method of claim 6, wherein said performing the automated edge-based thresholding comprises incrementing a threshold value over a preset range of values to determine the threshold value that maximizes a number of edges in a two-dimensional (2D) edge map of the iso-contour of the 3D composite volumetric surface function.

### Claim 8 (depends on 6)

The method of claim 6, wherein said performing the resolution-adaptive fusion further comprises resampling the iso-contour of the 3D composite volumetric surface function on a uniform grid to form the fused, denoised, and resolution-optimized 3D point cloud.

### Claim 9 (depends on 1)

The method of claim 1, further comprising presenting a representation of the environment associated with the vehicle using the fused, denoised, and resolution-optimized 3D point cloud.

### Claim 10 (depends on 9)

The method of claim 9, further comprising using the fused, denoised, and resolution-optimized 3D point cloud to detect and avoid the obstacles by the vehicle.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
generating a sequence of fused, denoised, and resolution-optimized 3D point clouds; and
tracking a moving obstacle from among the obstacles using the sequence of fused, denoised, and resolution-optimized 3D point clouds.

### Claim 12 (depends on 1)

The method of claim 1, wherein the first-type 3D scanning sensor comprises one of a radar, a stereo vision sensor, a monocular vision sensor, or a LIDAR sensor, and wherein the second-type 3D scanning sensor comprises a different-type sensor from the first-type 3D scanning sensor.

### Claim 13

A system for obstacle detection and navigation of a vehicle using resolution-adaptive fusion, the system comprising:
a processor; and
a memory associated with the processor, wherein the memory includes computer-readable program instructions that, when executed by the processor, causes the processor to perform a set of functions comprising:
performing a resolution-adaptive fusion of at least a first three-dimensional (3D) point cloud and a second 3D point cloud to generate a fused, denoised, and resolution-optimized 3D point cloud that represents an environment associated with the vehicle, wherein the first 3D point cloud is generated by a first-type 3D scanning sensor and the second 3D point cloud is generated by a second-type 3D scanning sensor, and wherein the second-type 3D scanning sensor includes a different resolution in each of a plurality of different measurement dimensions relative to the first-type 3D scanning sensor; and
detecting obstacles and navigating the vehicle using the fused, denoised, and resolution-optimized 3D point cloud.

### Claim 14 (depends on 13)

The system of claim 13, wherein said performing the resolution-adaptive fusion comprises:
generating a first volumetric surface function by performing a 3D convolution of each measured point from a plurality of measured points of the first 3D point cloud with an associated 3D point spread function of the first-type 3D scanning sensor for representing an uncertainty in a spatial location of each measured point, wherein the first volumetric surface function incorporates a resolution of the first-type 3D scanning sensor; and
generating a second volumetric surface function by performing a 3D convolution of each measured point from a plurality of measured points of the second 3D point cloud with an associated 3D points spread function of the second-type 3D scanning sensor for representing an uncertainty in a spatial location of each measured point, wherein the second volumetric surface function incorporates a resolution of the second-type 3D scanning sensor.

### Claim 15 (depends on 14)

The system of claim 14, wherein said performing the resolution-adaptive fusion further comprises forming a 3D composite volumetric surface function by multiplying or adding the first volumetric surface function for the first-type 3D scanning sensor and the second volumetric surface function for the second-type 3D scanning sensor, wherein inaccurate point cloud data from one type 3D scanning sensor will be compensated by accurate point cloud data from the other type scanning sensor by said forming the 3D composite volumetric surface function.

### Claim 16 (depends on 15)

The system of claim 15, wherein said forming the 3D composite volumetric surface function comprises adding the first volumetric surface function and the second volumetric surface function in response to a condition that causes one of the 3D scanning sensors to be ineffective.

### Claim 17 (depends on 15)

The system of claim 15, wherein said forming the 3D composite volumetric surface function comprises multiplying the first volumetric surface function and the second volumetric surface function to enhance a resolution for detecting an obstacle in the environment associated with the vehicle compared to using a volumetric surface function of only one of the 3D scanning sensors.

### Claim 18 (depends on 15)

The system of claim 15, wherein said performing the resolution-adaptive fusion further comprises generating an iso-contour of the 3D composite volumetric surface function by performing automated edge-based thresholding to find a best resolution-adaptive iso-contour of the 3D composite volumetric surface function, wherein the automated edge-based thresholding is based on volumetric surface function edge map optimization.

### Claim 19 (depends on 18)

The system of claim 18, wherein said performing the automated edge-based thresholding comprises increasing 614 a threshold value over a preset range of values to determine the threshold value that maximizes a number of edges in a two-dimensional (2D) edge map of the iso-contour of the 3D composite volumetric surface function.

### Claim 20 (depends on 18)

The system of claim 18, wherein said performing the resolution-adaptive fusion further comprises resampling the iso-contour of the 3D composite volumetric surface function on a uniform grid to form the fused, denoised, and resolution-optimized 3D point cloud.
