# US 11,113,584 B2 · Single frame 4D detection using deep fusion of camera image, imaging RADAR and LiDAR point cloud

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,584 B2
- Document ID: US11113584B2
- Status: Granted
- Kind: B2
- Application number: 16/781,672
- Filed: 2020-02-04
- Published: 2021-08-05
- Granted: 2021-09-07
- Earliest claimed: 2020-02-04
- Assignees: NIO USA, Inc.
- Applicants: NIO USA, Inc.
- Inventors: Huazeng Deng; Ajaya H S Rao; Ashwath Aithal; Xu Chen; Ruoyu Tan; Veera Ganesh Yalla
- Web page: https://patentgrep.com/patent/US11113584B2

## CPC

- G01S 7/40: Means for monitoring or calibrating
- G01S 7/41: using analysis of echo signal for target characterisation; Target signature; Target cross-section
- G01S 7/417: {involving the use of neural networks}
- G01S 7/4802: {using analysis of echo signal for target characterisation; Target signature; Target cross-section}
- G01S 7/497: Means for monitoring or calibrating
- G01S 13/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S13/46)
- G01S 13/589: {measuring the velocity vector}
- G01S 13/862: {Combination of radar systems with sonar systems}
- G01S 13/865: {Combination of radar systems with lidar systems}
- G01S 13/867: {Combination of radar systems with cameras}
- G01S 13/87: Combinations of radar systems, e.g. primary radar and secondary radar
- G01S 13/931: of land vehicles
- G01S 17/58: Velocity or trajectory determination systems; Sense-of-movement determination systems
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/89: for mapping or imaging
- G01S 17/931: of land vehicles
- G06F 18/24: Classification techniques
- G06F 18/253: {of extracted features}
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/62: of area, perimeter, diameter or volume
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G01S 13/726: {Multiple target tracking}
- G01S 13/89: for mapping or imaging
- G01S 2013/9316: {combined with communication equipment with other vehicles or with base stations}
- G01S 2013/9318: {Controlling the steering}
- G01S 2013/93185: {Controlling the brakes}
- G01S 2013/9319: {Controlling the accelerator}
- G01S 2013/932: {using own vehicle data, e.g. ground speed, steering wheel direction}
- G01S 2013/9322: {using additional data, e.g. driver condition, road state or weather data}
- G01S 2013/93271: {in the front of the vehicles}
- G01S 2013/93272: {in the back of the vehicles}
- G01S 2013/93273: {on the top of the vehicles}
- G01S 2013/93274: {on the side of the vehicles}
- G01S 2013/93277: {in the lights}
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G05D 1/0231: {using optical position detecting means (position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16)}
- G05D 1/0251: {extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision (stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593)}
- G05D 1/0257: {using a radar (radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931)}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10044: Radar image
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES

## Abstract

Embodiments of the present disclosure are directed to a method for object detection. The method includes receiving sensor data indicative of one or more objects for each of a camera subsystem, a LiDAR subsystem, and an imaging RADAR subsystem. The sensor data is received simultaneously and within one frame for each of the subsystems. The method also includes extracting one or more feature representations of the objects from camera image data, LiDAR point cloud data and imaging RADAR point cloud data and generating image feature maps, LiDAR feature maps and imaging RADAR feature maps. The method further includes combining the image feature maps, the LiDAR feature maps and the imaging RADAR feature maps to generate merged feature maps and generating object classification, object position, object dimensions, object heading and object velocity from the merged feature maps.

## Claims

### Claim 1

A method for object detection, the method comprising:
receiving, by a processor, sensor data indicative of one or more objects for each of a camera subsystem, a LiDAR subsystem, and an imaging RADAR subsystem,
wherein the sensor data includes camera image data, LiDAR point cloud data and imaging RADAR point cloud data and the sensor data is received simultaneously and within one frame for each of the camera subsystem, the LiDAR subsystem, and the imaging RADAR subsystem;
extracting, by the processor, one or more feature representations of the objects from the camera image data, LiDAR point cloud data and imaging RADAR point cloud data;
generating, by the processor, image feature maps from the extracted one or more feature representations of the objects from the camera image data, LiDAR feature maps from the LiDAR point cloud data, and imaging RADAR feature maps from the image RADAR point cloud data;
combining, by the processor, the image feature maps, the LiDAR feature maps, and the imaging RADAR feature maps to generate merged feature maps; and
generating, by the processor, object classification, object position, object dimensions, object heading and object velocity from the merged feature maps.

### Claim 2 (depends on 1)

The method of claim 1, wherein each of the image feature maps, LiDAR feature maps and imaging RADAR feature maps are feature vectors that have a same dimension of width, length and a number of channels.

### Claim 3 (depends on 1)

The method of claim 1, wherein the processor includes a feature extractor algorithm to extract the one or more feature representations of the objects from the camera image data, LiDAR point cloud data and imaging RADAR point cloud data and a classifier and regressor algorithm to generate the object classification, the object position, the object dimensions, the object heading and the object velocity.

### Claim 4 (depends on 3)

The method of claim 3, wherein the feature extractor algorithm includes a Deep Neural Network (DNN) algorithm, a Histogram of Oriented Gradients (HOG) algorithm, a Scale Invariant Feature Transform (SIFT) algorithm or a Speeded-Up Robust Feature (SURF) algorithm.

### Claim 5 (depends on 3)

The method of claim 3, wherein the classifier and regressor algorithm includes a Deep Neural Network (DNN) algorithm, a Decision Tree (DT) algorithm or a Support Vector Machine (SVM) algorithm.

### Claim 6 (depends on 1)

The method of claim 1, further comprising initializing and calibrating, by the processor, sensors from each of the camera subsystem, the LiDAR subsystem, and the imaging RADAR subsystem.

### Claim 7 (depends on 6)

The method of claim 6, wherein calibrating the sensors from each of the camera subsystem, the LiDAR subsystem, and the imaging RADAR subsystem includes intrinsic and extrinsic calibrations.

### Claim 8

A method for object detection, the method comprising:
receiving, by a processor, sensor data indicative of one or more objects for each of a camera subsystem, a LiDAR subsystem, and an imaging RADAR subsystem,
wherein the sensor data includes camera image data, LiDAR point cloud data and imaging RADAR point cloud data and the sensor data is received simultaneously and within one frame for each of the camera subsystem, the LiDAR subsystem, and the imaging RADAR subsystem;
combining, by the processor, the camera image data, LiDAR point cloud data and imaging RADAR point cloud data to create fused raw data;
extracting, by the processor, one or more feature representations of the objects from the fused raw data;
generating, by the processor, fused feature maps from the extracted one or more feature representations of the objects from the fused raw data; and
generating, by the processor, object classification, object position, object dimensions, object heading and object velocity from the fused feature maps.

### Claim 9 (depends on 8)

The method of claim 8, wherein the processor includes a feature extractor algorithm to extract the one or more feature representations of the objects from fused raw data and a classifier and regressor algorithm to generate the object classification, the object position, the object dimensions, the object heading and the object velocity.

### Claim 10 (depends on 9)

The method of claim 9, wherein the feature extractor algorithm includes a Deep Neural Network (DNN) algorithm, a Histogram of Oriented Gradients (HOG) algorithm, a Scale Invariant Feature Transform (SIFT) algorithm or a Speeded-Up Robust Feature (SURF) algorithm.

### Claim 11 (depends on 9)

The method of claim 9, wherein the classifier and regressor algorithm includes a Deep Neural Network (DNN) algorithm, a Decision Tree (DT) algorithm or a Support Vector Machine (SVM) algorithm.

### Claim 12 (depends on 8)

The method of claim 8, further comprising initializing and calibrating, by the processor, sensors from each of the camera subsystem, the LiDAR subsystem, and the imaging RADAR subsystem.

### Claim 13 (depends on 12)

The method of claim 12, wherein calibrating the sensors from each of the camera subsystem, the LiDAR subsystem, and the imaging RADAR subsystem includes intrinsic and extrinsic calibrations.

### Claim 14

A vehicle control system, comprising:
a processor; and
a memory coupled with and readable by the processor and storing therein a set of instructions, which when executed by the processor, cause the processor to detect objects in a single frame by:
receiving sensor data indicative of one or more objects for each of a camera subsystem, a LiDAR subsystem, and an imaging RADAR subsystem;
wherein the sensor data includes camera image data, LiDAR point cloud data and imaging RADAR point cloud data and the sensor data is received simultaneously and within one frame for each of the camera subsystem, the LiDAR subsystem, and the imaging RADAR subsystem;
extracting one or more feature representations of the objects from the camera image data, LiDAR point cloud data and imaging RADAR point cloud data;
generating image feature maps from the extracted one or more feature representations of the objects from the camera image data, LiDAR feature maps from the LiDAR point cloud data, and imaging RADAR feature maps from the image RADAR point cloud data;
combining the image feature maps, the LiDAR feature maps, and the imaging RADAR feature maps to generate merged feature maps; and
generating object classification, object position, object dimensions, object heading and object velocity from the merged feature maps.

### Claim 15 (depends on 14)

The vehicle control system of claim 14, wherein each of the image feature maps, LiDAR feature maps and imaging RADAR feature maps are feature vectors that have a same dimension of width, length and a number of channels.

### Claim 16 (depends on 14)

The vehicle control system of claim 14, wherein the processor includes a feature extractor algorithm to extract the one or more feature representations of the objects from the camera image data, LiDAR point cloud data and imaging RADAR point cloud data and a classifier and regressor algorithm to generate the object classification, the object position, the object dimensions, the object heading and the object velocity.

### Claim 17 (depends on 16)

The vehicle control system of claim 16, wherein the feature extractor algorithm includes a Deep Neural Network (DNN) algorithm, a Histogram of Oriented Gradients (HOG) algorithm, a Scale Invariant Feature Transform (SIFT) algorithm or a Speeded-Up Robust Feature (SURF) algorithm.

### Claim 18 (depends on 16)

The vehicle control system of claim 16, wherein the classifier and regressor algorithm includes a Deep Neural Network (DNN) algorithm, a Decision Tree (DT) algorithm or a Support Vector Machine (SVM) algorithm.

### Claim 19 (depends on 14)

The vehicle control system of claim 14, further comprising initializing and calibrating sensors from each of the camera subsystem, the LiDAR subsystem, and the imaging RADAR subsystem.

### Claim 20 (depends on 19)

The vehicle control system of claim 19, wherein calibrating the sensors from each of the camera subsystem, the LiDAR subsystem, and the imaging RADAR subsystem includes intrinsic and extrinsic calibrations.