US 11,768,292 B2 · Three-dimensional object detection

Abstract

Generally, the disclosed systems and methods implement improved detection of objects in three-dimensional (3D) space. More particularly, an improved 3D object detection system can exploit continuous fusion of multiple sensors and/or integrated geographic prior map data to enhance effectiveness and robustness of object detection in applications such as autonomous driving. In some implementations, geographic prior data (e.g., geometric ground and/or semantic road features) can be exploited to enhance three-dimensional object detection for autonomous vehicle applications. In some implementations, object detection systems and methods can be improved based on dynamic utilization of multiple sensor modalities. More particularly, an improved 3D object detection system can exploit both LIDAR systems and cameras to perform very accurate localization of objects within three-dimensional space relative to an autonomous vehicle. For example, multi-sensor fusion can be implemented via continuous convolutions to fuse image data samples and LIDAR feature maps at different levels of resolution.

Claims20

Dependent claims

An autonomous vehicle (AV) control system, comprising:

one or more processors;
one or more non-transitory computer-readable media that store instructions for execution by the one or more processors that cause the AV control system to perform operations, the operations comprising:
(i) fusing a three-dimensional (3D) representation of light detection and ranging (LIDAR) data associated with an environment of an autonomous vehicle with a 3D representation of geographic map data to create a 3D representation of map-modified LIDAR data;
(ii) fusing a two-dimensional (2D) representation of image data associated with the environment of the autonomous vehicle with the 3D representation of map-modified LIDAR data to create a 3D feature map comprising image features from the 2D representation of the image data and map-modified LIDAR features from the 3D representation of map-modified LIDAR data, wherein (ii) comprises:
processing the 2D representation of the image data and the 3D representation of the map-modified LIDAR data by a machine-learned neural network that comprises one or more fusion layers;
and
receiving, as an output of the machine-learned neural network, the 3D feature map;
and
(iii) determining one or more object detections in the environment of the autonomous vehicle based on the 3D feature map.

6 more elements

An autonomous vehicle, comprising:

a camera system configured to capture image data associated with an environment of an autonomous vehicle;
a RADAR system configured to capture radar data associated with the environment of the autonomous vehicle;
a LIDAR system configured to capture LIDAR point cloud data representing a three-dimensional (3D) view of the environment of the autonomous vehicle;
a map system configured to provide geographic data associated with the environment of the autonomous vehicle;
a fusion system configured to:
(i) fuse the LIDAR point cloud data with the radar data and the geographic data into a 3D representation of map-modified sensor data;
and (ii) fuse image features from the image data with map-modified sensor features from the map-modified sensor data to generate a 3D feature map comprising the fused image features and map-modified sensor features, wherein the fusion system is configured for (ii) to:
provide the image data and the 3D representation of the map-modified sensor data as input to a machine-learned neural network that comprises one or more fusion layers, and receive, as an output of the machine-learned neural network, the 3D feature map;
an object detector configured to determine one or more object detections based on the 3D feature map.

6 more elements

Dependencies are inferred from the claim text.

Three-dimensional object detection

Life of this patent

Three-dimensional object detection

Life of this patent

Abstract

Claims20