Nerf-based multi-sensor data fusion for vehicle applications
AssigneeQUALCOMM Incorporated
InventorsVenkatraman Narayanan, Varun Ravi Kumar, Senthil Kumar Yogamani
Life of this patent
Estimating the term…
Abstract
This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method of image processing includes receiving a plurality of image frames representative of a scene; receiving point cloud data representative of the scene; determining, using a NeRF model, a three-dimensional reconstruction of the scene based on the plurality of image frames; and outputting fused data that combines first BEV features of the three-dimensional reconstruction of the scene and second BEV features of the point cloud data. Other aspects and features are also claimed and described.
Claims30
A method for image processing for use in a vehicle assistance system, comprising:
- receiving a plurality of image frames representative of a scene;
- receiving point cloud data representative of the scene;
- determining, using a neural radiance field (NeRF) model, a three-dimensional reconstruction of the scene based on the plurality of image frames;
- and
- outputting fused data that combines first bird's eye view (BEV) features of the three-dimensional reconstruction of the scene and second BEV features of the point cloud data.
An apparatus, comprising:
- a processing system that includes one or more processors and one or more memories coupled with the one or more processors, the processing system configured to perform operations including:
- receiving a plurality of image frames representative of a scene;
- receiving point cloud data representative of the scene;
- determining, using a neural radiance field (NeRF) model, a three-dimensional reconstruction of the scene based on the plurality of image frames;
- and
- outputting fused data that combines first bird's eye view (BEV) features of the three-dimensional reconstruction of the scene and second BEV features of the point cloud data.
A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:
- receiving a plurality of image frames representative of a scene;
- receiving point cloud data representative of the scene;
- determining, using a neural radiance field (NeRF) model, a three-dimensional reconstruction of the scene based on the plurality of image frames;
- and
- outputting fused data that combines first bird's eye view (BEV) features of the three-dimensional reconstruction of the scene and second BEV features of the point cloud data.
A vehicle, comprising:
- a plurality of cameras comprising a plurality of image sensors;
- a ranging sensor;
- and
- a processing system that includes one or more processors and one or more memories coupled with the one or more processors, the processing system configured to perform operations including:
- receiving, from the plurality of image sensors, a plurality of image frames representative of a scene;
- receiving, from the ranging sensor, point cloud data representative of the scene;
- determining, using a neural radiance field (NeRF) model, a three-dimensional reconstruction of the scene based on the plurality of image frames;
- and
- outputting fused data that combines first bird's eye view (BEV) features of the three-dimensional reconstruction of the scene and second BEV features of the point cloud data.
Dependencies are inferred from the claim text.