LiDAR-camera spatio-temporal alignment using neural radiance fields

An apparatus, comprising:

• a memory storing processor readable code;
• and
• at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations including:
• receiving first kinematic information associated with an image sensor of a camera;
• receiving point cloud data from a light detection and ranging (LiDAR) sensor captured at a first time point;
• receiving image data from the image sensor of the camera captured at a second time point that is different than the first time point;
• determining a pose of the image sensor based on the first kinematic information, including the first time point, the second time point, and a velocity of an object in a field of view of the image sensor, wherein the pose corresponds to the first time point;
• generating first image data that is time-synchronized with the point cloud data based on the first kinematic information and a neural radiance fields (NeRF) model, wherein generating the first image data includes inputting the pose into the NeRF model such that the NeRF model outputs the first image data, and wherein the first image data corresponds to the first time point;
• performing artifact removal on the first image data based on the first image data and the image data captured at the second time point;
• receiving second kinematic information associated with the LiDAR sensor;
• generating modified point cloud data based on the point cloud data and the second kinematic information;
• and
• generating, after performing the artifact removal on the first image data, fused data that combines the first image data and the modified point cloud data.