Three-dimensional volumetric scene completion for multi-camera systems

An apparatus, comprising:

• a memory storing processor-readable code;
• and
• at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations including:
• receiving a plurality of rays determined based on a plurality of image frames representing a scene around a plurality of cameras, wherein each of the plurality of rays is associated with respective temporal information;
• determining, from a set of objects depicted by the plurality of image frames, a first object as stationary relative to the plurality of cameras and a second object as moving relative to the plurality of cameras;
• determining, using a machine learning (ML) model and based on the plurality of rays and the respective temporal information, a first feature set representing portions of the first object that correspond to a blind spot of the plurality of cameras such that the portions of the first object are absent from the plurality of image frames;
• determining, using the ML model and based on a mask and the plurality of rays and the respective temporal information, a second feature set representing portions of the second object that are absent from the plurality of image frames;
• and
• determining a three-dimensional representation of the scene based on the first feature set and the second feature set.

A non-transitory computer-readable medium storing instructions that, when executed by a processing system that includes one or more processors, cause the processing system to perform operations comprising:

• receiving a plurality of rays determined based on a plurality of image frames representing a scene around a plurality of cameras, wherein each of the plurality of rays is associated with respective temporal information;
• determining, from a set of objects depicted by the plurality of image frames, a first object as stationary relative to the plurality of cameras and a second object as moving relative to the plurality of cameras;
• determining, using a machine learning (ML) model and based on the plurality of rays and the respective temporal information, a first feature set representing portions of the first object that correspond to a blind spot of the plurality of cameras such that the portions of the first object are absent from the plurality of image frames;
• determining, using the ML model and based on a mask and the plurality of rays and the respective temporal information, a second feature set representing portions of the second object that are absent from the plurality of image frames;
• and
• determining a three-dimensional representation of the scene based on the first feature set and the second feature set.

A vehicle, comprising:

• a plurality of cameras;
• a memory storing processor-readable code;
• and
• at least one processor coupled to the memory, the at least one processor in communication with the plurality of cameras and configured to execute the processor-readable code to cause the at least one processor to perform operations including:
• receiving a plurality of rays determined based on a plurality of image frames representing a scene around the plurality of cameras, wherein each of the plurality of rays is associated with respective temporal information;
• determining, from a set of objects depicted by the plurality of image frames, a first object as stationary relative to the plurality of cameras and a second object as moving relative to the plurality of cameras;
• determining, using a machine learning (ML) model and based on the plurality of rays and the respective temporal information, a first feature set representing portions of the first object that correspond to a blind spot of the plurality of cameras such that the portions of the first object are absent from the plurality of image frames;
• determining, using the ML model and based on a mask and the plurality of rays and the respective temporal information, a second feature set representing portions of the second object that are absent from the plurality of image frames;
• and
• determining a three-dimensional representation of the scene based on the first feature set and the second feature set.