Artificial intelligence device for virtual multiview generation and fusion framework for semantic scene completion and method thereof

An artificial intelligence (AI) device, comprising:

• a memory configured to store information for information for a semantic scene completion model;
• and
• a controller configured to:
• receive a single two-dimensional (2D) image of a scene captured from an initial viewpoint,
• determine a plurality of virtual camera poses spherically distributed around a centroid of the scene at predetermined angular offsets relative to the initial viewpoint,
• generate, by a novel view synthesis model, a plurality of synthesized 2D images corresponding to the plurality of virtual camera poses,
• process, by the semantic scene completion model, the single 2D image and the plurality of synthesized 2D images to generate a plurality of corresponding three-dimensional (3D) scene predictions, and
• fuse, by a transformer-based multi-view fusion adaptor, the plurality of corresponding 3D scene predictions to generate a unified 3D semantic occupancy map,
• wherein the transformer-based multi-view fusion adaptor is configured to apply a self-attention mechanism utilizing spatial positional encoding and view positional encoding to resolve structural ambiguities in one or more occluded regions of the scene.

A non-transitory computer readable medium storing computer-executable instructions that when executed by a processor, cause the processor to perform the operations of:

• receiving a single two-dimensional (2D) image of a scene captured from an initial viewpoint;
• determining a plurality of virtual camera poses spherically distributed around a centroid of the scene at predetermined angular offsets relative to the initial viewpoint;
• generating, by a novel view synthesis model, a plurality of synthesized 2D images corresponding to the plurality of virtual camera poses;
• processing, by a semantic scene completion model, the single 2D image and the plurality of synthesized 2D images to generate a plurality of corresponding three-dimensional (3D) scene predictions;
• and
• fusing, by a transformer-based multi-view fusion adaptor, the plurality of corresponding 3D scene predictions to generate a unified 3D semantic occupancy map,
• wherein the fusing includes applying a self-attention mechanism utilizing spatial positional encoding and view positional encoding to resolve structural ambiguities in one or more occluded regions of the scene.