Scalable foundation model for three-dimensional scene understanding

An apparatus, comprising:

• one or more memories;
• processing circuitry in communication with the one or more memories, the processing circuitry configured to:
• obtain labeled three-dimensional sensor data;
• train a foundation model on three-dimensional scene understanding based on the labeled three-dimensional sensor data in a first training stage, wherein the foundation model comprises a multimodal large language model (LLM) encoder, a fusion transformer, and a transformer decoder;
• obtain unlabeled three-dimensional sensor data;
• and
• train the foundation model trained in the first training stage based on the unlabeled three-dimensional sensor data in a second training stage.

A method, comprising:

• obtaining three-dimensional sensor data;
• identifying one or more objects in the three-dimensional sensor data;
• determining one or more object features of the one or more objects;
• invoking a multimodal large language model (LLM) encoder that generates one or more text-based object descriptions of one or more objects in the three-dimensional sensor data;
• invoking a fusion transformer that combines object features and the one or more text-based object descriptions to produce a multimodal representation;
• and
• invoking a transformer decoder that performs a three-dimensional perception task based on multimodal representation.