Method, apparatus and system for grounding intermediate representations with foundational ai models for environment understanding

An apparatus for developing an understanding of at least one perceived environment, comprising:

• a processor;
• and
• a memory accessible to the processor, the memory having stored therein at least one of programs or instructions executable by the processor to configure the apparatus to:
• determine semantic features and respective positional information of the semantic features from received data related to images and respective depth-related content of the at least one perceived environment on the fly as changes in the received data occur;
• for each of the at least one perceived environments, combine information of the determined semantic features with the respective positional information to generate a compact intermediate representation of the perceived environment which provides information regarding positions of the semantic features in the perceived environment and at least spatial relationships among the sematic features;
• for each of the at least one perceived environments, combine information from the determined intermediate representation with information stored in a foundational model to determine a respective understanding of the perceived environment;
• and
• output an indication of the determined respective understanding of the perceived environment.

A system for developing an understanding of at least one perceived environment, comprising:

• a foundational model;
• and
• at least one machine agent, comprising:
• a pre-processing module;
• a graphing module;
• a processor;
• and
• a memory accessible to the processor, the memory having stored therein at least one of programs or instructions executable by the processor to configure the machine agent to:
• using the pre-processor module, determine semantic features and respective positional information of the semantic features from received data related to images and respective depth-related content of the at least one perceived environment on the fly as changes in the received data occur;
• using the graphing module, for each of the at least one perceived environments, combine information of the determined semantic features with the respective positional information to generate a compact intermediate representation of the perceived environment which provides information regarding positions of the semantic features in the perceived environment and at least spatial relationships among the sematic features;
• using the processor, for each of the at least one perceived environments, combine information from the determined intermediate representation with information stored in a foundational model to determine a respective understanding of the perceived environment;
• and
• output an indication of the determined respective understanding of the perceived environment.

A non-transitory computer readable storage medium having stored thereon instructions that when executed by a processor perform a method for developing an understanding of at least one perceived environment, comprising:

• determining semantic features and respective positional information of the semantic features from received data related to images and respective depth-related content of the at least one perceived environment on the fly as changes in the received data occur;
• for each of the at least one perceived environments, combining information of the determined semantic features with the respective positional information to generate a compact intermediate representation of the perceived environment which provides information regarding positions of the semantic features in the perceived environment and at least spatial relationships among the sematic features;
• for each of the at least one perceived environments, combining information from the determined intermediate representation with information stored in a foundational model to determine a respective understanding of the perceived environment;
• and
• outputting an indication of the determined respective understanding of the perceived environment.