# US 2025/0094675 A1 · Method, apparatus and system for grounding intermediate representations with foundational ai models for environment understanding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094675 A1
- Document ID: US20250094675A1
- Status: Published application
- Kind: A1
- Application number: 18/884,473
- Filed: 2024-09-13
- Published: 2025-03-20
- Earliest claimed: 2023-09-14
- Assignees: SRI International
- Applicants: SRI International
- Inventors: Han-Pang CHIU; Karan SIKKA; Louise YARNELL; Supun SAMARASEKERA; Rakesh KUMAR
- Web page: https://patentgrep.com/patent/US20250094675A1

## CPC

- G06F 30/27: using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
- G06F 30/13: Architectural design, e.g. computer-aided architectural design [CAAD] related to design of buildings, bridges, landscapes, production plants or roads
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method, apparatus, and system for developing an understanding of at least one perceived environment includes determining semantic features and respective positional information of the semantic features from received data related to images and respective depth-related content of the at least one perceived environment on the fly as changes in the received data occur, for each perceived environment, combining information of the determined semantic features with the respective positional information to determine a compact representation of the perceived environment which provides information regarding positions of the semantic features in the perceived environment and at least spatial relationships among the sematic features, for each of the at least one perceived environments, combining information from the determined intermediate representation with information stored in a foundational model to determine a respective understanding of the perceived environment, and outputting an indication of the determined respective understanding.

## Claims

### Claim 1

A method for developing an understanding of at least one perceived environment, comprising:
determining semantic features and respective positional information of the semantic features from received data related to images and respective depth-related content of the at least one perceived environment on the fly as changes in the received data occur;
for each of the at least one perceived environments, combining information of the determined semantic features with the respective positional information to generate a compact intermediate representation of the perceived environment which provides information regarding positions of the semantic features in the perceived environment and at least spatial relationships among the sematic features;
for each of the at least one perceived environments, combining information from the determined intermediate representation with information stored in a foundational model to determine a respective understanding of the perceived environment; and
outputting an indication of the determined respective understanding of the perceived environment.

### Claim 2 (depends on 1)

The method of claim 1, wherein the data related to the images and the respective depth-related content of the at least one perceived environment is received from at least one of a human, an agent or at least one sensor capable of capturing image content and depth information of the at least one perceived environment.

### Claim 3 (depends on 1)

The method of claim 1, wherein the intermediate representation comprises at least one of a hierarchical scene graph, which encodes the semantic features with their 3D spatial-temporal relationships across multiple levels or a hierarchical knowledge graph, which models at least one agent used in developing an understanding of the at least one perceived environment and capabilities of the at least one agent.

### Claim 4 (depends on 1)

The method of claim 1, wherein the foundational model comprises a large language model, which provides common sense knowledge for the at least one perceived environment.

### Claim 5 (depends on 1)

The method of claim 1, wherein the indication comprises at least one of a representation of the perceived environment annotated using information from the determined respective understanding, a navigation plan to enable an agent to navigate the perceived environment determined from the determined respective understanding, or a next step of a task to be completed by the agent in the perceived environment.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
navigating an agent through the at least one perceived environment using the developed understanding of the at least one perceived environment to locate at least one of an object or a location in the at least one perceived environment.

### Claim 7 (depends on 1)

The method of claim 1, wherein the compact intermediate representation of the perceived environment is generated using at least one of a neural network or predetermined rules.

### Claim 8

An apparatus for developing an understanding of at least one perceived environment, comprising:
a processor; and
a memory accessible to the processor, the memory having stored therein at least one of programs or instructions executable by the processor to configure the apparatus to:
determine semantic features and respective positional information of the semantic features from received data related to images and respective depth-related content of the at least one perceived environment on the fly as changes in the received data occur;
for each of the at least one perceived environments, combine information of the determined semantic features with the respective positional information to generate a compact intermediate representation of the perceived environment which provides information regarding positions of the semantic features in the perceived environment and at least spatial relationships among the sematic features;
for each of the at least one perceived environments, combine information from the determined intermediate representation with information stored in a foundational model to determine a respective understanding of the perceived environment; and
output an indication of the determined respective understanding of the perceived environment.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the data related to the images and the respective depth-related content of the at least one perceived environment is received from at least one of a human, an agent or at least one sensor capable of capturing image content and depth information of the at least one perceived environment.

### Claim 10 (depends on 8)

The apparatus of claim 8, wherein the intermediate representation comprises at least one of a hierarchical scene graph, which encodes the semantic features with their 3D spatial-temporal relationships across multiple levels or a hierarchical knowledge graph, which models at least one agent used in developing an understanding of the at least one perceived environment and capabilities of the at least one agent.

### Claim 11 (depends on 8)

The apparatus of claim 8, wherein the foundational model comprises a large language model, which provides common sense knowledge for the at least one perceived environment.

### Claim 12 (depends on 8)

The apparatus of claim 8, wherein the indication comprises at least one of a representation of the perceived environment annotated using information from the determined respective understanding, a navigation plan to enable an agent to navigate the perceived environment determined from the determined respective understanding, or a next step of a task to be completed by the agent in the perceived environment.

### Claim 13 (depends on 8)

The apparatus of claim 8, wherein the apparatus is further configured to:
navigate an agent through the at least one perceived environment using the developed understanding of the at least one perceived environment to locate at least one of an object or a location in the at least one perceived environment.

### Claim 14 (depends on 8)

The apparatus of claim 8, wherein the compact intermediate representation of the perceived environment is generated using at least one of a neural network or predetermined rules.

### Claim 15

A system for developing an understanding of at least one perceived environment, comprising:
a foundational model; and
at least one machine agent, comprising:
a pre-processing module;
a graphing module;
a processor; and
a memory accessible to the processor, the memory having stored therein at least one of programs or instructions executable by the processor to configure the machine agent to:
using the pre-processor module, determine semantic features and respective positional information of the semantic features from received data related to images and respective depth-related content of the at least one perceived environment on the fly as changes in the received data occur;
using the graphing module, for each of the at least one perceived environments, combine information of the determined semantic features with the respective positional information to generate a compact intermediate representation of the perceived environment which provides information regarding positions of the semantic features in the perceived environment and at least spatial relationships among the sematic features;
using the processor, for each of the at least one perceived environments, combine information from the determined intermediate representation with information stored in a foundational model to determine a respective understanding of the perceived environment; and
output an indication of the determined respective understanding of the perceived environment.

### Claim 16 (depends on 15)

The system of claim 15, wherein the data related to the images and the respective depth-related content of the at least one perceived environment is received from at least one of a human, an agent or at least one sensor capable of capturing image content and depth information of the at least one perceived environment.

### Claim 17 (depends on 15)

The system of claim 15, wherein the intermediate representation comprises at least one of a hierarchical scene graph, which encodes the semantic features with their 3D spatial-temporal relationships across multiple levels or a hierarchical knowledge graph, which models at least one agent used in developing an understanding of the at least one perceived environment and capabilities of the at least one agent.

### Claim 18 (depends on 15)

The system of claim 15, wherein the foundational model comprises a large language model, which provides common sense knowledge for the at least one perceived environment.

### Claim 19 (depends on 15)

The system of claim 15, wherein the indication comprises at least one of a representation of the perceived environment annotated using information from the determined respective understanding, a navigation plan to enable an agent to navigate the perceived environment determined from the determined respective understanding, or a next step of a task to be completed by the agent in the perceived environment.

### Claim 20 (depends on 15)

The system of claim 15, wherein the machine agent is further configured to:
navigate an agent through the at least one perceived environment using the developed understanding of the at least one perceived environment to locate at least one of an object or a location in the at least one perceived environment.

### Claim 21 (depends on 15)

The system of claim 15, wherein the compact intermediate representation of the perceived environment is generated using at least one of a neural network or predetermined rules.

### Claim 22

A non-transitory computer readable storage medium having stored thereon instructions that when executed by a processor perform a method for developing an understanding of at least one perceived environment, comprising:
determining semantic features and respective positional information of the semantic features from received data related to images and respective depth-related content of the at least one perceived environment on the fly as changes in the received data occur;
for each of the at least one perceived environments, combining information of the determined semantic features with the respective positional information to generate a compact intermediate representation of the perceived environment which provides information regarding positions of the semantic features in the perceived environment and at least spatial relationships among the sematic features;
for each of the at least one perceived environments, combining information from the determined intermediate representation with information stored in a foundational model to determine a respective understanding of the perceived environment; and
outputting an indication of the determined respective understanding of the perceived environment.
