# US 2026/0087803 A1 · Machine learning-based processing of environmental data with temporal dynamics and spatial awareness information

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0087803 A1
- Document ID: US20260087803A1
- Status: Published application
- Kind: A1
- Application number: 18/893,120
- Filed: 2024-09-23
- Published: 2026-03-26
- Earliest claimed: 2024-09-23
- Assignees: Dell Products L.P.
- Applicants: Dell Products L.P.
- Inventors: Zijia Wang; Sanping Li; Tianlu Fei; Zhen Jia; Linye Li
- Web page: https://patentgrep.com/patent/US20260087803A1

## CPC

- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06F 40/30: Semantic analysis
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/82: using neural networks
- G06V 10/86: using syntactic or structural representations of the image or video pattern, e.g. symbolic string recognition; using graph matching
- G06V 20/44: {Event detection}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An apparatus comprises at least one processing device configured to generate, based on an input prompt, a first data structure comprising a textual task description associated with tasks to be performed in an environment, and to generate, based on video data of the environment, a second data structure comprising temporal dynamics information characterizing changes in spatial features of the environment over time. The at least one processing device is also configured to generate, based on images of the environment, a third data structure comprising spatial relationship information characterizing spatial relationships between objects in the environment, and to determine, utilizing at least one machine learning model that takes as input at least portions of the first, second and third data structures, actions to execute in the environment to achieve the tasks. The at least one processing device is further configured to execute the determined actions in the environment.

## Claims

### Claim 1

An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
the at least one processing device being configured:
to generate, based at least in part on an obtained input prompt, a first data structure comprising a textual task description associated with one or more tasks to be performed in an environment;
to generate, based at least in part on video data of the environment, a second data structure comprising temporal dynamics information characterizing one or more changes in spatial features of the environment over time;
to generate, based at least in part on one or more images of the environment, a third data structure comprising spatial relationship information characterizing spatial relationships between two or more objects in the environment;
to determine, utilizing at least one machine learning model that takes as input at least portions of the first, second and third data structures, one or more actions to execute in the environment to achieve the one or more tasks; and
to execute the determined one or more actions in the environment.

### Claim 2 (depends on 1)

The apparatus of claim 1 wherein generating the first data structure comprises applying one or more natural language processing algorithms to the obtained input prompt.

### Claim 3 (depends on 1)

The apparatus of claim 1 wherein generating the second data structure comprises:
processing a sequence of two or more frames in the video data using a convolutional neural network machine learning model to extract feature vectors encapsulating spatial information of the environment; and
processing the extracted feature vectors using a recurrent neural network machine learning model to determine a set of hidden states representing temporal evolution of the spatial features.

### Claim 4 (depends on 3)

The apparatus of claim 3 wherein the recurrent neural network machine learning model comprises one or more long short term memory units.

### Claim 5 (depends on 3)

The apparatus of claim 3 wherein generating the second data structure further comprises:
utilizing a classifier to map the temporal evolution of the spatial features to action labels; and
determining event segmentation by identifying changes in the spatial features based at least in part on differences between consecutive ones of the hidden states in the set of hidden states.

### Claim 6 (depends on 3)

The apparatus of claim 3 wherein generating the second data structure further comprises utilizing a temporal relation network to identify relationships between events based on analysis of pairs of the hidden states in the set of hidden states.

### Claim 7 (depends on 1)

The apparatus of claim 1 wherein generating the third data structure comprises:
processing the one or more images of the environment utilizing a convolutional neural network machine learning model to extract feature maps comprising two-dimensional pixel coordinates and associated depth values; and
performing three-dimensional scene reconstruction of the environment utilizing a back-projection algorithm that translates the two-dimensional pixel coordinates and the associated depth values into three-dimensional coordinates relative to a position of a camera in the environment, the back-projection algorithm being based at least in part on a set of camera parameters of the camera, the set of camera parameters comprising focal length and optical center.

### Claim 8 (depends on 7)

The apparatus of claim 7 wherein generating the third data structure further comprises:
performing object detection by applying a region proposal network to the extracted feature maps to detect the two or more objects in the environment; and
applying a graph neural network machine learning model to classify and localize the two or more objects within the environment.

### Claim 9 (depends on 7)

The apparatus of claim 7 wherein generating the third data structure further comprises utilizing a spatial relationship graph that takes as input object information for the two or more objects in the environment and the three-dimensional coordinates of the environment to determine spatial relationships between pairs of the two or more objects.

### Claim 10 (depends on 9)

The apparatus of claim 9 wherein the spatial relationship graph is generated utilizing a graph neural network machine learning model.

### Claim 11 (depends on 1)

The apparatus of claim 1 wherein the at least one machine learning model comprises a multi-modal large language model implementing an attention mechanism configured to evaluate the significance of one or more words and phrases in the textual task description in the context of the temporal dynamics information and the spatial relationship information.

### Claim 12 (depends on 1)

The apparatus of claim 1 wherein the environment comprises a physical environment, and wherein the one or more tasks to be performed in the environment comprises navigation of an autonomous vehicle from a source location to a destination location in the physical environment.

### Claim 13 (depends on 1)

The apparatus of claim 1 wherein the environment comprises a physical environment, and wherein the one or more tasks to be performed in the environment comprises movement of robotic equipment to manipulate at least one of the two or more objects in the physical environment.

### Claim 14 (depends on 1)

The apparatus of claim 1 wherein the environment comprises an augmented reality or virtual reality environment, and wherein the one or more tasks to be performed in the environment comprises manipulation of at least one of the two or more objects in the augmented reality or virtual reality environment.

### Claim 15

A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device:
to generate, based at least in part on an obtained input prompt, a first data structure comprising a textual task description associated with one or more tasks to be performed in an environment;
to generate, based at least in part on video data of the environment, a second data structure comprising temporal dynamics information characterizing one or more changes in spatial features of the environment over time;
to generate, based at least in part on one or more images of the environment, a third data structure comprising spatial relationship information characterizing spatial relationships between two or more objects in the environment;
to determine, utilizing at least one machine learning model that takes as input at least portions of the first, second and third data structures, one or more actions to execute in the environment to achieve the one or more tasks; and
to execute the determined one or more actions in the environment.

### Claim 16 (depends on 15)

The computer program product of claim 15 wherein generating the second data structure comprises:
processing a sequence of two or more frames in the video data using a convolutional neural network machine learning model to extract feature vectors encapsulating spatial information of the environment; and
processing the extracted feature vectors using a recurrent neural network machine learning model to determine a set of hidden states representing temporal evolution of the spatial features.

### Claim 17 (depends on 15)

The computer program product of claim 15 wherein generating the third data structure comprises:
processing the one or more images of the environment utilizing a convolutional neural network machine learning model to extract feature maps comprising two-dimensional pixel coordinates and associated depth values; and
performing three-dimensional scene reconstruction of the environment utilizing a back-projection algorithm that translates the two-dimensional pixel coordinates and the associated depth values into three-dimensional coordinates relative to a position of a camera in the environment, the back-projection algorithm being based at least in part on a set of camera parameters of the camera, the set of camera parameters comprising focal length and optical center.

### Claim 18

A method comprising:
generating, based at least in part on an obtained input prompt, a first data structure comprising a textual task description associated with one or more tasks to be performed in an environment;
generating, based at least in part on video data of the environment, a second data structure comprising temporal dynamics information characterizing one or more changes in spatial features of the environment over time;
generating, based at least in part on one or more images of the environment, a third data structure comprising spatial relationship information characterizing spatial relationships between two or more objects in the environment;
determining, utilizing at least one machine learning model that takes as input at least portions of the first, second and third data structures, one or more actions to execute in the environment to achieve the one or more tasks; and
executing the determined one or more actions in the environment;
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

### Claim 19 (depends on 18)

The method of claim 18 wherein generating the second data structure comprises:
processing a sequence of two or more frames in the video data using a convolutional neural network machine learning model to extract feature vectors encapsulating spatial information of the environment; and
processing the extracted feature vectors using a recurrent neural network machine learning model to determine a set of hidden states representing temporal evolution of the spatial features.

### Claim 20 (depends on 18)

The method of claim 18 wherein generating the third data structure comprises:
processing the one or more images of the environment utilizing a convolutional neural network machine learning model to extract feature maps comprising two-dimensional pixel coordinates and associated depth values; and
performing three-dimensional scene reconstruction of the environment utilizing a back-projection algorithm that translates the two-dimensional pixel coordinates and the associated depth values into three-dimensional coordinates relative to a position of a camera in the environment, the back-projection algorithm being based at least in part on a set of camera parameters of the camera, the set of camera parameters comprising focal length and optical center.
