# US 2026/0064115 A1 · Network Architecture for a Mobility Foundation Model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0064115 A1
- Document ID: US20260064115A1
- Status: Published application
- Kind: A1
- Application number: 19/319,448
- Filed: 2025-09-04
- Published: 2026-03-05
- Earliest claimed: 2024-09-04
- Assignees: Vayu Robotics, Inc.
- Applicants: Vayu Robotics, Inc.
- Inventors: Soumith Udatha; Peter Jans Gillespie; Arul Gupta; Nitish Srivastava
- Web page: https://patentgrep.com/patent/US20260064115A1

## CPC

- G05D 1/229: Command input data, e.g. waypoints
- B60W 60/001: {Planning or execution of driving tasks}
- G01C 21/3407: {specially adapted for specific applications}
- G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera}
- G05B 13/027: {using neural networks only}
- G05D 1/243: Means capturing signals occurring naturally from the environment, e.g. ambient optical, acoustic, gravitational or magnetic signals (using passive navigation aids external to the vehicle G05D1/244; using signals from positioning sensors located off-board the vehicle G05D1/249)
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G05D 2101/15: using machine learning, e.g. neural networks
- G05D 2101/20: using external object recognition
- G05D 2105/28: of freight
- G05D 2109/10: Land vehicles
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods for implementing mobility foundation models in accordance with some embodiments of the invention are illustrated. One embodiment includes a method for operating a mobile device. The method receives initial state tokens, wherein each corresponds to data reflecting a previous state of a mobile device. The method determines a sub-task for the mobile device by applying an LLM to the initial state tokens. The method encodes sensor data into patch tokens. Each of the patch tokens reflects a recent state of the mobile device. The method updates the initial state tokens into updated state tokens, based on the patch tokens and the sub-task. The method produces navigation waypoints from the updated state tokens, wherein each of the navigation waypoints represents a distinct destination for the mobile device. The method controlling the mobile device according to the navigation waypoints.

## Claims

### Claim 1

A method for operating a mobile device, the method comprising:
receiving, at the mobile device:
an initial plurality of state tokens, wherein each of the initial plurality of state tokens corresponds to transformer model data reflecting at least one previous state of a mobile device; and
sensor data, from a set of one or more sensors that are appended to the mobile device;
determining at least one sub-task for the mobile device by inputting the initial plurality of state tokens into a particular large language model;
encoding the sensor data into a plurality of patch tokens, by inputting the sensor data into at least one vision transformer, wherein:
each of the plurality of patch tokens reflects at least one recent state of the mobile device; and
the at least one recent state corresponds to a period before the at least one previous state;
updating the initial plurality of state tokens into an updated plurality of state tokens by inputting the initial plurality of state tokens into at least one cross-attention transformer, wherein:
each of the updated plurality of state tokens corresponds to transformer model data reflecting the at least one recent state of the mobile device; and
the initial plurality of state tokens are updated based on the plurality of patch tokens and the at least one sub-task;
producing, by the mobile device, a set of navigation waypoints from the updated plurality of state tokens, wherein each of the set of navigation waypoints represents a distinct destination for the mobile device; and
controlling the mobile device according to the set of navigation waypoints.

### Claim 2 (depends on 1)

The method of claim 1, wherein the mobile device is an autonomous vehicle.

### Claim 3 (depends on 1)

The method of claim 1, wherein the set of one or more sensors includes at least one camera.

### Claim 4 (depends on 1)

The method of claim 1, wherein encoding the sensor data into the plurality of patch tokens further comprises adding learned ray embeddings, produced using at least one Multi-Layer Perceptron (MLP), into each of the plurality of patch tokens.

### Claim 5 (depends on 1)

The method of claim 1, wherein producing the set of navigation waypoints comprises producing, using a primary query decoder and the updated plurality of state tokens, at least one proposed path for the mobile device.

### Claim 6 (depends on 5)

The method of claim 5, wherein the primary query decoder comprises at least one Diffusion Policy Decoder.

### Claim 7 (depends on 5)

The method of claim 5, further comprising decoding a natural language query into a decoded query using a secondary query decoder, wherein the natural language query is received through a user interface.

### Claim 8 (depends on 7)

The method of claim 7, further comprising projecting an answer to the decoded query on the user interface, wherein the answer to the decoded query is based on the updated plurality of state tokens and the sensor data.

### Claim 9 (depends on 7)

The method of claim 7, wherein the secondary query decoder:
comprises an additional large language model; and
operates more slowly than the primary query decoder.

### Claim 10 (depends on 7)

The method of claim 7, wherein the particular large language model, the at least one vision transformer, the at least one cross-attention transformer, the primary query decoder, and the secondary query decoder are components of a foundation model.

### Claim 11 (depends on 6)

The method of claim 6, wherein the at least one proposed path is produced using a specific cross-attention transformer included in the primary query decoder, according to keys and values derived from the updated plurality of state tokens.

### Claim 12 (depends on 7)

The method of claim 7, wherein:
the primary decoder operates on the mobile device; and
the secondary decoder operates on a remote server that is communicatively coupled to the mobile device.

### Claim 13 (depends on 5)

The method of claim 5, wherein a given path of the at least one proposed path is selected from the group consisting of:
a planned spatial path comprising a set of spatially-separated points with consistent spacing;
a planned temporal path comprising a set of temporally-separated points with consistent spacing, derived according to a linear speed of the mobile device; and
a left boundary and a right boundary for the mobile device.

### Claim 14 (depends on 1)

The method of claim 1, wherein the plurality of patch tokens comprises encodings of at least one of:
an estimate of a kinematic state of the mobile device, wherein the estimate comprises a linear speed of the mobile device and an angular speed of the mobile device;
a set of dimensions for the mobile device; or
a set of route instructions that are pre-determined for the mobile device.

### Claim 15 (depends on 1)

The method of claim 1, wherein updating the initial plurality of state tokens comprises inputting the plurality of patch tokens and the at least one sub-task into a state aggregator.

### Claim 16 (depends on 15)

The method of claim 15, wherein updating the initial plurality of state tokens further comprises:
producing a set of queries from the initial plurality of state tokens;
generating answers to the set of queries, using a cross-attention transformer included in the state aggregator, according to keys and values derived from the plurality of patch tokens and the at least one sub-task; and
applying the answers to updating the initial plurality of state tokens.

### Claim 17 (depends on 16)

The method of claim 16, wherein updating the initial plurality of state tokens further comprises inputting the updated plurality of state tokens into at least one of a feed-forward network or a self-attention network.

### Claim 18 (depends on 17)

The method of claim 17, wherein the at least one of the feed-forward network or the self-attention network is incorporated into the state aggregator.

### Claim 19 (depends on 1)

The method of claim 1, wherein a quantity of the initial plurality of state tokens remains constant when updated into the updated plurality of state tokens.

### Claim 20

A method for operating a mobile device, the method comprising:
receiving, at the mobile device:
an initial plurality of state tokens, wherein each of the initial plurality of state tokens corresponds to transformer model data reflecting at least one previous state of a mobile device; and
sensor data, from a set of one or more sensors that are appended to the mobile device;
determining at least one sub-task for the mobile device by inputting the initial plurality of state tokens into a particular large language model;
encoding the sensor data into a plurality of patch tokens, by inputting the sensor data into at least one vision transformer, wherein:
each of the plurality of patch tokens reflects at least one recent state of the mobile device; and
the at least one recent state corresponds to a period before the at least one previous state;
updating the initial plurality of state tokens into an updated plurality of state tokens by inputting the initial plurality of state tokens into at least one cross-attention transformer, wherein:
each of the updated plurality of state tokens corresponds to transformer model data reflecting the at least one recent state of the mobile device;
a quantity of the initial plurality of state tokens remains constant when updated into the updated plurality of state tokens; and
the initial plurality of state tokens are updated based on the plurality of patch tokens and the at least one sub-task;
producing, by the mobile device, a set of navigation waypoints from the updated plurality of state tokens, wherein:
producing the set of navigation waypoints comprises producing at least one proposed path for the mobile device, wherein the at least one proposed path is produced using a specific cross-attention transformer included in a primary query decoder, according to keys and values derived from the updated plurality of state tokens; and
each of the set of navigation waypoints represents a distinct destination for the mobile device;
controlling the mobile device according to the set of navigation waypoints; and
when a natural language query, corresponding to the mobile device, is received through a user interface of the mobile device:
decoding the natural language query into a decoded query using a secondary query decoder, wherein:
the secondary query decoder:
comprises an additional large language model; and
operates more slowly than the primary query decoder; and
the particular large language model, the at least one vision transformer, the at least one cross-attention transformer, the primary query decoder, and the secondary query decoder are components of a foundation model; and
projecting an answer to the decoded query on the user interface, wherein the answer to the decoded query is based on the updated plurality of state tokens and the sensor data.
