# US 2026/0094428 A1 · Performing perception tasks by leveraging auto-regressive neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0094428 A1
- Document ID: US20260094428A1
- Status: Published application
- Kind: A1
- Application number: 19/348,113
- Filed: 2025-10-02
- Published: 2026-04-02
- Earliest claimed: 2024-10-02
- Assignees: Waymo LLC
- Applicants: Waymo LLC
- Inventors: Alex Zihao Zhu; Hao Xiang; Zhaoqi Leng; Mingxing Tan; Dragomir Anguelov
- Web page: https://patentgrep.com/patent/US20260094428A1

## CPC

- G06V 10/82: using neural networks
- G06V 10/40: Extraction of image or video features
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing perception tasks on received sensor data. The method includes obtaining one or more query images and a plurality of context images; generating a sequence of discrete tokens representing the context images; generating one or more continuous tokens representing the one or more query images; processing an input comprising the sequence of discrete tokens representing the context images and the one or more continuous tokens representing the one or more query images using a token processing neural network to generate one or more updated continuous tokens representing the one or more query images; and processing the one or more updated continuous tokens to generate a respective output for each of one or more prediction tasks.

## Claims

### Claim 1

A method performed by one or more computers, the method comprising:
obtaining one or more query images and a plurality of context images;
generating a sequence of discrete tokens representing the context images;
generating one or more continuous tokens representing the one or more query images;
processing an input comprising the sequence of discrete tokens representing the context images and the one or more continuous tokens representing the one or more query images using a token processing neural network to generate one or more updated continuous tokens representing the one or more query images; and
processing the one or more updated continuous tokens to generate a respective output for each of one or more prediction tasks.

### Claim 2 (depends on 1)

The method of claim 1, wherein processing the one or more updated continuous tokens to generate a respective output for each of one or more prediction tasks comprises:
generating, from the updated continuous tokens, an adapted feature representing the one or more query images; and
for each of the one or more prediction tasks, processing the adapted feature representing the one or more query images using a decoder neural network for the prediction task to generate the output for the prediction task.

### Claim 3 (depends on 2)

The method of claim 2, wherein generating, from the updated continuous tokens, an adapted feature representing the one or more query images comprises:
processing an input comprising the updated continuous tokens using a decoder adapter neural network to generate the adapted feature.

### Claim 4 (depends on 1)

The method of claim 1, wherein generating one or more continuous tokens representing the one or more query images comprises:
processing the one or more query images using an image encoder neural network to generate an encoded feature map representing the one or more query images; and
processing the encoded feature map using an encoder adapter neural network to generate the one or more continuous tokens.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
generating a sequence of discrete tokens representing the current image; and
wherein the input comprising the sequence of discrete tokens representing the context images and the one or more continuous tokens further comprises the sequence of discrete tokens representing the current image.

### Claim 6 (depends on 1)

The method of claim 1, wherein the input comprising the sequence of discrete tokens representing the context images and the one or more continuous tokens further comprises one or more learnable query tokens.

### Claim 7 (depends on 6)

The method of claim 6, wherein the learnable query tokens comprise a respective set of one or more learnable query tokens for each of the one or more prediction tasks.

### Claim 8 (depends on 1)

The method of claim 1, wherein the token processing neural network is a transformer neural network.

### Claim 9 (depends on 1)

The method of claim 1, wherein the token processing neural network comprises one or more causal self-attention layers.

### Claim 10 (depends on 1)

The method of claim 1, wherein generating a sequence of discrete tokens representing the context images comprises, for each context image:
processing the context image using a vision tokenizer neural network to generate one or more discrete tokens; and
including the one or more discrete tokens in the sequence of discrete tokens representing the context images.

### Claim 11 (depends on 10)

The method of claim 10, wherein generating a sequence of discrete tokens representing the context images comprises, for each context image and for each of one or more modalities:
generating a respective structured output for the context image for the modality;
processing the respective structured output using the vision tokenizer neural network to generate one or more discrete tokens; and
including the one or more discrete tokens in the sequence of discrete tokens representing the context images.

### Claim 12 (depends on 11)

The method of claim 11, wherein the one or more modalities include a depth prediction modality.

### Claim 13 (depends on 11)

The method of claim 11, wherein the one or more modalities include a segmentation modality.

### Claim 14 (depends on 11)

The method of claim 11, wherein generating a respective structured output for the context image for the modality comprises:
processing the context image using a task neural network for the modality to generate the respective structured output for the modality.

### Claim 15 (depends on 1)

The method of claim 1, wherein the token processing neural network comprises (i) an embedding layer and (ii) one or more continuous token updating layers, and wherein processing an input comprising the sequence of discrete tokens representing the context images and the one or more continuous tokens representing the one or more query images using a token processing neural network to generate one or more updated continuous tokens representing the one or more query images comprises:
processing each discrete token in the input using the embedding layer to generate a continuous token representing the discrete token; and
processing at least the continuous tokens representing the discrete tokens and the continuous tokens representing the one or more query images using the continuous token updating layers to generate the one or more updated continuous tokens representing the one or more query images.

### Claim 16 (depends on 1)

The method of claim 1, wherein the one or more query images are captured by a set of one or more cameras at a current time point and wherein the context images comprise a respective set of one or more context images captured by the set of one or more cameras at each of one or more preceding time points.

### Claim 17 (depends on 1)

The method of claim 1, wherein the token processing neural network has been pre-trained on a next token prediction task that requires predicting, given a current sequence of discrete tokens, a next discrete token that follows a last discrete token in the current sequence of discrete tokens.

### Claim 18 (depends on 17)

The method of claim 17, wherein, after the pre-training, the image encoder, the encoder adapter, the decoder adapter, and the decoder neural networks for the prediction tasks have been trained through supervised learning on labeled training data for the one or more prediction tasks.

### Claim 19 (depends on 18)

The method of claim 18, wherein the token processing neural network is fine-tuned during the training through supervised learning.

### Claim 20

A system comprising:
one or more computers; and
one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations comprising:
obtaining one or more query images and a plurality of context images;
generating a sequence of discrete tokens representing the context images;
generating one or more continuous tokens representing the one or more query images;
processing an input comprising the sequence of discrete tokens representing the context images and the one or more continuous tokens representing the one or more query images using a token processing neural network to generate one or more updated continuous tokens representing the one or more query images; and
processing the one or more updated continuous tokens to generate a respective output for each of one or more prediction tasks.

### Claim 21

One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
obtaining one or more query images and a plurality of context images;
generating a sequence of discrete tokens representing the context images;
generating one or more continuous tokens representing the one or more query images;
processing an input comprising the sequence of discrete tokens representing the context images and the one or more continuous tokens representing the one or more query images using a token processing neural network to generate one or more updated continuous tokens representing the one or more query images; and
processing the one or more updated continuous tokens to generate a respective output for each of one or more prediction tasks.
