# US 12,217,515 B2 · Training a codebook for trajectory determination

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,515 B2
- Document ID: US12217515B2
- Status: Granted
- Kind: B2
- Application number: 17/855,696
- Filed: 2022-06-30
- Published: 2024-03-28
- Granted: 2025-02-04
- Earliest claimed: 2022-06-30
- Assignees: Zoox, Inc.
- Applicants: Zoox, Inc.
- Inventors: Ethan Miller Pronovost
- Web page: https://patentgrep.com/patent/US12217515B2

## CPC

- G06V 10/82: using neural networks
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- B60W 40/10: related to vehicle motion
- B60W 60/0027: {using trajectory prediction for other traffic participants}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- B60W 2554/4049: Relationship among other objects, e.g. converging dynamic objects
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT

## Abstract

Techniques for training a codebook usable by a machine learned model to predict an object trajectory or scene data are described herein. For example, the techniques may include generating tokens representing discrete object behavior into a machine learned model that outputs a sequence of tokens that is usable by another machine learned model to generate the object trajectory (e.g., position data, velocity data, acceleration data, etc.) or the scene data associated with the environment. The object trajectory can be sent to a vehicle computing device for consideration during vehicle planning, which may include simulation.

## Claims

### Claim 1

A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the one or more processors to perform actions comprising:
receiving, by a training component, state data representing a previous state of an object in an environment;
receiving, by the training component, feature vectors representing a vehicle and the object in the environment;
training a codebook as a trained codebook, the training comprising:
assigning, based at least in part on the state data, a first token to represent the previous state of the object;
assigning a second token to represent a characteristic of the vehicle;
assigning a third token to represent a feature of the environment; and
mapping the feature vectors to respective tokens; and
outputting the trained codebook for use by a machine learned model configured to access tokens from the trained codebook and to arrange the first token, the second token, and the third token to represent a potential interaction between the vehicle and the object.

### Claim 2 (depends on 1)

The system of claim 1, wherein the machine learned model is a transformer model, and the actions further comprising:
causing the transformer model to arrange the tokens based at least in part on a number of tokens and an order of the tokens.

### Claim 3 (depends on 1)

The system of claim 1, wherein:
the feature vectors represent continuous variables,
the tokens represent discrete latent variables, and
mapping the feature vectors to respective tokens comprises mapping the continuous variables associated with the feature vectors to the discrete latent variables associated with the tokens.

### Claim 4 (depends on 1)

The system of claim 1, the actions further comprising:
receiving, by the training component, environmental data representing the feature of the environment,
wherein training the codebook further comprises assigning the second token to represent the feature of the environment, and
the codebook is usable by the machine learned model to cause generation of a scene with the feature in the environment.

### Claim 5 (depends on 1)

The system of claim 1, wherein the vehicle is an autonomous vehicle.

### Claim 6

A method comprising:
receiving, by a training component, state data representing a previous state of an object or a vehicle in an environment;
receiving, by the training component, feature vectors representing the vehicle and the object in the environment;
training a codebook to output a trained codebook, the training comprising:
assigning, based at least in part on the state data, a first token to represent the previous state of the object;
assigning a second token to represent a characteristic of the vehicle;
assigning a third token to represent a feature of the environment; and
mapping the feature vectors to respective tokens; and
outputting the trained codebook for use by a machine learned model configured to access tokens from the trained codebook and to arrange the first token, the second token, and the third token to represent a potential interaction between the vehicle and the object.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
determining a number of tokens to include in the codebook; and
determining an order of the tokens.

### Claim 8 (depends on 6)

The method of claim 6, further comprising:
causing the machine learned model to arrange the tokens based at least in part on a number of tokens and an order of the tokens.

### Claim 9 (depends on 6)

The method of claim 6, wherein:
the feature vectors represent continuous variables,
the tokens represent discrete latent variables, and
mapping the feature vectors to respective tokens comprises mapping the continuous variables associated with the feature vectors to the discrete latent variables associated with the tokens.

### Claim 10 (depends on 6)

The method of claim 6, further comprising:
receiving, by the training component, environmental data representing the feature of the environment,
wherein training the codebook further comprises assigning the second token to represent the feature of the environment, and
the codebook is usable by the machine learned model to cause generation of a scene with the feature in the environment.

### Claim 11 (depends on 6)

The method of claim 6, wherein the state data representing the previous state of the object or the vehicle comprising one or more of: position data, orientation data, heading data, velocity data, speed data, acceleration data, yaw rate data, or turning rate data.

### Claim 12 (depends on 6)

The method of claim 6, further comprising:
inputting image data into an encoder; and
outputting, by the encoder, the feature vectors,
wherein receiving the feature vectors by the training component comprises receiving the feature vectors from the encoder.

### Claim 13 (depends on 6)

The method of claim 6, wherein:
the feature vectors represent the characteristic of the vehicle and a characteristic of the object in the environment.

### Claim 14 (depends on 6)

The method of claim 6, wherein mapping the feature vectors to respective tokens comprises:
identifying an action or a state of the vehicle associated with a first feature vector of the feature vectors;
comparing the characteristic of the vehicle associated with the first feature vector to a characteristic associated with one or more of the tokens.

### Claim 15 (depends on 6)

The method of claim 6, wherein the vehicle is an autonomous vehicle.

### Claim 16

One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform actions comprising:
receiving, by a training component, state data representing a previous state of an object or a vehicle in an environment;
receiving, by the training component, feature vectors representing the vehicle and the object in the environment;
training a codebook to output a trained codebook, the training comprising:
assigning, based at least in part on the state data, a first token to represent the previous state of the object;
assigning a second token to represent a characteristic of the vehicle;
assigning a third token to represent a feature of the environment; and
mapping the feature vectors to respective tokens; and
outputting the trained codebook for use by a machine learned model configured to access tokens from the trained codebook and to arrange the first token, the second token, and the third token to represent a potential interaction between the vehicle and the object.

### Claim 17 (depends on 16)

The one or more non-transitory computer-readable media of claim 16, wherein the machine learned model is a first machine learned model, and the actions further comprising:
causing the machine learned model to arrange the tokens based at least in part on a number of tokens and an order of the tokens.

### Claim 18 (depends on 16)

The one or more non-transitory computer-readable media of claim 16, wherein:
the feature vectors represent continuous variables,
the tokens represent discrete latent variables, and
mapping the feature vectors to respective tokens comprises mapping the continuous variables associated with the feature vectors to the discrete latent variables associated with the tokens.

### Claim 19 (depends on 16)

The one or more non-transitory computer-readable media of claim 16, the actions further comprising:
receiving, by the training component, environmental data representing the feature of the environment,
wherein training the codebook further comprises assigning the second token to represent the feature of the environment, and
the codebook is usable by the machine learned model to cause generation of a scene with the feature in the environment.

### Claim 20 (depends on 16)

The one or more non-transitory computer-readable media of claim 16, wherein the vehicle is an autonomous vehicle.