World models for processing vehicle sensor data

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• obtaining an input sequence of tokens characterizing a driving environment for a vehicle, wherein the input sequence of tokens comprises, for each of one or more time steps, a respective set of tokens for each of a plurality of modalities of data from a set of multiple modalities of data;
• and
• processing the input sequence of tokens using a multi-modal token processing neural network to generate an output sequence of tokens, wherein:
• the output sequence of tokens characterizes a prediction for a prediction task for the vehicle, and
• the multi-modal token processing neural network has been trained to process input token sequences characterizing current states of example driving environments to generate output token sequences characterizing predicted future states of the example driving environments.

One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• obtaining an input sequence of tokens characterizing a driving environment for a vehicle, wherein the input sequence of tokens comprises, for each of one or more time steps, a respective set of tokens for each of a plurality of modalities of data from a set of multiple modalities of data;
• and
• processing the input sequence of tokens using a multi-modal token processing neural network to generate an output sequence of tokens, wherein:
• the output sequence of tokens characterizes a prediction for a prediction task for the vehicle, and
• the multi-modal token processing neural network has been trained to process input token sequences characterizing current states of example driving environments to generate output token sequences characterizing predicted future states of the example driving environments.