Scene tokenization for motion prediction

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• receiving sensor data of an environment having one or more agents;
• decomposing the sensor data into a plurality of scene elements in the environment;
• generating a plurality of tokens including a respective token for each respective scene element of the plurality of scene elements in the environment;
• and
• processing the plurality of tokens using a decoder model to generate a respective predicted trajectory for the one or more agents in the environment.

One or more non-transitory computer storage media encoded with computer program instructions that when executed by a plurality of computers cause the plurality of computers to perform operations comprising:

• receiving sensor data of an environment having one or more agents;
• decomposing the sensor data into a plurality of scene elements in the environment;
• generating a plurality of tokens including a respective token for each respective scene element of the plurality of scene elements in the environment;
• and
• processing the plurality of tokens using a decoder model to generate a respective predicted trajectory for the one or more agents in the environment.