Trajectory prediction using efficient attention neural networks

A system comprising:

• one or more computers;
• and
• one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• obtaining scene context data characterizing a scene in an environment at a current time point, wherein the scene includes a plurality of agents comprising a target agent and one or more context agents, and the scene context data comprises respective context data for each of multiple different modalities of context data;
• generating an encoded representation of the scene in the environment that comprises one or more embeddings, comprising:
• generating, for each of the multiple different modalities, a respective sequence of input elements for the modality from the context data for the modality;
• generating a combined sequence by concatenating the respective sequences of input elements for each of the different modalities;
• and
• processing the combined sequence using an attention-based encoder neural network to generate the one or more embeddings, wherein the attention-based encoder neural network comprises at least one cross-modal attention layer block that attends over input elements corresponding to each of the multiple different modalities;
• and
• processing the encoded representation of the scene context data using a decoder neural network to generate a trajectory prediction output for the target agent that predicts a future trajectory of the target agent after the current time point.

One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining scene context data characterizing a scene in an environment at a current time point, wherein the scene includes a plurality of agents comprising a target agent and one or more context agents, and the scene context data comprises respective context data for each of multiple different modalities of context data;
• generating an encoded representation of the scene in the environment that comprises one or more embeddings, comprising:
• generating, for each of the multiple different modalities, a respective sequence of input elements for the modality from the context data for the modality;
• generating a combined sequence by concatenating the respective sequences of input elements for each of the different modalities;
• and
• processing the combined sequence using an attention-based encoder neural network to generate the one or more embeddings, wherein the attention-based encoder neural network comprises at least one cross-modal attention layer block that attends over input elements corresponding to each of the multiple different modalities;
• and
• processing the encoded representation of the scene context data using a decoder neural network to generate a trajectory prediction output for the target agent that predicts a future trajectory of the target agent after the current time point.