# US 2025/0166366 A1 · Scene tokenization for motion prediction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0166366 A1 - Document ID: US20250166366A1 - Status: Published application - Kind: A1 - Application number: 18/950,830 - Filed: 2024-11-18 - Published: 2025-05-22 - Earliest claimed: 2023-11-17 - Assignees: Waymo LLC - Applicants: Waymo LLC - Inventors: Yin Zhou; Jingwei Ji; Zhenpei Yang; Nathan Paul Harada; Kan Chen; Rami Al-Rfou - Web page: https://patentgrep.com/patent/US20250166366A1 ## CPC - G06V 10/82: using neural networks - B60W 60/00274: {considering possible movement changes} - G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition} - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - B60W 2420/408: Radar; Laser, e.g. lidar - B60W 2554/4045: Intention, e.g. lane change or imminent movement - B60W 2555/60: Traffic rules, e.g. speed limits or right of way - B60W 2556/10: Historical data - B60W 2556/35: Data fusion - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT ## Abstract Methods, systems, and apparatus for predicting future trajectories of agents in an environment. In one aspect, a system comprises one or more computers configured to receive sensor data of an environment having one or more agents. The system decomposes the sensor data into a plurality of scene elements in the environment, and the system generates multiple tokens including a respective token for each respective scene element of the multiple scene elements in the environment. The system processes the multiple tokens using a decoder model to generate a respective predicted trajectory for the one or more agents in the environment. ## Claims ### Claim 1 A computer-implemented method comprising: receiving sensor data of an environment having one or more agents; decomposing the sensor data into a plurality of scene elements in the environment; generating a plurality of tokens including a respective token for each respective scene element of the plurality of scene elements in the environment; and processing the plurality of tokens using a decoder model to generate a respective predicted trajectory for the one or more agents in the environment. ### Claim 2 (depends on 1) The method of claim 1, wherein decomposing the sensor data into the plurality of scene elements comprises distinguishing an agent in the environment from a ground region in the environment, and wherein generating the plurality of tokens comprises generating a first token for the agent and a different second token for the ground region. ### Claim 3 (depends on 2) The method of claim 2, wherein generating the first token comprises using a first model and wherein generating the second token comprises using a different second model. ### Claim 4 (depends on 3) The method of claim 3, wherein decomposing the sensor data into the plurality of scene elements comprises generating a different third token for an object in the environment. ### Claim 5 (depends on 1) The method of claim 1, wherein generating a token for a scene element in the environment comprises encoding image features and geometry features of an entity in the environment. ### Claim 6 (depends on 1) The method of claim 1, wherein generating the plurality of tokens is performed by a multi-modality scene encoder configured to tokenize scene elements in a scene as well as perception outputs representing sensor information of a vehicle. ### Claim 7 (depends on 1) The method of claim 1, wherein generating the plurality of tokens comprises applying the operations of a plurality of self-attention layers. ### Claim 8 (depends on 7) The method of claim 7, wherein the plurality of self-attention layers fuse information extracted from tokens for the scene elements and tokens generated from perception systems of a vehicle. ### Claim 9 (depends on 1) The method of claim 1, wherein the decoder model has a Transformer-based architecture. ### Claim 10 (depends on 1) The method of claim 1, wherein generating the plurality of tokens comprises generating a token representing one or more features of a road graph for the environment. ### Claim 11 (depends on 1) The method of claim 1, generating the plurality of tokens comprises generating a token representing one or more motion-based characteristics of an agent in the environment. ### Claim 12 (depends on 11) The method of claim 11, wherein the motion-based characteristics include an agent velocity or a motion history. ### Claim 13 (depends on 1) The method of claim 1, wherein generating the plurality of tokens comprises generating a token representing traffic light information for a traffic light within the environment. ### Claim 14 (depends on 1) The method of claim 1, further comprising adjusting a trajectory of a vehicle based on the predicted trajectory of the scene element in the environment. ### Claim 15 (depends on 1) The method of claim 1, wherein the sensor data represents sensor observations over a plurality of history frames. ### Claim 16 (depends on 15) The method of claim 15, further comprising collapsing scene element representations that appear over the plurality of history frames into a single scene element representation. ### Claim 17 (depends on 15) The method of claim 15, wherein collapsing the scene element representations comprises downsampling LiDAR data. ### Claim 18 (depends on 17) The method of claim 17, wherein downsampling the LiDAR data comprises performing a first downsampling process for ground scene elements and a different second downsampling process for other scene elements. ### Claim 19 A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving sensor data of an environment having one or more agents; decomposing the sensor data into a plurality of scene elements in the environment; generating a plurality of tokens including a respective token for each respective scene element of the plurality of scene elements in the environment; and processing the plurality of tokens using a decoder model to generate a respective predicted trajectory for the one or more agents in the environment. ### Claim 20 One or more non-transitory computer storage media encoded with computer program instructions that when executed by a plurality of computers cause the plurality of computers to perform operations comprising: receiving sensor data of an environment having one or more agents; decomposing the sensor data into a plurality of scene elements in the environment; generating a plurality of tokens including a respective token for each respective scene element of the plurality of scene elements in the environment; and processing the plurality of tokens using a decoder model to generate a respective predicted trajectory for the one or more agents in the environment.