# US 12,183,204 B2 · Trajectory prediction on top-down scenes and associated model > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,204 B2 - Document ID: US12183204B2 - Status: Granted - Kind: B2 - Application number: 17/542,880 - Filed: 2021-12-06 - Published: 2022-03-24 - Granted: 2024-12-31 - Earliest claimed: 2018-10-04 - Assignees: Zoox, Inc. - Applicants: Zoox, Inc. - Inventors: Xi Joey Hong; Benjamin John Sapp; James William Vaisey Philbin; Kai Zhenyu Wang - Web page: https://patentgrep.com/patent/US12183204B2 ## CPC - G01C 21/3407: {specially adapted for specific applications} - G08G 1/164: {Centralised systems, e.g. external to vehicles} - B60W 30/0956: {the prediction being responsive to traffic or environmental parameters} - B60W 30/18159: {Traversing an intersection} - B60W 60/00272: {relying on extrapolation of current movement} - G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera} - G05D 1/0221: {involving a learning process} - G05D 1/0246: {using a video camera in combination with image processing means} - G05D 1/0276: {using signals provided by a source external to the vehicle (involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058)} - G05D 1/0278: {using satellite positioning signals, e.g. GPS} - G05D 1/246: using environment maps, e.g. simultaneous localisation and mapping [SLAM] - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06T 7/292: Multi-camera tracking - G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes} - B60W 2556/45: External transmission of data to or from the vehicle - G05D 2101/10: using artificial intelligence [AI] techniques - G06T 2207/10032: Satellite or aerial image; Remote sensing - G06T 2207/30236: Traffic on road, railway or crossing - G06T 2207/30241: Trajectory - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Techniques are discussed for determining prediction probabilities of an object based on a top-down representation of an environment. Data representing objects in an environment can be captured. Aspects of the environment can be represented as map data. A multi-channel image representing a top-down view of object(s) in the environment can be generated based on the data representing the objects and map data. The multi-channel image can be used to train a machine learned model by minimizing an error between predictions from the machine learned model and a captured trajectory associated with the object. Once trained, the machine learned model can be used to generate prediction probabilities of objects in an environment, and the vehicle can be controlled based on such prediction probabilities. ## Claims ### Claim 1 A system comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising: receiving map data associated with an environment; receiving object data associated with an object in the environment; generating, based at least in part on the map data and the object data, a multi-channel data structure representing the environment from a top-down perspective, wherein a channel of the multi-channel data structure includes a portion of the object data, wherein the portion of the object data comprises at least one of a semantic label, a bounding box associated with the object, a velocity associated with the object, or an acceleration associated with the object; inputting the multi-channel data structure into a machine learned model; receiving, from the machine learned model and based at least in part on the multi-channel data structure, a prediction probability associated with the object; and controlling, based at least in part on the prediction probability, a vehicle to traverse the environment. ### Claim 2 (depends on 1) The system of claim 1, wherein at least some of the object data is included in another channel of the multi-channel data structure. ### Claim 3 (depends on 1) The system of claim 1, wherein the object data comprises semantic information indicative of at least one of a bounding box associated with the object, movement information associated with the object, or a classification associated with the object. ### Claim 4 (depends on 1) The system of claim 1, wherein a classification associated with the object is represented as a color in the channel, and wherein an intensity of the color is indicative of a magnitude of an attribute of the object. ### Claim 5 (depends on 1) The system of claim 1, wherein another channel of the multi-channel data structure includes semantic information associated with the environment, the semantic information comprising at least one of road network information or a traffic light status. ### Claim 6 (depends on 1) The system of claim 1, wherein the portion of the object data included in the channel of the multi-channel data structure is encoded with an attribute indicating at least one of a color or a location associated with the object. ### Claim 7 (depends on 1) The system of claim 1, wherein the prediction probability comprises at least one of: a multi modal Gaussian trajectory; or an occupancy grid associated with a future time, wherein a cell of the occupancy grid is indicative of a probability of the object being in a region associated with the cell at the future time. ### Claim 8 (depends on 1) The system of claim 1, wherein the multi-channel data structure further comprises one or more additional channels including semantic information associated with at least one of the environment or objects in the environment. ### Claim 9 A method comprising: receiving map data associated with an environment; receiving object data associated with an object in the environment; generating, based at least in part on the map data and the object data, a multi-channel data structure representing the environment from a top-down perspective, wherein a channel of the multi-channel data structure includes a portion of the object data, wherein the portion of the object data comprises at least one of a semantic label, a bounding box associated with the object, a velocity associated with the object, or an acceleration associated with the object; inputting the multi-channel data structure into a machine learned model; receiving, from the machine learned model and based at least in part on the multi-channel data structure, a prediction probability associated with the object; and controlling, based at least in part on the prediction probability, a vehicle to traverse the environment. ### Claim 10 (depends on 9) The method of claim 9, wherein at least some of the object data is included in another channel of the multi-channel data structure. ### Claim 11 (depends on 9) The method of claim 9, wherein the object data comprises semantic information indicative of at least one of a bounding box associated with the object, movement information associated with the object, or a classification associated with the object. ### Claim 12 (depends on 9) The method of claim 9, wherein a classification associated with the object is represented as a color in the channel, and wherein an intensity of the color is indicative of a magnitude of an attribute of the object. ### Claim 13 (depends on 9) The method of claim 9, wherein another channel of the multi-channel data structure includes semantic information associated with the environment, the semantic information comprising at least one of road network information or a traffic light status. ### Claim 14 (depends on 9) The method of claim 9, wherein the portion of the object data included in the channel of the multi-channel data structure is encoded with an attribute indicating at least one of a color or a location associated with the object. ### Claim 15 (depends on 9) The method of claim 9, wherein the prediction probability comprises at least one of: a multi modal Gaussian trajectory; or an occupancy grid associated with a future time, wherein a cell of the occupancy grid is indicative of a probability of the object being in a region associated with the cell at the future time. ### Claim 16 (depends on 9) The method of claim 9, wherein the multi-channel data structure further comprises one or more additional channels including semantic information associated with at least one of the environment or objects in the environment. ### Claim 17 A non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform operations comprising: receiving map data associated with an environment; receiving object data associated with an object in the environment; generating, based at least in part on the map data and the object data, a multi-channel data structure representing the environment from a top-down perspective, wherein a channel of the multi-channel data structure includes a portion of the object data, wherein the portion of the object data comprises at least one of a semantic label, a bounding box associated with the object, a velocity associated with the object, or an acceleration associated with the object; inputting the multi-channel data structure into a machine learned model; receiving, from the machine learned model and based at least in part on the multi-channel data structure, a prediction probability associated with the object; and controlling, based at least in part on the prediction probability, a vehicle to traverse the environment. ### Claim 18 (depends on 17) The non-transitory computer-readable medium of claim 17, wherein at least some of the object data is included in another channel of the multi-channel data structure. ### Claim 19 (depends on 17) The non-transitory computer-readable medium of claim 17, wherein the object data is indicative of at least one of a bounding box associated with the object, movement information associated with the object, or a classification associated with the object. ### Claim 20 (depends on 17) The non-transitory computer-readable medium of claim 17, wherein another channel of the multi-channel data structure includes semantic information associated with the environment, the semantic information comprising at least one of road network information or a traffic light status.