# US 2021/0278539 A1 · Systems and Methods for Object Detection and Motion Prediction by Fusing Multiple Sensor Sweeps into a Range View Representation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0278539 A1 - Document ID: US20210278539A1 - Status: Published application - Kind: A1 - Application number: 17/091,577 - Filed: 2020-11-06 - Published: 2021-09-09 - Earliest claimed: 2020-03-05 - Assignees: Uatc, LLC - Applicants: Uatc, LLC - Inventors: Ankit Laddha; Gregory P. Meyer; Jake Scott Charland; Shivam Gautam; Shreyash Pandey; Carlos Vallespi-Gonzalez; Carl Knox Wellington - Web page: https://patentgrep.com/patent/US20210278539A1 ## CPC - G01S 17/58: Velocity or trajectory determination systems; Sense-of-movement determination systems - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - B60W 50/0097: {Predicting future conditions} - B60W 60/0027: {using trajectory prediction for other traffic participants} - G01S 7/4808: {Evaluating distance, position or velocity data} - G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46) - G01S 17/89: for mapping or imaging - G01S 17/931: of land vehicles - G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)} - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - B60W 30/09: Taking automatic action to avoid collision, e.g. braking and steering - B60W 30/0956: {the prediction being responsive to traffic or environmental parameters} - B60W 2420/408: Radar; Laser, e.g. lidar - G06N 3/045: Combinations of networks - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20081: Training; Learning - G06T 2207/20221: Image fusion; Image merging - G06T 2207/30261: Obstacle - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems and methods for detecting objects and predicting their motion are provided. In particular, a computing system can obtain a plurality of sensor sweeps. The computing system can determine movement data associated with movement of the autonomous vehicle. For each sensor sweep, the computing system can generate an image associated with the sensor sweep. The computing system can extract, using the respective image as input to one or more machine-learned models, feature data from the respective image. The computing system can transform the feature data into a coordinate frame associated with a next time step. The computing system can generate a fused image. The computing system can generate a final fused image. The computing system can predict, based, at least in part, on the final fused representation of the plurality of sensors sweeps from the plurality of sensor sweeps, movement associated with the feature data at one or more time steps in the future. ## Claims ### Claim 1 A computer-implemented method for predicting motion of objects, the method comprising: obtaining, by a computing system with one or more processors, a plurality of sensor sweeps from a sensor associated with an autonomous vehicle, each sensor sweep being associated with a time step between a current point in time and a past point in time; determining, by the computing system, movement data associated with movement of the autonomous vehicle from the past point in time to the current point in time; for each respective sensor sweep in the plurality of sensor sweeps, beginning with the least recent sensor sweep in the plurality of sensor sweeps and repeating with each subsequent sensor sweep until the most recent sensor sweep: generating, by the computing system and based on the respective sensor sweep, a respective range view image associated with the respective sensor sweep, the respective range view image comprising sensor data included in the respective sensor sweep arranged in a two-dimensional representation; extracting, by the computing system using the respective range view image as input to one or more machine-learned models, feature data from the respective image; transforming, by the computing system, the feature data into a coordinate frame associated with a next time step; and mapping, by the computing system, each point from the respective range view image to the range-view image associated with the next time step to generate a fused image; generating, by the computing system, a final fused representation of the plurality of sensors sweeps once a sensor sweep associated with the current time step has been combined with all previously generated sensor sweeps; and predicting, by the computing system based, at least in part, on the final fused representation of the plurality of sensors sweeps, movement associated with the feature data at one or more future time steps. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the sensor data is point cloud data collected by a LIDAR sensor. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein generating, by the computing system, a respective range view image, the respective range view image comprising sensor data included in the respective sensor sweep arranged in a two-dimensional representation further comprises: for each point in the point cloud data, determining, by the computing system, a position in the range view image that corresponds to a three-dimensional position of the point in the point cloud data. ### Claim 4 (depends on 2) The computer-implemented method of claim 2, wherein each sweep in the plurality of sweeps represents a complete revolution of the LIDAR sensor. ### Claim 5 (depends on 2) The computer-implemented method of claim 2, wherein each point in the point cloud data has an associated luminance value. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein the respective image is generated based, at least in part, on the luminance values associated with the points in the point cloud data. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein each sensor sweep has an associated coordinate frame. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, wherein the coordinate frame associated with each sensor sweep is based on a location of the autonomous vehicle when the sensor data associated with the sweep was captured. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein predicting, by the computing system based, at least in part, on the final fused representation of the plurality of sensors sweeps, movement associated with the feature data at one or more future time steps further comprises: identifying, by the computing system, one or more objects in the feature data; and determining, by the computing system, past movement for the one or more objects based on the final fused representation of the plurality of sensors sweeps. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, wherein predicting, by the computing system based, at least in part, on the final fused representation of the plurality of sensors sweeps, movement associated with the feature data at one or more future time steps further comprises: determining, by the computing system, a predicted position for the one or more objects at one or more future time steps. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the final fused representation of the plurality of sensors sweeps uses the coordinate frame of the current sensor sweep. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein the one or more machine-learned models are convolutional neural networks. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, further comprising, prior to extracting, by the computing system, feature data from the respective range view image, concatenating the respective range view image with mapped feature data associated with the previous sensor sweep. ### Claim 14 An autonomous vehicle, comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising: obtaining a plurality of sensor sweeps from a sensor associated with the autonomous vehicle, each sensor sweep representing a time step between a current point in time and a past point in time; determining movement data associated with movement of the autonomous vehicle from the past time to the current time; for each respective sensor sweep in the plurality of sensor sweeps, beginning with the least recent sensor sweep and repeating until the sensor sweep associated with the current time: generating a respective image, the respective image comprising sensor data included in the respective sensor sweep arranged in a two-dimensional representation; extracting feature data using the respective image as input to one or more machine-learned models, feature data from the respective image; transforming the feature data into a coordinate frame associated with a next time step; and mapping each point from the respective image to the image associated with the next time step to generate a fused image; generating a final fused representation of the plurality of sensors sweeps once a sensor sweep associated with the current time step has been combined with all previously generated sensor sweeps; and predicting, based, at least in part, on the final fused representation of the plurality of sensors sweeps from the plurality of sensor sweeps, movement associated with the feature data one or more future time steps. ### Claim 15 (depends on 14) The autonomous vehicle of claim 14, wherein predicting, using mapped feature data from the plurality of sensor sweeps as input to a machine-learned model, movement associated with the feature data at a future point in time further comprises: identifying one or more objects in the feature data; and determining past movement for the one or more objects based on the mapped feature data. ### Claim 16 (depends on 15) The autonomous vehicle of claim 15, wherein predicting, using mapped feature data from the plurality of sensor sweeps as input to a machine-learned model, movement associated with the feature data at a future point in time further comprises: determining a predicted position for the one or more objects at one or more time steps in the future. ### Claim 17 (depends on 14) The autonomous vehicle of claim 14, wherein the final fused representation of the plurality of sensors sweeps uses the coordinate frame of the current sensor sweep. ### Claim 18 A computer-implemented method for predicting motion of objects, the method comprising: obtaining, by a computing system with one or more processors, a plurality of sensor sweeps from a sensor associated with an autonomous vehicle, each sensor sweep representing a particular time step between a current point in time and a past point in time; determining, by the computing system, movement data associated with movement of the autonomous vehicle from the past time to the current time; concurrently, for each respective sensor sweep in the plurality of sensor sweeps: generating, by the computing system, a respective image, the respective image comprising sensor data associated with the respective sensor sweep arranged in a two-dimensional representation; extracting, by the computing system using one or more first machine-learned models, feature data from the respective image; transforming, by the computing system using one or more second machine-learned models with the feature data and the movement data of the autonomous vehicle as input, the feature data into a current coordinate frame associated with the current point in time; and mapping, by the computing system, each point from the respective image to the image associated with the most recent sensor sweep; generating, by the computing system, a fused image by combining the images for each of the sensor sweeps; and predicting, by the computing system based, at least in part, on the fused image from the plurality of sensor sweeps, movement associated with the feature data at one or future more time steps. ### Claim 19 (depends on 18) The computer-implemented method of claim 18, wherein the one or more first machine-learned models and the one or more second machine-learned models are convolutional neural networks. ### Claim 20 (depends on 18) The computer-implemented method of claim 18, wherein the one or more first machine-learned models have the same weights.