Systems and Methods for Object Detection and Motion Prediction by Fusing Multiple Sensor Sweeps into a Range View Representation

An autonomous vehicle, comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
• obtaining a plurality of sensor sweeps from a sensor associated with the autonomous vehicle, each sensor sweep representing a time step between a current point in time and a past point in time;
• determining movement data associated with movement of the autonomous vehicle from the past time to the current time;
• for each respective sensor sweep in the plurality of sensor sweeps, beginning with the least recent sensor sweep and repeating until the sensor sweep associated with the current time:
• generating a respective image, the respective image comprising sensor data included in the respective sensor sweep arranged in a two-dimensional representation;
• extracting feature data using the respective image as input to one or more machine-learned models, feature data from the respective image;
• transforming the feature data into a coordinate frame associated with a next time step;
• and
• mapping each point from the respective image to the image associated with the next time step to generate a fused image;
• generating a final fused representation of the plurality of sensors sweeps once a sensor sweep associated with the current time step has been combined with all previously generated sensor sweeps;
• and
• predicting, based, at least in part, on the final fused representation of the plurality of sensors sweeps from the plurality of sensor sweeps, movement associated with the feature data one or more future time steps.

A computer-implemented method for predicting motion of objects, the method comprising:

• obtaining, by a computing system with one or more processors, a plurality of sensor sweeps from a sensor associated with an autonomous vehicle, each sensor sweep representing a particular time step between a current point in time and a past point in time;
• determining, by the computing system, movement data associated with movement of the autonomous vehicle from the past time to the current time;
• concurrently, for each respective sensor sweep in the plurality of sensor sweeps:
• generating, by the computing system, a respective image, the respective image comprising sensor data associated with the respective sensor sweep arranged in a two-dimensional representation;
• extracting, by the computing system using one or more first machine-learned models, feature data from the respective image;
• transforming, by the computing system using one or more second machine-learned models with the feature data and the movement data of the autonomous vehicle as input, the feature data into a current coordinate frame associated with the current point in time;
• and
• mapping, by the computing system, each point from the respective image to the image associated with the most recent sensor sweep;
• generating, by the computing system, a fused image by combining the images for each of the sensor sweeps;
• and
• predicting, by the computing system based, at least in part, on the fused image from the plurality of sensor sweeps, movement associated with the feature data at one or future more time steps.