Conversion of sensor-based pedestrian motion into three-dimensional (3d) animation data

A computer-implemented system, comprising:

• one or more processing units;
• and
• one or more non-transitory computer-readable media storing instructions, when executed by the one or more processing units, cause the one or more processing units to perform operations comprising:
• receiving sensor data captured from a real-world driving environment across a time period, the sensor data including a capture of at least one moving character in the real-world driving environment;
• generating, based on the sensor data, perception data including main annotations indicating the moving character and auxiliary annotations indicating movements of the moving character across the time period;
• and
• training a machine learning (ML) model to generate skeletal markers using the sensor data and the perception data.

A method comprising:

• obtaining, by a computer-implemented system, real-world road data including a sequence of images of a real-world driving environment across a time period, the sequence of images including at least one character in motion in the real-world driving environment;
• generating, by the computer-implemented system, perception data based on the sequence of images, the perception data including a label for the character in each of one or more images in the sequence;
• generating, by the computer-implemented system, a sequence of three-dimensional (3D) skeletal structures representing the character in motion across the time period by processing the perception data;
• and
• outputting, by the computer-implemented system, a motion library including the sequence of 3D skeletal structures.