Asset evaluation system for autonomous vehicle simulations

A system comprising:

• a storage configured to store instructions;
• a processor configured to execute the instructions and cause the processor to:
• receive drive data recorded in a physical environment by a vehicle having a first sensor;
• simulate the first sensor associated with a virtual autonomous vehicle in a virtual environment of a three-dimensional (3D) scene including an object that at least partially corresponds to the physical environment;
• evaluate simulated data based on the simulation of the first sensor using a machine learning (ML) model for navigating an autonomous vehicle in the physical environment;
• compare evaluation data recorded during the evaluation of the simulation of the first sensor using the ML model to the drive data recorded in the physical environment;
• determine, based on a comparison of the evaluation data to a portion of the drive data, metrics associated with a perception of the object in the virtual environment by the ML model and a perception of a corresponding physical object in the physical environment by the ML model;
• generate a report comprising the metrics;
• modify or maintain a 3D asset corresponding to the object in the virtual environment, based on the metrics, wherein:
• modifying the 3D asset comprises modifying a correlation between the perception of the object in the virtual environment by the ML model and the perception of the corresponding physical object in the physical environment by the ML model;
• and
• maintaining the 3D asset comprises maintaining the correlation;
• receive second drive data recorded in the physical environment by the autonomous vehicle;
• predict, based on the second drive data and using the ML model and the 3D asset, a future event associated with at least the corresponding physical object in the physical environment;
• and
• navigate the autonomous vehicle in the physical environment, based on predicting the future event.

A non-transitory computer readable medium comprising instructions, the instructions, when executed by a computing system, cause the computing system to:

• receive drive data recorded in a physical environment by a vehicle having a first sensor;
• simulate the first sensor associated with a virtual autonomous vehicle in a virtual environment of a three-dimensional (3D) scene including an object that at least partially corresponds to the physical environment;
• evaluate simulated data based on the simulation of the first sensor using a machine learning (ML) model for navigating an autonomous vehicle in the physical environment;
• compare evaluation data recorded during the evaluation of the simulation of the first sensor using the ML model to the drive data recorded in the physical environment;
• determine, based on a comparison of the evaluation data to a portion of the drive data, metrics associated with a perception of the object in the virtual environment by the ML model and a perception of a corresponding physical object in the physical environment by the ML model;
• generate a report comprising the metrics;
• modify or maintain a 3D asset corresponding to the object in the virtual environment, based on the metrics, wherein:
• modifying the 3D asset comprises modifying a correlation between the perception of the object in the virtual environment by the ML model and the perception of the corresponding physical object in the physical environment by the ML model;
• and
• maintaining the 3D asset comprises maintaining the correlation;
• receive second drive data recorded in the physical environment by the autonomous vehicle;
• predict, based on the second drive data and using the ML model and the 3D asset, a future event associated with at least the corresponding physical object in the physical environment;
• and
• navigate the autonomous vehicle in the physical environment, based on predicting the future event.