Systems and methods for autonomous vehicle systems simulation

A computer-implemented method, comprising:

• obtaining, by a computing system comprising one or more computing devices, simplified scenario data associated with a simulated scenario, wherein the simplified scenario data comprises a simplified entity representation for a simulated entity, wherein the simulated entity is derived from a bounding box, and wherein the bounding box is based on a user-generated sketch sketched via a user input device, and wherein the user-generated sketch is indicative of the bounding box and one or more characteristics of the simulated entity represented by the simplified entity representation;
• determining, by the computing system using a machine-learned perception-prediction simulation model, a simulated perception-prediction output based at least in part on the simplified scenario data, wherein the simulated perception-prediction output comprises simulated perception data and simulated prediction data, wherein the machine-learned perception-prediction simulation model is configured to simulate a perception system and a prediction system of an autonomous vehicle, and wherein the simulated perception-prediction output comprises a prediction for the simulated entity;
• evaluating, by the computing system, a loss function comprising a perception loss term and a prediction loss term, wherein the perception loss term evaluates a difference between the simulated perception data and ground truth perception data and the prediction loss term evaluates a difference between the simulated prediction data and ground truth prediction data;
• and
• adjusting, by the computing system, one or more parameters of the machine-learned perception-prediction simulation model based at least in part on the loss function.

One or more tangible, non-transitory computer readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising:

• obtaining simplified scenario data that represents an intermediate output associated with a simulated scenario for a perception system and a prediction system of an autonomous vehicle, wherein the simplified scenario data comprises a simplified entity representation for a simulated entity, wherein the simulated entity is derived from a bounding box, and wherein the bounding box is based on a user-generated sketch sketched via a user input device, and wherein the user-generated sketch is indicative of the bounding box and one or more characteristics of the simulated entity represented by the simplified entity representation;
• processing the simplified scenario data with a perception submodel of a machine-learned perception-prediction simulation model to obtain simulated perception data, wherein the machine-learned perception-prediction simulation model is configured to simulate the perception system and the prediction system of the autonomous vehicle, and wherein the simulated prediction data comprises a prediction for the simulated entity;
• processing the simplified scenario data and the simulated perception data with a prediction submodel of the machine-learned perception-prediction simulation model to obtain simulated prediction data;
• evaluating a loss function comprising a perception loss term and a prediction loss term, wherein the perception loss term evaluates a difference between the simulated perception data and ground truth perception data and the prediction loss term evaluates a difference between the simulated prediction data and ground truth prediction data;
• and
• adjusting one or more parameters of the machine-learned perception-prediction simulation model based at least in part on the loss function.