Control system testing utilizing rulebook scenario generation

A system, comprising:

• at least one processor, and
• at least one non-transitory storage media storing instructions that, when executed by the at least one processor, cause the at least one processor to:
• determine a simulated environment, wherein the simulated environment includes a plurality of simulated environmental parameters and at least one simulated agent, wherein the at least one simulated agent comprises at least one agent parameter;
• receive a plurality of autonomous vehicle rules and a hierarchy for the plurality of autonomous vehicle rules;
• determine a trajectory of a simulated autonomous vehicle within the simulated environment;
• generate a plurality of simulated scenarios for the simulated autonomous vehicle, wherein to generate a particular simulated scenario of the plurality of simulated scenarios, execution of the instructions by the at least one processor, causes the at least one processor to modify the at least one agent parameter of the at least one simulated agent;
• identify at least one violation of at least one autonomous vehicle rule by the simulated autonomous vehicle in a set of simulated scenarios of the plurality of simulated scenarios, wherein the at least one violation of the at least one autonomous vehicle rule is based on a response of the simulated autonomous vehicle to the set of simulated scenarios;
• determine a set of rule violation priorities for the set of simulated scenarios, the set of rule violation priorities comprising a respective rule violation priority for each respective simulated scenario of the set of simulated scenarios based on the at least one violation of at least one autonomous vehicle rule and a position of the at least one autonomous vehicle rule within the hierarchy for the plurality of autonomous vehicle rules;
• compare the set of rule violation priorities;
• based on comparing the set of rule violation priorities, determine that a simulated scenario of the set of simulated scenarios is more likely, as compared to other simulated scenarios of the set of simulated scenarios, to cause a trained neural network of an autonomous vehicle to identify a first trajectory of the autonomous vehicle that violates a first autonomous vehicle rule of the plurality of autonomous vehicle rules with a first rule violation priority instead of a second autonomous vehicle rule of the plurality of autonomous vehicle rules with a second rule violation priority, wherein the first rule violation priority exceeds the second rule violation priority;
• in response to determining that the simulated scenario is more likely to cause the trained neural network to identify the first trajectory that violates the first autonomous vehicle rule instead of the second autonomous vehicle rule, select the simulated scenario from the set of simulated scenarios to test a response of the trained neural network to the simulated scenario;
• provide the simulated scenario to a computing device associated with the trained neural network based on selecting the simulated scenario, wherein the trained neural network is configured to respond to the simulated scenario by identifying a respective trajectory of the autonomous vehicle based on the simulated scenario;
• and
• adjust and implement a control strategy of a control system of the autonomous vehicle based on the response of the trained neural network to the simulated scenario such that the simulated scenario causes the trained neural network to identify a second trajectory of the autonomous vehicle that violates the second autonomous vehicle rule instead of the first autonomous vehicle rule.

At least one non-transitory storage media storing instructions that, when executed by a computing system comprising a processor, cause the computing system to:

• determine a simulated environment, wherein the simulated environment includes a plurality of simulated environmental parameters and at least one simulated agent, wherein the at least one simulated agent comprises at least one agent parameter;
• receive a plurality of autonomous vehicle rules and a hierarchy for the plurality of autonomous vehicle rules;
• determine a trajectory of a simulated autonomous vehicle within the simulated environment;
• generate a plurality of simulated scenarios for the simulated autonomous vehicle, wherein to generate a particular simulated scenario of the plurality of simulated scenarios, execution of the instructions by the computing system, causes the computing system to modify the at least one agent parameter of the at least one simulated agent;
• identify at least one violation of at least one autonomous vehicle rule by the simulated autonomous vehicle in a set of simulated scenarios of the plurality of simulated scenarios, wherein the at least one violation of the at least one autonomous vehicle rule is based on a response of the simulated autonomous vehicle to the set of simulated scenarios;
• determine a set of rule violation priorities for the set of simulated scenarios, the set of rule violation priorities comprising a respective rule violation priority for each respective simulated scenario of the set of simulated scenarios based on the at least one violation of at least one autonomous vehicle rule and a position of the at least one autonomous vehicle rule within the hierarchy for the plurality of autonomous vehicle rules;
• compare the set of rule violation priorities;
• based on comparing the set of rule violation priorities, determine that a simulated scenario of the set of simulated scenarios is more likely, as compared to other simulated scenarios of the set of simulated scenarios, to cause a trained neural network of an autonomous vehicle to identify a first trajectory of the autonomous vehicle that violates a first autonomous vehicle rule of the plurality of autonomous vehicle rules with a first rule violation priority instead of a second autonomous vehicle rule of the plurality of autonomous vehicle rules with a second rule violation priority, wherein the first rule violation priority exceeds the second rule violation priority;
• in response to determining that the simulated scenario is more likely to cause the trained neural network to identify the first trajectory that violates the first autonomous vehicle rule instead of the second autonomous vehicle rule, select the simulated scenario from the set of simulated scenarios to test a response of the trained neural network to the simulated scenario;
• provide the simulated scenario to a computing device associated with the trained neural network based on selecting the simulated scenario, wherein the trained neural network is configured to respond to the simulated scenario by identifying a respective trajectory of the autonomous vehicle based on the simulated scenario;
• and
• adjust and implement a control strategy of a control system of the autonomous vehicle based on the response of the trained neural network to the simulated scenario such that the simulated scenario causes the trained neural network to identify a second trajectory of the autonomous vehicle that violates the second autonomous vehicle rule instead of the first autonomous vehicle rule.