Scenario-based behavior specification and validation

A rule-based trajectory evaluation system, comprising:

• a user interface configured for receiving user input selecting at least a driving scenario;
• at least one processor;
• and
• a memory storing instructions thereon that, when executed by the at least one processor, cause the at least one processor to:
• evaluate, using at least one rulebook, at least one trajectory in the at least one driving scenario to determine whether the at least one trajectory violates at least one rule in the at least one rulebook, wherein each rule in the rulebook is associated with at least one violation metric that is used to determine a degree to which the rule was satisfied or violated;
• determine, using the at least one violation metric, a score for the at least one trajectory;
• generate a report or visual related to the score;
• and
• provide data associated with the report, the data associated with the report configured to cause the user interface to present the report via the user interface.

A non-transient, computer-readable storage medium having stored thereon instructions, that when executed by at least one processor, cause the at least one processor to perform operations comprising:

• obtaining, using at least one processor, at least one trajectory associated with a driving scenario for an autonomous vehicle system;
• evaluating, using the at least one processor and at least one rulebook, the at least one trajectories to determine whether the at least one trajectory violates at least one rule in the at least one rulebook, wherein each rule in the rulebook is associated with at least one violation metric that is used to determine a degree to which the rule was satisfied or violated;
• determining, using the at least one processor and the at least one metric, a score for the at least one trajectory;
• and
• sending, using the at least one processor, the score to at least one of a software module in a software stack of the autonomous vehicle system, a simulation of the autonomous vehicle system or as a report or in a visual presented through a user interface of a cloud-based platform.