Systems and methods for assessing the performance of an automated autonomous driving evaluator

A non-transitory computer-readable medium for assessing performance of an automated autonomous driving evaluator and storing instructions that, when executed by a processor, cause the processor to:

• receive a driving log that includes at least one of raw vehicle sensor data correlated with a time index and information derived from the raw vehicle sensor data correlated with the time index;
• receive ground-truth data associated with the driving log, wherein the ground-truth data includes, for one or more respective values of the time index, at least one of human-initiated disengagements of an autonomous driving system and human-entered annotations indicating mistakes made by the autonomous driving system;
• analyze the driving log using the automated autonomous driving evaluator to generate a report, wherein the automated autonomous driving evaluator is a computerized system that automatically analyzes driving logs to identify mistakes made by autonomous driving systems;
• compare automatically the report with the ground-truth data to generate a performance assessment for the automated autonomous driving evaluator;
• and
• provide the performance assessment to a development process in which the automated autonomous driving evaluator is modified based, at least in part, on the performance assessment to improve the performance of the automated autonomous driving evaluator.

A method, comprising:

• receiving a driving log that includes at least one of raw vehicle sensor data correlated with a time index and information derived from the raw vehicle sensor data correlated with the time index;
• receiving ground-truth data associated with the driving log, wherein the ground-truth data includes, for one or more respective values of the time index, at least one of human-initiated disengagements of an autonomous driving system and human-entered annotations indicating mistakes made by the autonomous driving system;
• analyzing the driving log using an automated autonomous driving evaluator to generate a report, wherein the automated autonomous driving evaluator is a computerized system that automatically analyzes driving logs to identify mistakes made by autonomous driving systems;
• comparing automatically the report with the ground-truth data to generate a performance assessment for the automated autonomous driving evaluator;
• and
• providing the performance assessment to a development process in which the automated autonomous driving evaluator is modified based, at least in part, on the performance assessment to improve the performance of the automated autonomous driving evaluator.