# US 2023/0410567 A1 · Systems and methods for assessing the performance of an automated autonomous driving evaluator > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0410567 A1 - Document ID: US20230410567A1 - Status: Published application - Kind: A1 - Application number: 17/842,040 - Filed: 2022-06-16 - Published: 2023-12-21 - Earliest claimed: 2022-06-16 - Assignees: Woven By Toyota, Inc. - Applicants: Woven by Toyota, Inc. - Inventors: Daniel Walker; Michael Jared Benisch - Web page: https://patentgrep.com/patent/US20230410567A1 ## CPC - G07C 5/0808: {Diagnosing performance data (testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005)} - B60W 50/06: Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot - B60W 60/0054: {Selection of occupant to assume driving tasks} - G07C 5/0841: {Registering performance data (recording measured values G01D; information storage G11B)} - B60W 2050/0088: {Adaptive recalibration} - B60W 2556/10: Historical data - G07C 5/0816: {Indicating performance data, e.g. occurrence of a malfunction} - G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT ## Abstract Systems and methods for assessing the performance of an automated autonomous driving evaluator are disclosed herein. One embodiment receives a driving log including at least one of raw vehicle sensor data and information derived from the raw vehicle sensor data correlated with a time index; receives ground-truth data associated with the driving log, wherein the ground-truth data includes at least one of human-initiated disengagements of an autonomous driving system and human-entered annotations indicating mistakes made by the autonomous driving system; analyzes the driving log using an automated autonomous driving evaluator to generate a report; automatically compares the report with the ground-truth data to generate a performance assessment for the automated autonomous driving evaluator; and provides the performance assessment to a development process in which the automated autonomous driving evaluator is modified based, at least in part, on the performance assessment to improve the performance of the automated autonomous driving evaluator. ## Claims ### Claim 1 A system for assessing performance of an automated autonomous driving evaluator, the system comprising: a processor; and a memory storing machine-readable instructions that, when executed by the processor, cause the processor to: receive a driving log that includes at least one of raw vehicle sensor data correlated with a time index and information derived from the raw vehicle sensor data correlated with the time index; receive ground-truth data associated with the driving log, wherein the ground-truth data includes, for one or more respective values of the time index, at least one of human-initiated disengagements of an autonomous driving system and human-entered annotations indicating mistakes made by the autonomous driving system; analyze the driving log using the automated autonomous driving evaluator to generate a report, wherein the automated autonomous driving evaluator is a computerized system that automatically analyzes driving logs to identify mistakes made by autonomous driving systems; compare automatically the report with the ground-truth data to generate a performance assessment for the automated autonomous driving evaluator; and provide the performance assessment to a development process in which the automated autonomous driving evaluator is modified based, at least in part, on the performance assessment to improve the performance of the automated autonomous driving evaluator. ### Claim 2 (depends on 1) The system of claim 1, wherein: the ground-truth data is generated in an autonomous vehicle controlled by the autonomous driving system as the driving log is being recorded; and a human observer riding in the autonomous vehicle initiates the human-initiated engagements and enters the human-entered annotations. ### Claim 3 (depends on 1) The system of claim 1, wherein the ground-truth data is generated while the driving log is being replayed in a simulator by a human observer who enters the human-entered annotations. ### Claim 4 (depends on 1) The system of claim 1, wherein the automated autonomous driving evaluator is designed to identify mistakes pertaining to a particular predetermined aspect of autonomous driving. ### Claim 5 (depends on 1) The system of claim 1, wherein the performance assessment for the automated autonomous driving evaluator includes one or more of false positives, false negatives, true positives, and true negatives. ### Claim 6 (depends on 1) The system of claim 1, wherein the driving log is recorded in an operating autonomous vehicle. ### Claim 7 (depends on 1) The system of claim 1, wherein the driving log is generated synthetically by a computing system. ### Claim 8 (depends on 1) The system of claim 1, wherein the automated autonomous driving evaluator is one of rule-based and machine-learning-based. ### Claim 9 (depends on 1) The system of claim 1, wherein: the driving log is recorded in an autonomous vehicle operating in a shadow mode in which the autonomous driving system simulates controlling the autonomous vehicle while the autonomous vehicle is being controlled by a human operator; and simulated actions taken by the autonomous driving system that deviate in a predetermined manner from actions taken by the human operator are automatically annotated as mistakes in the ground-truth data. ### Claim 10 A non-transitory computer-readable medium for assessing performance of an automated autonomous driving evaluator and storing instructions that, when executed by a processor, cause the processor to: receive a driving log that includes at least one of raw vehicle sensor data correlated with a time index and information derived from the raw vehicle sensor data correlated with the time index; receive ground-truth data associated with the driving log, wherein the ground-truth data includes, for one or more respective values of the time index, at least one of human-initiated disengagements of an autonomous driving system and human-entered annotations indicating mistakes made by the autonomous driving system; analyze the driving log using the automated autonomous driving evaluator to generate a report, wherein the automated autonomous driving evaluator is a computerized system that automatically analyzes driving logs to identify mistakes made by autonomous driving systems; compare automatically the report with the ground-truth data to generate a performance assessment for the automated autonomous driving evaluator; and provide the performance assessment to a development process in which the automated autonomous driving evaluator is modified based, at least in part, on the performance assessment to improve the performance of the automated autonomous driving evaluator. ### Claim 11 (depends on 10) The non-transitory computer-readable medium of claim 10, wherein: the ground-truth data is generated in an autonomous vehicle controlled by the autonomous driving system as the driving log is being recorded; and a human observer riding in the autonomous vehicle initiates the human-initiated engagements and enters the human-entered annotations. ### Claim 12 A method, comprising: receiving a driving log that includes at least one of raw vehicle sensor data correlated with a time index and information derived from the raw vehicle sensor data correlated with the time index; receiving ground-truth data associated with the driving log, wherein the ground-truth data includes, for one or more respective values of the time index, at least one of human-initiated disengagements of an autonomous driving system and human-entered annotations indicating mistakes made by the autonomous driving system; analyzing the driving log using an automated autonomous driving evaluator to generate a report, wherein the automated autonomous driving evaluator is a computerized system that automatically analyzes driving logs to identify mistakes made by autonomous driving systems; comparing automatically the report with the ground-truth data to generate a performance assessment for the automated autonomous driving evaluator; and providing the performance assessment to a development process in which the automated autonomous driving evaluator is modified based, at least in part, on the performance assessment to improve the performance of the automated autonomous driving evaluator. ### Claim 13 (depends on 12) The method of claim 12, wherein: the ground-truth data is generated in an autonomous vehicle controlled by the autonomous driving system as the driving log is being recorded; and a human observer riding in the autonomous vehicle initiates the human-initiated engagements and enters the human-entered annotations. ### Claim 14 (depends on 12) The method of claim 12, wherein the ground-truth data is generated while the driving log is being replayed in a simulator by a human observer who enters the human-entered annotations. ### Claim 15 (depends on 12) The method of claim 12, wherein the automated autonomous driving evaluator is designed to identify mistakes pertaining to a particular predetermined aspect of autonomous driving. ### Claim 16 (depends on 12) The method of claim 12, wherein the performance assessment for the automated autonomous driving evaluator includes one or more of false positives, false negatives, true positives, and true negatives. ### Claim 17 (depends on 12) The method of claim 12, wherein the driving log is recorded in an operating autonomous vehicle. ### Claim 18 (depends on 12) The method of claim 12, wherein the driving log is generated synthetically by a computing system. ### Claim 19 (depends on 12) The method of claim 12, wherein the automated autonomous driving evaluator is one of rule-based and machine-learning-based. ### Claim 20 (depends on 12) The method of claim 12, wherein: the driving log is recorded in an autonomous vehicle operating in a shadow mode in which the autonomous driving system simulates controlling the autonomous vehicle while the autonomous vehicle is being controlled by a human operator; and simulated actions taken by the autonomous driving system that deviate in a predetermined manner from actions taken by the human operator are automatically annotated as mistakes in the ground-truth data.