Performance testing for mobile robot trajectory planners

A computer system comprising one or more computers configured to implement the method of evaluating the performance of a trajectory planner for a mobile robot in a real or simulated scenario, the method comprising:

• receiving scenario ground truth of the scenario, the scenario ground truth generated using the trajectory planner to control an ego agent of the scenario responsive to at least one other agent of the scenario, and comprising an ego trace of the ego agent and an agent trace of the other agent:
• evaluating the ego trace, by a test oracle, in order to assign at least one time series of test results to the ego agent, the time-series of test results pertaining to at least one performance evaluation rule:
• extracting one or more predetermined blame assessment parameters based on the agent trace;
• and
• applying one or more predetermined blame assessment rules to the blame assessment parameters, and thereby determining whether failure on the at least one performance evaluation rule is acceptable.

One or more computer programs, embodied in non-transitory media, and configured when executed by one or more computers to implement the method of evaluating the performance of a trajectory planner for a mobile robot in a real or simulated scenario, the method comprising:

• receiving scenario ground truth of the scenario, the scenario ground truth generated using the trajectory planner to control an ego agent of the scenario responsive to at least one other agent of the scenario, and comprising an ego trace of the ego agent and an agent trace of the other agent:
• evaluating the ego trace, by a test oracle, in order to assign at least one time series of test results to the ego agent, the time-series of test results pertaining to at least one performance evaluation rule;
• extracting one or more predetermined blame assessment parameters based on the agent trace;
• and
• applying one or more predetermined blame assessment rules to the blame assessment parameters, and thereby determining whether failure on the at least one performance evaluation rule is acceptable.