Autonomous system action explanation

A method for improving collaboration with a user of an autonomous system by identifying actions of the autonomous system that are unexpected to the user, the method comprising:

• determining, by an autonomous system executing on processing circuitry and configured to interact with the user, actions of the autonomous system for one or more tasks;
• tracking, by the autonomous system, previous tasks that were adopted, modified, or dropped by the autonomous system;
• storing, by the autonomous system, data that indicates a rationale for the autonomous system having adopted, modified, or dropped each task of the previous tasks;
• storing, by the autonomous system, plays that indicate a respective strategy for achieving each task of the previous tasks;
• generating, by a self-reflection unit executing on the processing circuitry, a model of a behavior of the autonomous system based on the stored tracked previous tasks, the stored plays, and the stored data, wherein the model is configured to expose decision making of the autonomous system;
• identifying, by an explanation unit executing on the processing circuitry and based on the model, one or more actions of the actions of the autonomous system for the one or more tasks;
• determining that the one or more actions result from one or more of:
• a deviation from one or more previous actions for the one or more tasks, or
• a selection, by the autonomous system, of at least two decisions that have indistinguishable optimizations;
• determining, by the explanation unit and based on the model, a context for the one or more actions and a rationale for the one or more actions, the context including information known to the autonomous system and used to determine the one or more actions, the rationale including information explaining reasoning used by the autonomous system to determine the one or more actions;
• processing, by a set of one or more classifiers of the explanation unit, the one or more actions, the context for the one or more actions, and the rationale for the one or more actions to output a probability the one or more actions are unexpected to the user;
• and
• outputting, by the explanation unit and for display, based on the probability the one or more actions are unexpected to the user exceeding a threshold, an explanation for the one or more actions comprising the context and the rationale for the one or more actions and outputting one or more of an acknowledgement of the deviation, or an explanation that the at least two decisions have indistinguishable optimizations, to improve collaboration with the user of the autonomous system.

A non-transitory, computer-readable medium comprising instructions, that, when executed, cause processing circuitry of a system to:

• determine actions of an autonomous system for one or more tasks;
• track, previous tasks that were adopted, modified, or dropped by the autonomous system;
• store data that indicates a rationale for the autonomous system having adopted, modified, or dropped each task of the previous tasks;
• store plays that indicate a respective strategy for achieving each task of the previous tasks;
• generate a model of a behavior of the autonomous system based on the tracked previous tasks, the stored plays, and the stored data, wherein the model is configured to expose decision making of the autonomous system;
• identify, based on the model, one or more actions of the actions of the autonomous system for the one or more tasks;
• determine that the one or more actions result from one or more of:
• a deviation from one or more previous actions for the one or more tasks, or
• a selection, by the autonomous system, of at least two decisions that have indistinguishable optimizations;
• determine, based on the model, a context for the one or more actions and a rationale for the one or more actions, the context including information known to the autonomous system and used to determine the one or more actions, the rationale including information explaining reasoning used by the autonomous system to determine the one or more actions;
• process, by a set of one or more classifiers, the one or more actions, the context for the one or more actions, and the rationale for the one or more actions to output a probability the one or more actions are unexpected to the user;
• and
• output, for display, based on the probability the one or more actions are unexpected to the user exceeding a threshold, an explanation for the one or more actions comprising the context and the rationale for the one or more actions and output one or more of an acknowledgement of the deviation, or an explanation that the at least two decisions have indistinguishable optimizations, to improve collaboration with the user of the autonomous system.