Explaining behavior by autonomous devices

A method of explaining a behavior of an autonomous device, the method comprising:

• obtaining, by a computation engine executed by one or more processors, first state data representative of a first observational state detected by the autonomous device that operates according to one or more trained machine learning models;
• obtaining, by the computation engine, first action data representative of one or more first actions the autonomous device performs responsive to detecting the first observational state;
• identifying, based on the first action data and by the computation engine, a first inflection point representative of changing behavior of the autonomous device;
• determining, based on the first inflection point and by the computation engine, first explanatory data representative of portions of the first state data on which the one or more trained machine learning models relied that explain the changing behavior of the autonomous device;
• and
• outputting, by the computation engine, the first explanatory data.

A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to execute a computation engine configured to:

• obtain first state data representative of a first observational state detected by an autonomous device, wherein the autonomous device is operating according to one or more trained machine learning models;
• obtain first action data representative of one or more first actions the autonomous device performs responsive to detecting the first observational state;
• identifying, based on the first action data, a first inflection point representative of changing behavior of the autonomous device;
• determining, responsive to identifying the first inflection point, first explanatory data representative of portions of the first state data on which the one or more trained machine learning models relied that explain the changing behavior of the autonomous device;
• and
• outputting the first explanatory data.