# US 2021/0240190 A1 · Explainability of Autonomous Vehicle Decision Making > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0240190 A1 - Document ID: US20210240190A1 - Status: Published application - Kind: A1 - Application number: 16/778,890 - Filed: 2020-01-31 - Published: 2021-08-05 - Earliest claimed: 2020-01-31 - Assignees: Nissan North America, Inc.; University of Massachusetts; RENAULT s.a.s. - Applicants: Nissan North America, Inc.; The University of Massachusetts; Renault S.A.S. - Inventors: Kyle Hollins Wray; Stefan Witwicki; Shlomo Zilberstein; Omar Bentahar; Arec Jamgochian - Web page: https://patentgrep.com/patent/US20210240190A1 ## CPC - G06F 40/56: Natural language generation - B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention - B60W 60/0011: {involving control alternatives for a single driving scenario, e.g. planning several paths to avoid obstacles} - G05D 1/0219: {ensuring the processing of the whole working surface} - G05D 1/0221: {involving a learning process} - G06F 40/186: Templates - B60W 60/001: {Planning or execution of driving tasks} - B60W 2050/146: {Display means} - G06F 11/3013: {where the computing system is an embedded system, i.e. a combination of hardware and software dedicated to perform a certain function in mobile devices, printers, automotive or aircraft systems (testing or monitoring of control systems or parts thereof G05B23/02)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES ## Abstract A processor is configured to execute instructions stored in a memory to identify distinct vehicle operational scenarios; instantiate decision components, where each of the decision components is an instance of a respective decision problem, and where the each of the decision components maintains a respective state describing the respective vehicle operational scenario; receive respective candidate vehicle control actions from the decision components; select an action from the respective candidate vehicle control actions, where the action is from a selected decision component of the decision components, and where the action is used to control the AV to traverse a portion of the vehicle transportation network; and generate an explanation as to why the action was selected, where the explanation includes respective descriptors of the action, the selected decision component, and a state factor of the respective state of the selected decision component. ## Claims ### Claim 1 An apparatus for traversing a vehicle transportation network by an autonomous vehicle (AV), comprising: a memory; and a processor, the processor configured to execute instructions stored in the memory to: identify distinct vehicle operational scenarios; instantiate decision components, wherein each of the decision components is an instance of a respective decision problem that models a respective distinct vehicle operational scenario of the distinct vehicle operational scenarios, and wherein the each of the decision components maintains a respective state describing the respective vehicle operational scenario; receive respective candidate vehicle control actions from the decision components; select an action from the respective candidate vehicle control actions, wherein the action is from a selected decision component of the decision components, and wherein the action is used to control the AV to traverse a portion of the vehicle transportation network; control the AV to traverse the portion of the vehicle transportation network using the action; and generate an explanation as to why the action was selected, wherein the explanation comprises respective descriptors of the action, the selected decision component, and a state factor of the respective state of the selected decision component. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the explanation indicates a level of certainty or uncertainty of the selected decision component with respect to the state factor. ### Claim 3 (depends on 1) The apparatus of claim 1, wherein the respective decision problem corresponding to the selected decision component maintains a prioritized list of state factors and the state factor of the explanation being either a lowest or highest priority state factor of the prioritized list of state factors. ### Claim 4 (depends on 1) The apparatus of claim 1, wherein the selected decision component is a partially observable Markov decision process (POMDP) and the state factor corresponding to the state factor. ### Claim 5 (depends on 1) The apparatus of claim 1, wherein the instructions further comprise instructions to: construct the explanation by inserting values into a template. ### Claim 6 (depends on 5) The apparatus of claim 5, wherein the template has a format “I because I had about for ,” and wherein each of , , , and is a placeholder for a respective semantic descriptor. ### Claim 7 (depends on 1) The apparatus of claim 1, wherein the instructions further comprise instructions to: receive the action from a second selected decision component, wherein the explanation further comprising the second selected decision component and a second state factor of a second state of the second selected decision component. ### Claim 8 (depends on 1) The apparatus of claim 1, wherein the explanation is provided to at least one of a tele-operator of the AV or an occupant of the AV. ### Claim 9 (depends on 1) The apparatus of claim 1, wherein the instructions further comprise instructions to: output the explanation in at least one of an graphical, textual, audio, or haptic format. ### Claim 10 (depends on 9) The apparatus of claim 9, wherein the explanation is output in response to a request from an occupant of the AV for the explanation. ### Claim 11 (depends on 1) The apparatus of claim 1, wherein the instructions further comprise instructions to: output the explanation to a log, wherein the log comprises historical actions of controlling the AV. ### Claim 12 A method for use in traversing a vehicle transportation network by an autonomous vehicle (AV), the method comprising: identifying distinct vehicle operational scenarios; solving, to provide candidate actions, each of the distinct vehicle operational scenarios using respective decision components, wherein each of the respective decision components maintains a semantic state of the respective vehicle operational scenario; controlling the AV to traverse a portion of the vehicle transportation network based on a selected action of the candidate actions, wherein the selected action is from a selected decision component of the respective decision components; and generating, using the semantic state, an explanation of the selected action, wherein the explanation comprises the selected action. ### Claim 13 (depends on 12) The method of claim 12, wherein solving each of the distinct vehicle operational scenarios using the respective decision components comprising: identifying a respective decision problem for at least some of the each of the distinct vehicle operational scenarios, wherein the respective decision problem is solved offline to provide a policy for the each of the distinct vehicle operational scenarios; and instantiating the respective decision problems to provide the respective decision components. ### Claim 14 (depends on 12) The method of claim 12, wherein the explanation indicates a level of certainty or uncertainty of the selected decision component with respect to at least one state factor. ### Claim 15 (depends on 14) The method of claim 14, wherein the selected decision component maintains a prioritized list of state factors and the at least one state factor of the explanation being either a lowest or highest priority state factor of the prioritized list of state factors. ### Claim 16 (depends on 12) The method of claim 12, further comprising: constructing the explanation by inserting values into a template, wherein the template has a format “I because I had about for ,” and wherein each of , , , and is a placeholder for a respective semantic descriptor. ### Claim 17 (depends on 12) The method of claim 12, further comprising: outputting the explanation in at least one of an graphical, textual, audio, or haptic format. ### Claim 18 (depends on 12) The method of claim 12, further comprising: outputting the explanation to a log, wherein the log comprises historical actions of controlling the AV. ### Claim 19 A method for use in traversing a vehicle transportation network by an autonomous vehicle (AV), the method comprising: in response to identifying at least one vehicle operational scenario, selecting an action for controlling the AV, wherein the action is provided by a decision component, wherein the decision component selects the action based on at least one state factor associated with the at least one vehicle operational scenario, and wherein a respective human-understandable, semantic meaning is associated with the at least one state factor; controlling the AV according to the action; and outputting an explanation of the action, wherein the explanation comprises the action, the at least one state factor, and the decision component. ### Claim 20 (depends on 19) The method of claim 19, wherein outputting the explanation of the action comprises: constructing the explanation by inserting values into a template having respective placeholders for the action, an importance measure corresponding to the at least one state factor, the at least one state factor, and the decision component.