# US 2021/0240191 A1 · Using Plays for Human-AI Decision-Making in Remote Autonomous Vehicle Support > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0240191 A1 - Document ID: US20210240191A1 - Status: Published application - Kind: A1 - Application number: 16/779,393 - Filed: 2020-01-31 - Published: 2021-08-05 - Earliest claimed: 2020-01-31 - Assignees: Nissan North America, Inc. - Applicants: Nissan North America, Inc. - Inventors: Laura Cesafsky; Astha Vagadia; Hillary Abraham; Sparsh Sharma; Stefan Witwicki; Najamuddin Mirza Baig; Omar Bentahar - Web page: https://patentgrep.com/patent/US20210240191A1 ## CPC - G05D 1/0016: {characterised by the operator's input device (input arrangements for computing systems in general G06F3/00)} - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G05D 1/0038: {by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246)} - G06N 20/00: Machine learning - G08G 1/00: Traffic control systems for road vehicles (arrangement of road signs or traffic signals E01F9/00 {; automatic vehicle control B62D}) - G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)} - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) ## Abstract An apparatus for traversing a vehicle transportation network by an autonomous vehicle (AV) includes a human-autonomy teaming manager. The manager includes a first processor configured to initiate a dialogue associated with a predefined dynamic task sequence that is responsive to a condition experienced by the AV while traversing from a starting location to an ending location within the vehicle transportation network. The sequence is one of a plurality of predefined dynamic task sequences that uses multiple agents to resolve the condition. The manager receives, from an interface accessible to a human agent, an input responsive to the dialogue. The input confirms the sequence as a selected predefined dynamic task sequence or selects an other of the plurality of predefined dynamic task sequences as the selected predefined dynamic task sequence. The manager delegates tasks of the selected predefined dynamic task sequence to resolve the condition. ## Claims ### Claim 1 An apparatus for traversing a vehicle transportation network by an autonomous vehicle (AV), comprising: a human-autonomy teaming manager comprising a first processor configured to: initiate a dialogue associated with a predefined dynamic task sequence that is responsive to a condition experienced by the AV while traversing from a starting location to an ending location within the vehicle transportation network, wherein the predefined dynamic task sequence is one of a plurality of predefined dynamic task sequences that delegates tasks to multiple agents to resolve the condition; receive, from an interface accessible to a human agent of the multiple agents, an input responsive to the dialogue, wherein the input confirms the predefined dynamic task sequence as a selected predefined dynamic task sequence or selects an other predefined dynamic task sequence of the plurality of predefined dynamic task sequences as the selected predefined dynamic task sequence; and manage the selected predefined dynamic task sequence to resolve the condition. ### Claim 2 (depends on 1) The apparatus of claim 1, further comprising: an autonomous mobility agent of the multiple agents comprising a second processor configured to, responsive to the human-autonomy teaming manager initiating the dialogue, generate a display including the interface, the interface including the dialogue. ### Claim 3 (depends on 2) The apparatus of claim 2, wherein the dialogue describes at least one uncertainty associated with the predefined dynamic task sequence. ### Claim 4 (depends on 1) The apparatus of claim 1, wherein: the human-autonomy teaming manager comprises a memory storing the plurality of predefined dynamic task sequences within a library, the interface enabling access to the library by the human agent. ### Claim 5 (depends on 1) The apparatus of claim 1, wherein: an autonomous mobility agent of the multiple agents comprising a second processor configured to generate a representation for the human agent of how the predefined dynamic task sequence is projected to resolve the condition. ### Claim 6 (depends on 5) The apparatus of claim 5, wherein: the representation comprises projected positions for the AV while the condition is being resolved overlaying an image of a portion of the vehicle transportation network. ### Claim 7 (depends on 1) The apparatus of claim 1, further comprising: an autonomous mobility agent of the multiple agents comprising a second processor configured to: receive a request for assistance from the AV; and responsive to the request, generate a window on a display for confirmation of the request by the human agent, and transmit data associated with the condition to the human-autonomy teaming manager. ### Claim 8 (depends on 1) The apparatus of claim 1, wherein each of the plurality of predefined dynamic task sequences, when managed by the human-autonomy teaming manager, delegates a task to the human agent. ### Claim 9 (depends on 1) The apparatus of claim 1, wherein the condition comprises one of a plurality of conditions including an emergency condition of the AV, a behavioral condition of an occupant of the AV, or a driving condition of the AV. ### Claim 10 An apparatus for traversing a vehicle transportation network by an autonomous vehicle (AV), comprising: an autonomous mobility agent comprising a first processor configured to: receive a request for remote assistance support from the AV responsive to a condition observed by the AV while traversing from a starting location to an ending location within the vehicle transportation network; and a human-autonomy teaming manager comprising a second processor configured to: initiate a dialogue associated with a predefined dynamic task sequence that is one of a plurality of predefined dynamic task sequences using multiple agents to resolve the condition, the multiple agents including a human agent and the autonomous mobility agent; present the dialogue to the human agent at a user interface of the human-autonomy teaming manager; receive, from the user interface, an input responsive to the dialogue, wherein the input confirms the predefined dynamic task sequence as a selected predefined dynamic task sequence or selects an other predefined dynamic task sequence of the plurality of predefined dynamic task sequences as the selected predefined dynamic task sequence; and manage the selected predefined dynamic task sequence by delegating tasks of the selected predefined dynamic task sequence to resolve the condition, wherein the first processor is configured to, responsive to the request, call the human-autonomy teaming manager to select the predefined dynamic task sequence. ### Claim 11 (depends on 10) The apparatus of claim 10, wherein: the human-autonomy teaming manager delegates a task of the selected predefined dynamic task sequence to the human agent, the task comprising to label the condition for input to a machine-learning algorithm. ### Claim 12 (depends on 10) The apparatus of claim 10, wherein: the human-autonomy teaming manager delegates a task of the selected predefined dynamic task sequence to the autonomous mobility agent, the task comprising to instruct additional autonomous vehicles regarding their actions responsive to the condition. ### Claim 13 (depends on 10) The apparatus of claim 10, wherein: the human-autonomy teaming manager delegates a task of the selected predefined dynamic task sequence to the autonomous mobility agent, the task comprising to remotely control the AV. ### Claim 14 A method for traversing a vehicle transportation network by an autonomous vehicle (AV), comprising: initiating, by a first processor of a human-autonomy teaming manager, a dialogue associated with a predefined dynamic task sequence that is responsive to a condition experienced by the AV while traversing from a starting location to an ending location within the vehicle transportation network, wherein the predefined dynamic task sequence is one of a plurality of predefined dynamic task sequences using multiple agents to resolve the condition; receiving, from an interface accessible to a human agent of the multiple agents, an input responsive to the dialogue, wherein the input confirms the predefined dynamic task sequence as a selected predefined dynamic task sequence or selects an other predefined dynamic task sequence of the plurality of predefined dynamic task sequences as the selected predefined dynamic task sequence; and managing, by the first processor, the selected predefined dynamic task sequence to resolve the condition. ### Claim 15 (depends on 14) The method of claim 14, wherein managing the selected predefined dynamic task sequence to resolve the condition comprises: delegating at least one task of the selected predefined dynamic task sequence to the AV, the AV being one of the multiple agents. ### Claim 16 (depends on 14) The method of claim 14, wherein the dialogue includes a task for the human agent. ### Claim 17 (depends on 14) The method of claim 14, wherein the condition is one of a plurality of conditions, each of the plurality of conditions having a respective plurality of available predefined dynamic task sequences that use the multiple agents to resolve the condition. ### Claim 18 (depends on 14) The method of claim 14, further comprising: receiving, by a second processor of an autonomous mobility agent of the multiple agents, a request for assistance from the AV triggered by the condition; generating, by the second processor, a window on a display for confirmation of the request by the human agent; and transmitting, from the autonomous mobility agent, data associated with the condition to the human-autonomy teaming manager for selection of the predefined dynamic task sequence, wherein initiating the dialogue is responsive to the autonomous mobility agent receiving the confirmation of the request by the human agent. ### Claim 19 (depends on 14) The method of claim 14, further comprising: providing, to the human agent, access to a library through a dialogue window, the library storing the plurality of predefined dynamic task sequences. ### Claim 20 (depends on 14) The method of claim 14, further comprising: generating a representation, viewable to the human agent, of how the predefined dynamic task sequence is projected to resolve the condition.