# US 12,222,716 B2 · Method and system for operating an autonomous agent with a remote operator > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,222,716 B2 - Document ID: US12222716B2 - Status: Granted - Kind: B2 - Application number: 18/075,778 - Filed: 2022-12-06 - Published: 2023-06-08 - Granted: 2025-02-11 - Earliest claimed: 2021-12-06 - Assignees: Gatik AI Inc. - Applicants: Gatik AI Inc. - Inventors: Apeksha Kumavat; Arjun Narang; Gautam Narang; Dharmateja Kadem; Gunnar Newquist - Web page: https://patentgrep.com/patent/US12222716B2 ## CPC - G05D 1/0027: {involving a plurality of vehicles, e.g. fleet or convoy travelling (fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00)} - G05D 1/0038: {by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246)} - B60W 50/0097: {Predicting future conditions} - B60W 50/0205: {Diagnosing or detecting failures; Failure detection models} - B60W 50/029: Adapting to failures or work around with other constraints, e.g. circumvention by avoiding use of failed parts - B60W 60/001: {Planning or execution of driving tasks} - B60W 60/00186: {related to the vehicle} - B60W 60/007: {Emergency override (Handing over between remote control and on-board control or handing over between remote control arrangements G05D1/227)} - G05D 1/0225: {involving docking at a fixed facility, e.g. base station or loading bay (parking aids B62D15/027)} - G05D 1/2247: {providing the operator with simple or augmented images from one or more cameras} - G05D 1/661: Docking at a base station (delivering or retrieving payloads G05D1/667) - B60W 2050/021: {Means for detecting failure or malfunction} - B60W 2050/0292: {Fail-safe or redundant systems, e.g. limp-home or backup systems} - B60W 2556/45: External transmission of data to or from the vehicle - B60W 2756/10: Involving external transmission of data to or from the vehicle - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT ## Abstract A system for operating an autonomous agent with a remote operator includes a remote subsystem, a local subsystem, and a communication subsystem. A method for operating an autonomous agent with a remote operator includes receiving and processing a set of inputs; presenting a set of outputs to a remote subsystem; receiving an input from a remote operator; and operating the autonomous agent. Additionally or alternatively, the method can include any or all of: detecting a failure and/or triggering a failure response; detecting a particular scenario associated with the autonomous agent; triggering a remote operator request; processing remote operator input; and/or any other processes. ## Claims ### Claim 1 A method for utilizing remote operator input in an operation of an autonomous vehicle, the method comprising: receiving a set of sensor inputs at a sensor subsystem of the autonomous vehicle; processing the set of sensor inputs with: a set of trained models at a computing subsystem of the autonomous vehicle to characterize a scenario associated with the autonomous vehicle; and a diagnostic aggregator of the autonomous vehicle to determine an aggregated predicted failure score of the autonomous vehicle; comparing the aggregated predicted failure score with a set of multiple predetermined thresholds, wherein each threshold is associated with a different category of severity for a single subsystem of the autonomous vehicle, the set of multiple predetermined thresholds comprising at least a first threshold and a second threshold, the second threshold separate and distinct from the first threshold, wherein: in an event that the aggregated predicted failure score exceeds the first threshold of the set of multiple predetermined thresholds, a first failure response is initiated; and in an event that the aggregated predicted failure score exceeds the second threshold of the set of multiple predetermined thresholds, a second failure response is initiated; determining that the characterized scenario is within a first subset of predetermined scenarios; responsive to determining that the characterized scenario is within the first subset of predetermined scenarios, presenting information to a remote operator, the information comprising a predefined set of behavior options for the vehicle; determining that the characterized scenario is within a second subset of predetermined scenarios; responsive to determining that the characterized scenario is within the second subset of predetermined scenarios, presenting a set of remote operator input options enabling the remote provider to provide a set of control commands for actuators of the autonomous vehicle to the remote operator receiving an input from the remote operator; comparing the input with a set of constraints determined based on the set of sensor inputs; validating the input by on comparing the input with the set of constraints; and maneuvering the autonomous vehicle based on the validated input. ### Claim 2 (depends on 1) The method of claim 1, wherein the set of remote operator input options is determined based on the characterized scenario, wherein the characterized scenario is determined to be within the second subset of the predetermined scenarios, and wherein the second subset of predetermined scenarios comprises an arrival of the autonomous vehicle at a loading dock site. ### Claim 3 (depends on 2) The method of claim 2, wherein the arrival of the autonomous vehicle at the loading dock site is detected based on a predetermined map and a geofence associated with the loading dock site. ### Claim 4 (depends on 2) The method of claim 2, wherein the information presented to the remote operator further comprises a notification from at least one of: a fleet command center, a second autonomous vehicle arranged at the loading dock site, or a user interface associated with a site worker located at the loading dock site, wherein the notification comprises an identifier for a particular loading dock at the loading dock site. ### Claim 5 (depends on 1) The method of claim 1, wherein the second predetermined threshold has a higher value than the first predetermined threshold. ### Claim 6 (depends on 1) The method of claim 1, wherein the first failure response comprises transmitting an alert to the remote operator, wherein the information comprises the alert. ### Claim 7 (depends on 6) The method of claim 6, wherein the first failure response is separate and distinct from the second failure response, and wherein the method further comprises initiating the second failure response, the second failure response comprising autonomously operating the autonomous vehicle according to a minimal risk maneuver selected from a set of multiple predetermined minimal risk maneuvers. ### Claim 8 (depends on 7) The method of claim 7, wherein the minimal risk maneuver is a first in the set of multiple predetermined minimal risk maneuvers, wherein the set of multiple predetermined minimal risk maneuvers is an ordered list of minimal risk maneuvers ordered in increasing intensity, wherein the selected minimal risk maneuver has a lowest intensity. ### Claim 9 (depends on 8) The method of claim 8, wherein the increasing intensity comprises an increasing value in braking applied by a set of brakes of the autonomous vehicle. ### Claim 10 (depends on 8) The method of claim 8, wherein maneuvering the autonomous vehicle based on the validated input prevents at least one of: the first failure response from escalating into the second failure response; and a second minimal risk maneuver in the ordered list of minimal risk maneuvers from being implemented. ### Claim 11 (depends on 1) The method of claim 1, wherein the second failure response comprises autonomously implementing a minimal risk maneuver for the autonomous vehicle, wherein the minimal risk maneuver is implemented absent of the input from the remote operator. ### Claim 12 (depends on 1) The method of claim 1, wherein at least a portion of the information presented to the remote operator are each associated with a time duration limit for monitoring for input from the remote operator. ### Claim 13 (depends on 12) The method of claim 12, wherein a first time duration limit is associated with information provided in response to initiation of the first failure response, and wherein a second time duration limit is associated with information provided in response to initiation of the second failure response, wherein the first duration limit is greater than the second duration limit. ### Claim 14 (depends on 13) The method of claim 13, wherein in an event that at least one of the first time duration limit and the second time duration limit is exceeded, the method further comprises implementing a minimal risk maneuver for the autonomous vehicle. ### Claim 15 (depends on 1) The method of claim 1, wherein the input from the remote operator comprises a selection of an alternative fixed route for the autonomous relative to a current fixed route being traversed by the autonomous vehicle. ### Claim 16 (depends on 1) The method of claim 1, further comprising: receiving a next set of sensor inputs at the sensor subsystem of the autonomous vehicle; determining a next characterized scenario using the set of trained models based on the next set of sensor inputs; and in an event that any or all are satisfied: the next characterized scenario is within the first subset of predetermined scenarios, wherein a next set of remote operator input options comprises a next set of behavior options for the vehicle; the next characterized scenario is within the second subset of predetermined scenarios, wherein the next set of remote operator input options enables the remote operator to provide at least one of: a next set of control commands for actuators of the autonomous vehicle; or a set of locations for the autonomous vehicle to pass through; or at least one of the first and second failure responses is initiated, presenting a next set of information to the remote operators, wherein the next set of information comprises: the next set of sensor inputs; and the next set of remote operator input options determined based on at least one of: the next characterized scenario; or a next aggregated predicted failure score determined by the diagnostic aggregator. ### Claim 17 (depends on 16) The method of claim 16, wherein providing the set of locations for the autonomous vehicle to pass through comprises providing at least one of: a set of waypoints for the autonomous vehicle and a path for the autonomous vehicle. ### Claim 18 (depends on 17) The method of claim 17, wherein the set of locations comprises the set of waypoints, wherein the autonomous vehicle generates a trajectory based on the set of waypoints and with a classical motion planner. ### Claim 19 (depends on 1) The method of claim 1, further comprising: automatically selecting the predefined set of behavior options from a plurality of predefined behavior options based on the set of sensor inputs before presenting the information to the remote operator. ### Claim 20 (depends on 1) The method of claim 1, wherein each threshold in the set of multiple predetermined thresholds is associated with a different subsystem of the autonomous vehicle. ### Claim 21 (depends on 1) The method of claim 1, wherein enabling the remote operator to provide a set of control commands comprises enabling direct remote control of the actuators of the autonomous vehicle. ### Claim 22 (depends on 21) The method of claim 21, wherein enabling direct remote control of the actuators of the autonomous vehicle comprises receiving physical control inputs from the remote operator at a physical input; and converting the physical control inputs into actuator control commands.