# US 12,223,778 B2 · Methods and systems for providing remote assistance to a stopped vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,778 B2 - Document ID: US12223778B2 - Status: Granted - Kind: B2 - Application number: 17/143,706 - Filed: 2021-01-07 - Published: 2021-04-29 - Granted: 2025-02-11 - Earliest claimed: 2017-07-11 - Assignees: Waymo LLC - Applicants: Waymo LLC - Inventors: Dmitri Dolgov; Joshua Herbach - Web page: https://patentgrep.com/patent/US12223778B2 ## CPC - G07C 5/008: {communicating information to a remotely located station (transmission systems for measured values G08C)} - B60W 30/00: Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units - B60W 60/0025: {specially adapted for specific operations} - G05D 1/0027: {involving a plurality of vehicles, e.g. fleet or convoy travelling (fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00)} - G05D 1/0038: {by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246)} - G05D 1/0044: {by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps (maps used for automatic navigation G05D1/0274; flight directors G01C23/005)} - G05D 1/2247: {providing the operator with simple or augmented images from one or more cameras} - G05D 1/225: operated by off-board computers - G05D 1/692: involving a plurality of disparate vehicles - B60W 2050/146: {Display means} - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2520/04: Vehicle stop - B60W 2540/215: Selection or confirmation of options - B60W 2556/45: External transmission of data to or from the vehicle - G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES ## Abstract Examples described may enable provision of remote assistance for an autonomous vehicle. An example method includes a computing system operating in a rewind mode. In the rewind mode, the system may be configured to provide information to a remote assistance operated based on a remote-assistance triggering criteria being met, such as determining that the autonomous vehicle has been stopped for a period of time. When the triggering criteria is met, the remote assistance system may provide data from the time leading up to when the remote-assistance triggering criteria was met that was capture of the environment of autonomous vehicle to the remote assistance operator. Based on viewing the data, the remote assistance operator may provide and input to the system that causes a command to be issued to the autonomous vehicle. ## Claims ### Claim 1 A method comprising: receiving, at a computing device, a request for assistance from a vehicle operating in an environment, wherein the computing device is positioned remotely from the vehicle, and wherein the request for assistance includes sensor data from a vehicle sensor; based on the request for assistance, determining, by the computing device and using the sensor data, that the vehicle is stopped blocking a driveway or in a double-parked arrangement; determining a threshold period of time based on the vehicle being stopped blocking the driveway or in the double-parked arrangement, wherein the computing device determines a different threshold period of time based on whether the vehicle is blocking the driveway or in the double-parked arrangement; determining that the vehicle has been stationary for the threshold period of time; based on determining that the vehicle has been stationary for the threshold period of time, prompting a remote operator to provide assistance and displaying, by the computing device using the sensor data, a representation of the environment of the vehicle, wherein the representation of the environment includes images or video that correspond to a predetermined period of time before the vehicle stopped and remained stationary for the threshold period of time; receiving, at the computing device, an input responsive to displaying the representation of the environment of the vehicle; and based on the input, providing control instructions to the vehicle. ### Claim 2 (depends on 1) The method of claim 1, wherein receiving the request for assistance from the vehicle operating in the environment comprises: receiving the request for assistance with one or more images from a vehicle camera, wherein the one or more images depict the environment of the vehicle. ### Claim 3 (depends on 2) The method of claim 2, wherein receiving the request for assistance with the one or more images from the vehicle camera further comprises: receiving at least one image that represents the environment of the vehicle prior to an event that caused the vehicle to transmit the request for assistance. ### Claim 4 (depends on 3) The method of claim 3, wherein the event that caused the vehicle to transmit the request for assistance corresponds to the vehicle remaining stopped for the threshold period of time. ### Claim 5 (depends on 1) The method of claim 1, wherein receiving the request for assistance from the vehicle operating in the environment comprises: receiving the request for assistance with video depicting the environment of the vehicle, wherein at least a portion of the video represents the environment of the vehicle prior to the vehicle transmitting the request for remote assistance. ### Claim 6 (depends on 1) The method of claim 1, wherein determining that the vehicle has been stationary for the threshold period of time comprises: determining that the vehicle has been stationary for the threshold period of time based on an indication received from the vehicle. ### Claim 7 (depends on 1) The method of claim 1, further comprising: estimating, using the sensor data, a reason that caused the vehicle to initiate and remain stationary for the threshold period of time; and displaying the representation of the environment of the vehicle with text that represents the reason that caused the vehicle to initiate and remain stationary for the threshold period of time. ### Claim 8 (depends on 7) The method of claim 7, wherein displaying the representation of the environment of the vehicle with text that represents the reason that caused the vehicle to initiate and remain stationary for the threshold period of time further comprises: display the text as a natural language question based on the reason. ### Claim 9 (depends on 8) The method of claim 8, wherein receiving the input responsive to displaying the representation of the environment of the vehicle comprises: receiving input that addresses the natural language question. ### Claim 10 (depends on 1) The method of claim 1, wherein providing control instructions to the vehicle comprises: providing control instructions that cause the vehicle to navigate in a particular direction. ### Claim 11 (depends on 1) The method of claim 1, wherein providing control instructions to the vehicle comprises: providing control instructions that cause the vehicle to remain stationary for a second threshold of time. ### Claim 12 (depends on 1) The method of claim 1, further comprising: determining, based on the sensor data included with the request for assistance, that the vehicle is stopped proximate the driveway; and determining the threshold period of time based on the vehicle being stopped proximate the driveway. ### Claim 13 (depends on 1) The method of claim 1, further comprising: determining, based on the sensor data included with the request for assistance, that the vehicle is in the double-parked arrangement with at least one parked vehicle; and determining the threshold period of time based on the vehicle being in the double-parked arrangement with at least one parked vehicle. ### Claim 14 (depends on 1) The method of claim 1, wherein the threshold period of time is a first threshold period of time when the vehicle is stopped proximate the driveway and a second threshold period of time when the vehicle is in the double-parked arrangement with at least one parked vehicle, and wherein the first threshold period of time and the second threshold period of time are different. ### Claim 15 A system comprising: a vehicle; a computing device positioned remotely from the vehicle, wherein the computing device is configured to: receive a request for assistance from the vehicle operating in an environment, wherein the request for assistance includes sensor data from a vehicle sensor; based on the request for assistance, determine, using the sensor data, that the vehicle is stopped blocking a driveway or in a double-parked arrangement; determine a threshold period of time based on the vehicle being stopped blocking the driveway or in the double-parked arrangement, wherein the computing device determines a different threshold period of time based on whether the vehicle is blocking the driveway or in the double-parked arrangement; determine that the vehicle has been stationary for the threshold period of time; based on determining that the vehicle has been stationary for the threshold period of time, prompt a remote operator to provide assistance and display, using the sensor data, a representation of the environment of the vehicle, wherein the representation of the environment includes images or video that correspond to a predetermined period of time before the vehicle stopped and remained stationary for the threshold period of time; receive an input responsive to displaying the representation of the environment of the vehicle; and based on the input, provide control instructions to the vehicle. ### Claim 16 (depends on 15) The system of claim 15, wherein the computing device is further configured to: receive the request for assistance with one or more images from a vehicle camera, wherein the one or more images depict the environment of the vehicle, and wherein at least one image from the one or more images represents the environment of the vehicle prior to an event that caused the vehicle to transmit the request for assistance. ### Claim 17 (depends on 16) The system of claim 16, wherein the computing device is further configured to: determine that the vehicle is waiting to pick up a passenger. ### Claim 18 (depends on 15) The system of claim 15, wherein the computing device is configured to: determine that the vehicle has been stationary for the threshold period of time based on an indication received from the vehicle, wherein the indication is received with the request for remote assistance. ### Claim 19 A non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor, cause the processor to perform operations comprising: receiving a request for assistance from a vehicle operating in an environment, wherein the request for assistance includes sensor data from a vehicle sensor; based on the request for assistance, determining, using the sensor data, that the vehicle is stopped blocking a driveway or in a double-parked arrangement, wherein the processor is positioned remotely from the vehicle and determines a different threshold period of time based on whether the vehicle is blocking the driveway or in the double-parked arrangement; determining a threshold period of time based on the vehicle being stopped blocking the driveway or in the double-parked arrangement; determining that the vehicle has been stationary for the threshold period of time; based on determining that the vehicle has been stationary for the threshold period of time, prompting a remote operator to provide assistance and displaying, using the sensor data, a representation of the environment of the vehicle, wherein the representation of the environment includes images or video that correspond to a predetermined period of time before the vehicle stopped and remained stationary for the threshold period of time; receiving an input responsive to displaying the representation of the environment of the vehicle; and based on the input, providing control instructions to the vehicle. ### Claim 20 (depends on 19) The non-transitory computer-readable medium of claim 19, further comprising: determining, using the sensor data, that the vehicle is doubled parked; and based on determining that the vehicle is doubled parked, determining a duration for the threshold period of time.