# US 10,501,014 B2 · Remote assist feedback system for autonomous vehicles > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,501,014 B2 - Document ID: US10501014B2 - Status: Granted - Kind: B2 - Application number: 15/918,043 - Filed: 2018-03-12 - Published: 2019-09-05 - Granted: 2019-12-10 - Earliest claimed: 2018-03-02 - Assignees: Uatc, LLC - Applicants: Uatc, LLC - Inventors: Alex Castro; Molly Nix; Scott Poeppel; Sean Chin; Eric Hanson; Dennis Zhao - Web page: https://patentgrep.com/patent/US10501014B2 ## CPC - B60Q 9/00: Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 {, e.g. haptic signalling} - G05D 1/224: Output arrangements on the remote controller, e.g. displays, haptics or speakers - G07C 5/008: {communicating information to a remotely located station (transmission systems for measured values G08C)} - G07C 5/0808: {Diagnosing performance data (testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005)} - G05D 2105/22: of humans - G05D 2107/13: Spaces reserved for vehicle traffic, e.g. roads, regulated airspace or regulated waters - G05D 2109/10: Land vehicles - H04W 4/44: for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H] - B60Q: ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract An interaction feedback system for passengers of an autonomous vehicle (AV) can detect a remote assist trigger, corresponding to a remote assist event, from a control system of the AV. The feedback system can then initiate a feedback mode using one or more feedback devices to enable interaction between one or more passengers of the AV and a remote operator based on detecting the remote assist trigger from the AV control system. ## Claims ### Claim 1 A passenger feedback system for an autonomous vehicle (AV), comprising: one or more feedback devices to output remote assist feedback for one or more passengers of the AV; and a feedback controller comprising one or more processing resources executing instructions that cause the feedback controller to: detect a remote assist trigger, corresponding to a remote assist event, from a control system of the AV; and based on detecting the remote assist trigger, transition from a normal operational mode to a feedback mode using the one or more feedback devices to enable two-way interaction between the one or more passengers and a remote operator, wherein the remote operator can transmit control commands to the AV while in feedback mode and the two-way interaction includes voice communication; and a control interface connecting the feedback controller to the control system of the AV. ### Claim 2 (depends on 1) The passenger feedback system of claim 1, wherein the executed instructions further cause the feedback controller to: based on the remote assist trigger, establish a connection, using the communication interface, with a transport service application executing on one or more computing devices of the one or more passengers of the AV to enable the two-way interaction between the one or more passengers and the remote operator. ### Claim 3 (depends on 1) The passenger feedback system of claim 1, further comprising a communication interface enabling the two-way interaction between the one or more passengers of the AV and the remote operator, wherein the communication interface comprises an on-board display screen displaying a user interface enabling the one or more passengers to communicate with the remote operator. ### Claim 4 (depends on 3) The passenger feedback system of claim 3, wherein the communication interface further comprises a voice input device enabling the one or more passengers to provide voice input to the remote operator. ### Claim 5 (depends on 1) The passenger feedback system of claim 1, wherein the remote operator comprises a human operator. ### Claim 6 (depends on 1) The passenger feedback system claim 1, wherein the remote operator comprises a computer-based, virtual operator. ### Claim 7 (depends on 1) The passenger feedback system of claim 1, wherein the one or more feedback devices comprise a plurality of output devices including at least one audio output device and at least one visual output device. ### Claim 8 (depends on 1) The passenger feedback system of claim 1, wherein the remote assist event corresponds to at least one of a perception fault, a connectivity fault, a mechanical fault, or a motion planning fault. ### Claim 9 (depends on 1) The passenger feedback system of claim 1, wherein the executed instructions further cause the feedback controller to: prior to initiating the feedback mode, initiate a transitional mode between the normal operational mode and the feedback mode, the transitional mode providing the one or more passengers an alert, using the one or more feedback devices, indicating that the AV has experienced the remote assist event. ### Claim 10 (depends on 1) The passenger feedback system of claim 1, wherein the remote assist event occurs at the autonomous vehicle. ### Claim 11 (depends on 1) The passenger feedback system of claim 1, wherein the feedback controller detects a remote assist trigger while in the normal operational mode of the autonomous vehicle. ### Claim 12 A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of an autonomous vehicle (AV), cause the one or more processors to: detect a remote assist trigger, corresponding to a remote assist event, from a control system of the AV; and based on detecting the remote assist trigger, transition from a normal operational mode to a feedback mode using one or more feedback devices to enable two-way interaction between one or more passengers of the AV and a remote operator, wherein the remote operator can transmit control commands to the AV while in feedback mode and the two-way interaction includes voice communication. ### Claim 13 (depends on 12) The non-transitory computer readable medium of claim 12, wherein the executed instructions further cause the one or more processors to: based on the remote assist trigger, establish a connection with a transport service application executing on one or more computing devices of the one or more passengers of the AV to enable the two-way interaction between the one or more passengers and the remote operator. ### Claim 14 (depends on 12) The non-transitory computer readable medium of claim 12, wherein the executed instructions cause the one or more processors to enable the two-way interaction between the one or more passengers and the remote operator via a communication interface, wherein the communication interface comprises an on-board display screen displaying a user interface enabling the one or more passengers to communicate with the remote operator. ### Claim 15 (depends on 14) The non-transitory computer readable medium of claim 14, wherein the communication interface further comprises a voice input device enabling the one or more passengers to provide voice input to the remote operator. ### Claim 16 (depends on 12) The non-transitory computer readable medium of claim 12, wherein the remote operator comprises a human operator. ### Claim 17 (depends on 12) The non-transitory computer readable medium of claim 12, wherein the remote operator comprises a computer-based, virtual operator. ### Claim 18 (depends on 12) The non-transitory computer readable medium of claim 12, wherein the one or more feedback devices comprise a plurality of output devices including at least one audio output device and at least one visual output device. ### Claim 19 (depends on 12) The non-transitory computer readable medium of claim 12, wherein the remote assist event corresponds to at least one of a perception fault, a connectivity fault, a mechanical fault, or a motion planning fault. ### Claim 20 A method of providing feedback to passengers of an autonomous vehicle (AV), then method being performed by one or more processors of the AV and comprising: detecting a remote assist trigger, corresponding to a remote assist event, from a control system of the AV; and based on detecting the remote assist trigger, transitioning from a normal operational mode to a feedback mode using one or more feedback devices to enable two-way interaction between one or more passengers of the AV and a remote operator, wherein the remote operator can transmit control commands to the AV while in feedback mode and the two-way interaction includes voice communication.