User Interface Techniques for Recommending Remote Assistance Actions

A system comprising:

• a vehicle having a vehicle sensor;
• and
• a computing system configured to:
• receive a request for assistance from the vehicle operating in an environment, wherein the request includes sensor data depicting the environment as captured by the vehicle sensor and data representing a behavior prediction for an agent located in the environment as determined by the vehicle;
• display, based on the sensor data, a representation of the environment of the vehicle showing the agent, wherein the representation conveys the behavior prediction for the agent;
• receive an input provided by an operator, the input modifying the behavior prediction for the agent;
• and
• provide, to the vehicle, a response based on the input modifying the behavior prediction for the agent, wherein the vehicle is configured to determine a control strategy for subsequent navigation based on the response.

A non-transitory computer-readable medium configured to store instructions, that when executed by a computing system comprising one or more processors, causes the computing system to perform operations comprising:

• receiving a request for assistance from a vehicle operating in an environment, wherein the request includes sensor data depicting the environment as captured by a vehicle sensor and data representing a behavior prediction for an agent located in the environment as determined by the vehicle;
• displaying, based on the sensor data, a representation of the environment of the vehicle showing the agent, wherein the representation conveys the behavior prediction for the agent;
• receiving an input provided by an operator, the input modifying the behavior prediction for the agent;
• and
• providing a response to the vehicle based on the input modifying the behavior prediction for the agent, wherein the vehicle is configured to determine a control strategy for subsequent navigation based on the response.