# US 10,311,726 B2 · Systems and methods for a parallel autonomy interface

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,311,726 B2
- Document ID: US10311726B2
- Status: Granted
- Kind: B2
- Application number: 15/656,422
- Filed: 2017-07-21
- Published: 2019-01-24
- Granted: 2019-06-04
- Earliest claimed: 2017-07-21
- Assignees: Toyota Research Institute, Inc.
- Applicants: Toyota Research Institute, Inc.
- Inventors: Andrew Patrikalakis; Philipp Michel
- Web page: https://patentgrep.com/patent/US10311726B2

## CPC

- G08G 1/09626: {where the origin of the information is within the own vehicle, e.g. a local storage device, digital map}
- B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
- B60K 35/29: Instruments characterised by the way in which information is handled, e.g. showing information on plural displays or prioritising information according to driving conditions
- B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes}
- B60K 35/23: Head-up displays [HUD] (optical aspects of head-up displays G02B27/01)
- B60K 35/26: using acoustic output
- B60K 2360/175: Autonomous driving
- B60K 2360/178: Warnings
- B60K 2360/179: Distances to obstacles or vehicles
- B60K 2360/191: Highlight information
- B60R 2300/207: using multi-purpose displays, e.g. camera image and navigation or video on same display
- B60R 2300/8093: for obstacle warning
- G06F 3/1423: {controlling a plurality of local displays, e.g. CRT and flat panel display}
- G09G 2380/10: Automotive applications
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

A parallel autonomy interface system can include processing circuitry communicably coupling one or more sensors of an autonomous vehicle and one or more displays of the autonomous vehicle. The processing circuitry can be configured to receive output from the one or more sensors, determine if potential danger with respect to vehicle operation is detected based on the output of the one or more sensors, activate an alert in response to detecting the potential danger, the alert corresponding to avatar motion, and enable the avatar to indicate a direction of the potential danger in response to activation of the alert.

## Claims

### Claim 1

A parallel autonomy interface system, comprising:
one or more sensors coupled to a vehicle capable of autonomous operation;
one or more displays configured to display an avatar, the avatar including at least a head and a hand; and
processing circuitry communicably coupling the one or more sensors and the one or more displays, the processing circuitry configured to
receive output from the one or more sensors,
determine if potential danger with respect to vehicle operation is detected based on the output of the one or more sensors,
activate an alert in response to detecting the potential danger, the alert corresponding to a change in a size of the avatar, and
cause the avatar to indicate a direction of the potential danger, in response to activation of the alert, by turning the head of the avatar to face the potential danger and pointing hand of the avatar towards the potential danger.

### Claim 2 (depends on 1)

The parallel autonomy interface system of claim 1, wherein the processing circuitry is configured to
display the avatar in a first location, the first location being within a predetermined area relative to a first focal point of the operator of the vehicle,
determine if the first focal point changes to a second focal point of the operator of the vehicle,
determine if the avatar is in the predetermined area relative to the first focal point in response to the first focal point changing to the second focal point, and
display the avatar in a second location in response to the avatar not being in the predetermined area relative to the second focal point, the second location being within the predetermined area relative to the second focal point.

### Claim 3 (depends on 2)

The parallel autonomy interface system of claim 2, wherein the predetermined area corresponds to a predetermined range of peripheral vision of the operator of the vehicle.

### Claim 4 (depends on 1)

The parallel autonomy interface system of claim 1, wherein the processing circuitry is configured to
determine if the one or more displays are producing sound, and
reduce a volume of the sound produced from the one or more displays in response to the potential danger being detected and the one or more displays producing sound.

### Claim 5 (depends on 2)

The parallel autonomy interface system of claim 2, wherein the first location corresponds to a first display and the second location corresponds to a second display.

### Claim 6 (depends on 1)

The parallel autonomy interface system of claim 1, wherein the avatar can be selected by the operator from a plurality of avatars.

### Claim 7 (depends on 1)

The parallel autonomy interface system of claim 1, wherein the avatar further indicates the direction of the potential danger by walking in the direction of the potential danger.

### Claim 8

A method of operating a parallel autonomy interface, comprising:
receiving output from one or more sensors, the one or more sensors communicably coupled to one or more displays;
determining, via processing circuitry, if potential danger with respect to vehicle operation is detected based on the output of the one or more sensors;
activating an alert in response to detecting the potential danger, the alert corresponding to a change in a size of a displayed avatar, the avatar including at least a head and a hand; and
causing the avatar to indicate a direction of the potential danger, in response to activation of the alert, by turning the head of the avatar to face the potential danger and pointing the hand of the avatar towards the potential danger.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
displaying the avatar, via the one or more displays, in a first location, the first location being within a predetermined area relative to a first focal point of an operator of the vehicle;
determining if the first focal point changes to a second focal point of the operator of the vehicle;
determining if the avatar is in the predetermined area relative to the first focal point in response to the first focal point changing to the second focal point; and
displaying the avatar, via the one or more displays, in a second location in response to the avatar not being in the predetermined area relative to the second focal point, the second location being within the predetermined area relative to the second focal point.

### Claim 10 (depends on 9)

The method of claim 9, wherein the predetermined area corresponds to a predetermined range of peripheral vision of the operator of the vehicle.

### Claim 11 (depends on 8)

The method of claim 8, further comprising:
determining if the one or more displays are producing sound; and
reducing a volume of the sound produced from the one or more displays in response to the potential danger being detected and the one or more displays producing sound.

### Claim 12 (depends on 9)

The method of claim 9, wherein the first location corresponds to a first display and the second location corresponds to a second display.

### Claim 13 (depends on 8)

The method of claim 8, wherein the avatar can be selected by an operator of the vehicle from a plurality of avatars.

### Claim 14 (depends on 8)

The method of claim 8, wherein the avatar further indicates the direction of the potential danger by walking in the direction of the potential danger.

### Claim 15

A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a computer, cause the computer to perform a method of operating a parallel autonomy interface, comprising:
receiving output from one or more sensors, the one or more sensors communicably coupled to one or more displays;
determining if potential danger with respect to vehicle operation is detected based on the output of the one or more sensors;
activating an alert in response to detecting the potential danger, the alert corresponding to a change in a size of a displayed avatar, the avatar including at least a head and a hand; and
causing the avatar to indicate a direction of the potential danger, in response to activation of the alert, by turning the head of the avatar to face the potential danger and pointing the hand of the avatar towards the potential danger.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
displaying the avatar, via the one or more displays, in a first location, the first location being within a predetermined area relative to a first focal point of an operator of the vehicle;
determining if the first focal point changes to a second focal point of the operator of the vehicle;
determining if the avatar is in the predetermined area relative to the first focal point in response to the first focal point changing to the second focal point; and
displaying the avatar, via the one or more displays, in a second location in response to the avatar not being in the predetermined area relative to the second focal point, the second location being within the predetermined area relative to the second focal point.

### Claim 17 (depends on 15)

The method of claim 15, further comprising:
determining if the one or more displays are producing sound; and
reducing a volume of the sound produced from the one or more displays in response to the potential danger being detected and the one or more displays producing sound.