# US 2016/0258776 A1 · Eye-gaze enabled navigation system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0258776 A1
- Document ID: US20160258776A1
- Status: Published application
- Kind: A1
- Application number: 15/156,171
- Filed: 2016-05-16
- Published: 2016-09-08
- Earliest claimed: 2013-12-09
- Assignees: Harman International Industries, Incorporated
- Applicants: Harman International Industries, Inc.
- Inventors: Davide Di CENSO; Stefan MARTI
- Web page: https://patentgrep.com/patent/US20160258776A1

## CPC

- G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera}
- B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
- B60K 35/285: {for improving awareness by directing driver's gaze direction or eye points}
- G01C 21/28: with correlation of data from several navigational instruments
- G01C 21/3626: {Details of the output of route guidance instructions (traffic control systems for road vehicles involving transmission of navigation instructions to the vehicle G08G1/0968)}
- G01C 21/3644: {Landmark guidance, e.g. using POIs or conspicuous other objects}
- G01C 21/3655: {Timing of guidance instructions}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A navigation system is configured to generate a sequence of driving instructions for directing a driver of a vehicle from a current location to a final destination. The navigation system is configured to capture sensor data that reflects the field of view of the driver and to identify objects in that field of view. The navigation system then customizes the driving instructions to reference specific objects, thereby generating context-specific driving instructions. The navigation system is also configured to identify particular objects upon which the driver is focused, and to generate context-specific driving instructions relative to such objects of focus. The navigation system may also provide correction information to the driver upon determining that the driver has incorrectly focused on a different object than an object referenced in a context-specific driving instruction.

## Claims

### Claim 1

A system configured to generate context-specific driving instructions for a driver of a vehicle, the system comprising:
an eye-gaze sensor configured to determine an eye-gaze direction of the driver;
a computing device configured to:
identify, based on the eye-gaze direction of the driver, a first object within the field of view of the driver, and
generate a context-specific driving instruction that references the first object; and
at least one output device configured to output the context-specific driving instruction to the driver.

### Claim 2 (depends on 1)

The system of claim 1, wherein the eye-gaze sensor is further configured to determine a depth at which the driver is focusing, and the computing device is further configured to identify the first object by determining, based on the depth at which the driver is focusing, that the driver is focused on the first object.

### Claim 3 (depends on 1)

The system of claim 1, wherein the computing device is further configured to:
determine that the driver is focused on a second object within the field of view of the driver instead of the first object based on information received from the eye-gaze sensor;
generate correction information that distinguishes the first object from the second object and indicates that the driver should be focused upon the first object; and
cause the at least one output device to output the correction information to the driver.

### Claim 4 (depends on 3)

The system of claim 3, wherein the computing device is further configured to:
receive a notification from the eye-gaze sensor indicating that the driver has shifted focus from the second object to the first object;
generate a confirmation message confirming that the driver is properly focused upon the first object instead of the second object; and
cause the at least one output device to output the confirmation message to the driver.

### Claim 5 (depends on 1)

The system of claim 1 wherein the computing device is further configured to:
receive a query from the driver that references a second object within the field of view of the driver;
generate an error message indicating that the driver is incorrectly focused upon the second object; and
cause the at least one output device to output the error message to the driver.

### Claim 6 (depends on 2)

The system of claim 2 wherein the computing device is further configured to:
receive a query from the driver that references the first object;
generate a confirmation message indicating that the driver is correctly focused upon the first object; and
cause the at least one output device to output the confirmation message to the driver.

### Claim 7 (depends on 1)

The system of claim 1, wherein the computing device is further configured to:
receive input from the driver specifying a final destination;
determine a current location of the vehicle;
generate a sequence of driving instructions to direct the driver from the current location to the final destination, wherein each driving instruction in the sequence of driving instructions indicates a particular driving action to be performed by the driver;
identify a current driving instruction in the sequence of driving instructions that is associated with the current location of the vehicle; and
generate the context-specific driving instruction by modifying the current driving instruction to include a reference to the first object.

### Claim 8

A non-transitory computer-readable medium including instructions that, when executed by a processor, configures the processor to generate context-specific driving instructions for a driver of a vehicle, by performing the steps of:
receiving an eye-gaze direction of the driver from an eye-gaze sensor;
identifying a first object based on the eye-gaze direction of the driver;
generating a context-specific driving instruction that references the first object; and
outputting the context-specific driving instruction to the driver.

### Claim 9 (depends on 8)

The non-transitory computer-readable medium of claim 8, further comprising the step of receiving a depth at which the driver is focusing from the eye-gaze sensor, and wherein identifying the first object further comprises determining, based on the depth at which the driver is focusing, that the driver is focused on the first object.

### Claim 10 (depends on 8)

The non-transitory computer-readable medium of claim 8, further comprising the steps of:
determining that the driver is focused on a second object within the visual panorama instead of the first object based on the information received from the eye-gaze sensor;
generating correction information indicating that the driver should be focused upon the first object and should not be focused on the second object; and
outputting the correction information to the driver.

### Claim 11 (depends on 10)

The non-transitory computer-readable medium of claim 10, further comprising the steps of:
receiving a notification from the eye-gaze sensor indicating that the driver has shifted focus from the second object to the first object;
generating a confirmation message based on the notification; and
outputting the confirmation message to the driver.

### Claim 12 (depends on 9)

The non-transitory computer-readable medium of claim 9, further comprising the steps of:
receiving a query from the driver regarding a second object within a field of view of the driver;
generating an error message indicating that the driver should ignore the second object; and
outputting the error message to the driver.

### Claim 13 (depends on 9)

The non-transitory computer-readable medium of claim 9, further comprising the steps of:
receiving a query from the driver regarding the first object;
generating a confirmation message indicating that the driver should continue to focus upon the first object; and
outputting the confirmation message to the driver.

### Claim 14 (depends on 8)

The non-transitory computer-readable medium of claim 8, further comprising the step of:
acquiring a sequence of driving instructions to direct the driver from a current location to a final destination, wherein each driving instruction in the sequence of driving instructions indicates a particular driving action to be performed by the driver;
identifying a current driving instruction in the sequence of driving instructions that indicates a subsequent driving action to be performed by the driver; and
generating the context-specific driving instruction by modifying the current driving instruction to include a reference to the first object.

### Claim 15

A computer-implemented method for generating context-specific driving instructions for a driver of a vehicle, the method comprising:
receiving, via an eye-gaze sensor, an eye-gaze direction of the driver;
identifying, based on the eye-gaze direction of the driver, a first object within a field of view of the driver;
generating a context-specific driving instruction that references the first object; and
outputting the context-specific driving instruction to the driver.

### Claim 16 (depends on 15)

The computer-implemented method of claim 15, further comprising:
determining that the driver is focused on a second object within the field of view of the driver instead of the first object based on information received from the eye-gaze sensor;
generating correction information that distinguishes the first object from the second object and indicates that the driver should be focused upon the first object; and
outputting the correction information to the driver.

### Claim 17 (depends on 16)

The computer-implemented method of claim 16, further comprising:
receiving a notification from the eye-gaze sensor indicating that the driver has shifted focus from the second object to the first object;
generating a confirmation message based on the notification; and
outputting the confirmation message to the driver.

### Claim 18 (depends on 15)

The computer-implemented method of claim 15, further comprising receiving a depth at which the driver is focusing from the eye-gaze sensor, and wherein identifying the first object further comprises determining, based on the depth at which the driver is focusing, that the driver is focused upon the first object.

### Claim 19 (depends on 15)

The computer-implemented method of claim 15, further comprising:
determining a current location of the vehicle;
acquiring a sequence of driving instructions to direct the driver from the current location to a final destination, wherein each driving instruction in the sequence of driving instructions indicates a particular driving action to be performed by the driver;
identifying a current driving instruction in the sequence of driving instructions that indicates a subsequent driving action to be performed by the driver; and
generating the context-specific driving instruction by modifying the current driving instruction to include a reference to the first object.

### Claim 20 (depends on 15)

The computer-implemented method of claim 15, wherein the computing device is configured to identify the first object by determining, based on the eye-gaze direction of the driver, that the driver is looking towards the first object.
