# US 2021/0334565 A1 · Systems and methods for performing operations in a vehicle using gaze detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334565 A1
- Document ID: US20210334565A1
- Status: Published application
- Kind: A1
- Application number: 16/859,741
- Filed: 2020-04-27
- Published: 2021-10-28
- Earliest claimed: 2020-04-27
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Jason Roche; Niranjan Avadhanam
- Web page: https://patentgrep.com/patent/US20210334565A1

## CPC

- G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness}
- B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
- B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
- B60K 35/29: Instruments characterised by the way in which information is handled, e.g. showing information on plural displays or prioritising information according to driving conditions
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06N 3/02: Neural networks
- G06N 20/00: Machine learning
- B60K 2360/148: Instrument input by voice
- B60K 2360/149: Instrument input by detecting viewing direction not otherwise provided for
- B60K 2360/164: Infotainment
- B60K 2360/199: for avoiding maloperation
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

In various examples, systems and methods are disclosed herein for a vehicle command operation system that may use technology across multiple modalities to cause vehicular operations to be performed in response to determining a focal point based on a gaze of an occupant. The system may utilize sensors to receive first data indicative of an eye gaze of an occupant of the vehicle. The system may utilize sensors to receive second data indicative of other data from the occupant. The system may then calculate a gaze vector based on the data indicative of the eye gaze of the occupant. The system may determine a focal point based on the gaze vector. In response to determining the focal point, the system causes an operation to be performed in the vehicle based on the second data.

## Claims

### Claim 1

A method of performing operations in a vehicle, the method comprising:
receiving, from one or more sensors, first data indicative of an eye gaze of an occupant in the vehicle;
receiving, from one or more other sensors, second data indicative of other data from the occupant;
calculating, using parallel processing circuitry that implements a machine learning model, a gaze vector based on the first data;
determining, using parallel processing circuitry, a focal point based on the gaze vector; and
responsive to determining the focal point, causing, using parallel processing circuitry, an operation to be performed in the vehicle based on the second data.

### Claim 2 (depends on 1)

The method of claim 1, wherein calculating the gaze vector comprises calculating the gaze vector using at least one hardware accelerator.

### Claim 3 (depends on 1)

The method of claim 1, wherein calculating the gaze vector comprises calculating the gaze vector using one more neural networks that receive the first data as input.

### Claim 4 (depends on 1)

The method of claim 1, wherein calculating the gaze vector based on the first data comprises:
determining a first focal point based on the gaze vector and a second focal point based on the gaze vector;
retrieving a historical focal point associated with the occupant, wherein the historical focal point is within a proximity threshold of the first focal point; and
determining the first focal point to be the focal point.

### Claim 5 (depends on 4)

The method of claim 4, wherein retrieving a historical focal point associated with the occupant comprises:
determining an operation count comprising a number of occurrences a particular operation was performed in the vehicle associated with the historical focal point; and
in response to the operation count exceeding a confidence value, retrieving the historical focal point associated with the occupant.

### Claim 6

A method of performing operations in a vehicle, the method comprising.
retrieving, from a memory storage, a spatial mapping of an interior of the vehicle;
receiving, from one or more sensors, first data indicative of an eye gaze of an occupant in the vehicle;
calculating, using parallel processing circuitry that implements a machine learning model, a gaze vector based on the first data;
determining, using parallel processing circuitry, a position of at least one intersection point where the gaze vector intersects with a region of the spatial mapping; and
in response to the determination of the position of at least one intersection point, causing, using parallel processing circuitry, an operation to be performed at a region within in the vehicle associated with the region of the spatial mapping.

### Claim 7 (depends on 6)

The method of claim 6, wherein determining the at least one intersection point further comprises:
receiving, from one or more sensors, second data indicative of the region of the spatial mapping from an occupant of the vehicle.

### Claim 8 (depends on 7)

The method of claim 7, wherein the second data comprises selection of the region via a graphical user interface by the occupant.

### Claim 9 (depends on 6)

The method of claim 6, wherein calculating the gaze vector comprises calculating the gaze vector using one more neural networks that receive the first data as input.

### Claim 10 (depends on 6)

The method of claim 6, wherein determining at least one intersection point the gaze vector intersects with a region of the spatial mapping comprises retrieving a historical intersection point associated with the occupant, wherein the historical intersection point is within a proximity threshold of the at least one intersection point.

### Claim 11

A system of performing an operation in a vehicle, the system comprising:
parallel processing circuitry configured to:
receive, from one or more sensors, first data indicative of an eye gaze of an occupant in the vehicle;
receive, from one or more other sensors, second data indicative of other data from the occupant;
calculate, using parallel processing circuitry that implements a machine learning model, a gaze vector based on the first data;
determine, using parallel processing circuitry, a focal point based on the gaze vector; and
responsive to determining the focal point, cause, using parallel processing circuitry, an operation to be performed in the vehicle based on the second data.

### Claim 12 (depends on 11)

The system of claim 11, wherein calculating the gaze vector comprises calculating the gaze vector using at least one hardware accelerator.

### Claim 13 (depends on 11)

The system of claim 11, wherein calculating the gaze vector comprises calculating the gaze vector using one more neural networks that receive the first data as input.

### Claim 14 (depends on 11)

The system of claim 11, wherein the parallel processing circuitry is configured, when calculating the gaze vector based on the first data, to:
determine a first focal point based on the gaze vector and a second focal point based on the gaze vector;
retrieve a historical focal point associated with the occupant, wherein the historical focal point is within a proximity threshold of the first focal point; and
determine the first focal point to be the focal point.

### Claim 15 (depends on 14)

The system of claim 14, wherein the parallel processing circuitry is configured, when retrieving a historical focal point associated with the occupant, to:
determine an operation count comprising a number of occurrences a particular operation was performed in the vehicle associated with the historical focal point; and
in response to the operation count exceeding a confidence value, retrieve the historical focal point associated with the occupant.

### Claim 16

A system of performing an operation in a vehicle, the system comprising:
parallel processing circuitry configured to:
retrieve, from a memory storage, a spatial mapping of an interior of the vehicle;
receiving, from one or more sensors, first data indicative of an eye gaze of an occupant in the vehicle;
calculate, using parallel processing circuitry that implements a machine learning model, a gaze vector based on the first data;
determine, using parallel processing circuitry, a position of at least one intersection point where the gaze vector intersects with a region of the spatial mapping; and
in response to the determination of the position of at least one intersection point, cause, using parallel processing circuitry, an operation to be performed at a region within in the vehicle associated with the region of the spatial mapping.

### Claim 17 (depends on 16)

The system of claim 16, wherein the parallel processing circuitry is configured, when determining the at least one intersection point, to further:
receive, from one or more sensors, second data indicative of the region of the spatial mapping from an occupant of the vehicle.

### Claim 18 (depends on 17)

The system of claim 17, wherein the second data comprises selection of the region via a graphical user interface by the occupant.

### Claim 19 (depends on 16)

The system of claim 16, wherein calculating the gaze vector comprises calculating the gaze vector using one more neural networks that receive the first data as input.

### Claim 20 (depends on 16)

The system of claim 16, wherein determining at least one intersection point the gaze vector intersects with a region of the spatial mapping comprises retrieving a historical intersection point associated with the occupant, wherein the historical intersection point is within a proximity threshold of the at least one intersection point.
