# US 2018/0024695 A1 · Detecting user interactions with a computing system of a vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0024695 A1
- Document ID: US20180024695A1
- Status: Published application
- Kind: A1
- Application number: 15/218,873
- Filed: 2016-07-25
- Published: 2018-01-25
- Earliest claimed: 2016-07-25
- Assignees: Google Inc.
- Applicants: Google Inc.
- Inventors: Rakesh Narayan Iyer; Joseph Onorato
- Web page: https://patentgrep.com/patent/US20180024695A1

## CPC

- G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)}
- B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
- B60K 35/21: using visual output, e.g. blinking lights or matrix displays
- B60K 35/22: Display screens
- B60K 35/26: using acoustic output
- B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
- B60K 35/50: Instruments characterised by their means of attachment to or integration in the vehicle (B60K35/231 takes precedence)
- B60K 35/60: Instruments characterised by their location or relative disposition in or on vehicles (arrangements of lighting devices on dashboards B60Q3/10)
- B60K 35/80: Arrangements for controlling instruments
- B60K 35/90: Calibration of instruments, e.g. setting initial or reference parameters; Testing of instruments, e.g. detecting malfunction
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/03547: {Touch pads, in which fingers can move on a surface}
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- H04N 23/20: for generating image signals from infrared radiation only
- B60K 2360/141: Activation of instrument input devices by approaching fingers or pens
- B60K 2360/1438: Touch screens
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A computing system of a vehicle for controlling a vehicle or systems within a vehicle. The computing system comprises a presence-sensitive panel within the vehicle, an infrared camera configured to capture images in a three-dimensional space within the vehicle, at least one processor; and at least one storage device that stores instructions. When the instructions are executed, they case the at least one processor to: receive, from the presence-sensitive panel, a first indication of input, receive, from the infrared camera, a second indication of input, and determine, based on at least one of the first indication of input or the second indication of input, an operation to be performed.

## Claims

### Claim 1

A computing system of a vehicle comprising:
a presence-sensitive panel within the vehicle;
an infrared camera configured to capture images in a three-dimensional space within the vehicle;
at least one processor; and
at least one storage device that stores instructions that, when executed, cause the at least one processor to:
receive, from the presence-sensitive panel, a first indication of input,
receive, from the infrared camera, a second indication of input, and
determine, based on at least one of the first indication of input or the second indication of input, an operation to be performed.

### Claim 2 (depends on 1)

The computing system of claim 1,
wherein the presence-sensitive panel is positioned between the infrared camera and the three-dimensional space and is transparent to infrared; and
wherein the infrared camera is positioned under the presence-sensitive panel and is oriented to capture images in the three-dimensional space above the presence-sensitive panel.

### Claim 3 (depends on 2)

The computing system of claim 2, wherein the presence-sensitive panel is positioned substantially horizontally on a central console of the vehicle.

### Claim 4 (depends on 1)

The computing system of claim 1,
wherein the presence-sensitive panel is configured to detect two-dimensional gesture inputs performed within a threshold distance of the presence-sensitive panel; and
wherein the infrared camera is configured to detect three-dimensional gesture inputs performed beyond the threshold distance of an input surface of the presence-sensitive panel and in the three-dimensional space.

### Claim 5 (depends on 1)

The computing system of claim 1, wherein determining the operation to be performed is based on both the first indication of input and the second indication of input.

### Claim 6 (depends on 1)

The computing system of claim 1, wherein determining the operation to be performed is based on the first indication of input.

### Claim 7 (depends on 6)

The computing system of claim 6, wherein the operation to be performed is a first operation to be performed, and wherein the instructions, when executed, further cause the at least one processor to:
responsive to determining the first operation to be performed, output, for display, information about the first operation to be performed;
determine, based on the second indication of input, a second operation to be performed; and
responsive to determining the second operation to be performed, output, for display, information about the second operation to be performed.

### Claim 8 (depends on 6)

The computing system of claim 6, further comprising a display screen, wherein the operation to be performed is a first operation to be performed, and wherein the instructions, when executed, further cause the at least one processor to:
responsive to determining the first operation to be performed, display on the display screen information about the first operation to be performed;
determine, based on the second indication of input, a second operation to be performed; and
responsive to determining the second operation to be performed, display on the display screen information about the second operation to be performed.

### Claim 9 (depends on 6)

The computing system of claim 6, further comprising a display screen, and wherein the instructions, when executed, further cause the at least one processor to:
responsive to determining the operation to be performed and prior to receiving the second indication of input, display on the display screen information requesting confirmation to perform the operation;
determine, based on the second indication of input, that the operation should be performed; and
perform the operation.

### Claim 10 (depends on 6)

The computing system of claim 6,
wherein the three-dimensional space has high ambient light conditions, and
wherein determining the operation to be performed is based on the second indication of input and includes determining gestures from images captured by the infrared camera in the high ambient light conditions.

### Claim 11 (depends on 6)

The computing system of claim 6,
wherein the three-dimensional space has low ambient light conditions, and
wherein determining the operation to be performed is based on the second indication of input and includes determining gestures from images captured by the infrared camera in the low ambient light conditions.

### Claim 12 (depends on 6)

The computing system of claim 6,
wherein the three-dimensional space includes structure in a field of view of the infrared camera, and
wherein determining the operation to be performed is based on the second indication of input and includes discerning gestures from the structure in the field of view.

### Claim 13

A method comprising:
receiving, by a computing system of a vehicle, from a presence-sensitive panel positioned within the vehicle, a first indication of input;
receiving, by the computing system, from an infrared camera configured to capture images in a three-dimensional space within the vehicle, a second indication of input; and
determining, based on at least one of the first indication of input or the second indication of input, an operation to be performed.

### Claim 14 (depends on 13)

The method of claim 13,
wherein the presence-sensitive panel is positioned between the infrared camera and the three-dimensional space and is transparent to infrared; and
wherein the infrared camera is positioned under the presence-sensitive panel and is oriented to capture images in the three-dimensional space above the presence-sensitive panel.

### Claim 15 (depends on 14)

The method of claim 14, wherein the presence-sensitive panel is positioned substantially horizontally on a central console of the vehicle.

### Claim 16 (depends on 13)

The method of claim 13,
wherein the presence-sensitive panel is configured to detect two-dimensional gesture inputs performed within a threshold distance of the presence-sensitive panel; and
wherein the infrared camera is configured to detect three-dimensional gesture inputs performed beyond the threshold distance of an input surface of the presence-sensitive panel and in the three-dimensional space.

### Claim 17 (depends on 13)

The method of claim 13, wherein determining the operation to be performed is based on both the first indication of input and the second indication of input.

### Claim 18 (depends on 13)

The method of claim 13, wherein determining the operation to be performed is based on the first indication of input.

### Claim 19 (depends on 18)

The method of claim 18, wherein the operation to be performed is a first operation to be performed, the method further comprising:
responsive to determining the first operation to be performed, outputting, for display, information about the first operation to be performed;
determining, based on the second indication of input, a second operation to be performed; and
responsive to determining the second operation to be performed, outputting, for display, information about the second operation to be performed.

### Claim 20

A computer-readable storage medium comprising instructions that, when executed by a computing system of a vehicle, cause at least one processor of the computing system to:
receive, from a presence-sensitive panel positioned within the vehicle, a first indication of input;
receive, from an infrared camera configured to capture images in a three-dimensional space within the vehicle, a second indication of input; and
determine, based on at least one of the first indication of input or the second indication of input, an operation to be performed.
