# US 2021/0278905 A1 · Depth sensing infrared input device and associated methods thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0278905 A1
- Document ID: US20210278905A1
- Status: Published application
- Kind: A1
- Application number: 16/809,438
- Filed: 2020-03-04
- Published: 2021-09-09
- Earliest claimed: 2020-03-04
- Assignees: Panasonic Avionics Corporation
- Applicants: Panasonic Avionics Corporation
- Inventors: Gurmukh Khabrani
- Web page: https://patentgrep.com/patent/US20210278905A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- B60R 11/0229: {for displays, e.g. cathodic tubes}
- B60R 11/0235: {of flat type, e.g. LCD}
- B60R 11/04: Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06V 20/64: Three-dimensional [3D] objects
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- B60R 2011/0014: {Arm-rests}
- B60R 2011/0015: {Back-rests}
- G06T 2207/10012: Stereo images
- G06T 2207/10048: Infrared image
- G06T 2207/20081: Training; Learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods and systems are provided for a hand tracking device. The hand tracking device includes a first infrared camera for capturing a plurality of images of a user hand indicating a first gesture for requesting a first action associated with operating a display device. The hand tracking device utilizes the plurality of images to generate a three-dimensional image of the user hand; and detects the first gesture from the three-dimensional image using a trained model that stores a plurality of input gestures corresponding to three-dimensional images of user hands. A first operation is executed in response to the detected first gesture. The hand tracking device continuously monitors the user hand to detect a second gesture from the trained model for executing a second operation.

## Claims

### Claim 1

(canceled)

### Claim 2

(canceled)

### Claim 3

(canceled)

### Claim 4

(canceled)

### Claim 5

(canceled)

### Claim 6

(canceled)

### Claim 7

A method, comprising:
capturing a plurality of images of a user hand by an infrared camera of a hand tracking device interfacing with a display device, the user hand indicating a first gesture requesting a first action associated with operating the display device;
wherein the plurality of images include a first image of the user hand taken directly from the user hand by the infrared camera, and at least a second image that is based on a reflection of the user hand at one of a plurality of mirrors located in the hand tracking device;
wherein the plurality of mirrors includes a first mirror and a second mirror and a third mirror, the first mirror and the second mirror are located across from each other and oriented in a vertical direction from a first surface that includes the infrared camera;
detecting by the hand tracking device, the first gesture from a three-dimensional image generated from the plurality of images;
wherein the hand tracking device utilizes a trained model to detect the first gesture based on stored plurality of input gestures, each gesture corresponding to one of a plurality three-dimensional images of user hands;
executing a first operation by the display device in response to the detected first gesture; and
monitoring the user hand by the hand tracking device using the infrared camera and the plurality of mirrors to detect a second gesture from the trained model for executing a second operation.

### Claim 8 (depends on 7)

The method of claim 7, wherein the infrared camera is located at a bottom surface of the hand tracking device to take the first image.

### Claim 9 (depends on 7)

The method of claim 7, wherein the plurality of images, the first gesture and the second gesture are used to update the trained model.

### Claim 10 (depends on 7)

The method of claim 7, wherein the plurality of images are captured by the infrared camera and another infrared camera, the infrared camera and the other infrared camera located at different surfaces of the hand tracking device at different elevations within the hand tracking device.

### Claim 11 (depends on 7)

The method of claim 7, wherein the third mirror is located adjacent to the first mirror and the second mirror, and oriented in a vertical direction from a bottom surface of the hand tracking device that includes the infrared camera.

### Claim 12 (depends on 7)

The method of claim 7, wherein the display device is located on an aircraft, integrated with an In-flight Entertainment (IFE) System, and the first operation and the second operation are associated with displaying content using the IFE system.

### Claim 13 (depends on 7)

The method of claim 7, wherein the display device is part of a seat device on a transportation vehicle, and the hand tracking device is integrated with an armrest of a passenger seat on the transportation vehicle.

### Claim 14 (depends on 7)

The method of claim 7, wherein a media file is played to instruct a user to move the user hand, when the captured plurality of images do not indicate a gesture recognized by the trained model.

### Claim 15

A hand tracking device, comprising:
a first infrared camera for capturing a plurality of images of a user hand indicating a first gesture for requesting a first action associated with operating a display device;
wherein the hand tracking device utilizes the plurality of images to generate a three-dimensional image of the user hand; and detects the first gesture from the three-dimensional image using a trained model that stores a plurality of input gestures corresponding to three-dimensional images of user hands;
wherein a first operation is executed in response to the detected first gesture; and the hand tracking device continuously monitors the user hand to detect a second gesture from the trained model for executing a second operation; and
a plurality of mirrors that provide reflections of the user hand, and the first infra-red camera takes a direct image of the user hand and images of the reflections of the user hand; wherein the plurality of mirrors includes a first mirror, a second mirror and a third mirror, the first mirror and the second mirror are located across from each other, and oriented in a vertical direction from a bottom surface of the hand tracking device that includes the first infrared camera.

### Claim 16 (depends on 15)

The hand held device of claim 15, further comprising: a second infrared camera located a different surface of the hand tracking device at a different elevation than a location of the first infrared camera for capturing the plurality of images of the user hand.

### Claim 17 (depends on 15)

The hand held device of claim 15, further comprising:
wherein the third mirror is located adjacent to the first mirror and the second mirror and oriented in a vertical direction from the bottom surface.

### Claim 18 (depends on 15)

The hand held device of claim 15, wherein the display device is located on an aircraft, integrated with an In-flight Entertainment (IFE) System, and the first operation and the second operation are associated with displaying content using the IFE system.

### Claim 19 (depends on 15)

The hand tracking device of claim 15, wherein the display device is part of a seat device on a transportation vehicle, and the hand tracking device is integrated with an armrest of a passenger seat on the transportation vehicle.

### Claim 20 (depends on 15)

The hand tracking device of claim 15, wherein a media file is played to instruct a user to move the user hand, when the captured plurality of images do not indicate a gesture recognized by the trained model.

### Claim 21

A non-transitory, machine-readable storage medium having stored thereon instructions for performing a method, comprising machine executable code which when executed by a hand tracking device, causes the hand tracking device to:
capture a plurality of images of a user hand by an infrared camera of the hand tracking device interfacing with a display device, the user hand indicating a first gesture requesting a first action associated with operating the display device;
wherein the plurality of images include a first image of the user hand taken directly from the user hand by the infrared camera, and at least a second image that is based on a reflection of the user hand at one of a plurality of mirrors located in the hand tracking device;
wherein the plurality of mirrors includes a first mirror and a second mirror and a third mirror, the first mirror and the second mirror are located across from each other and oriented in a vertical direction from a first surface that includes the infrared camera;
detect the first gesture from a three-dimensional image generated from the plurality of images based on a trained model that stores a plurality of input gestures, each gesture corresponding to one of a plurality three-dimensional images of user hands;
execute a first operation by the display device in response to the detected first gesture; and
monitor the user hand by the infrared camera and the plurality of mirrors to detect a second gesture from the trained model for executing a second operation.

### Claim 22 (depends on 21)

The non-transitory machine-readable storage medium of claim 21, wherein the infrared camera is located at a bottom surface of the hand tracking device to take the first image.

### Claim 23 (depends on 21)

The non-transitory machine-readable storage medium of claim 21, wherein the plurality of images, the first gesture and the second gesture are used to update the trained model.

### Claim 24 (depends on 21)

The non-transitory, machine-readable storage medium of claim 21, wherein the plurality of images is captured by the infrared camera and another infrared camera, the infrared camera and the other infrared camera located at different surfaces of the hand tracking device at different elevations within the hand tracking device.

### Claim 25 (depends on 21)

The non-transitory machine-readable storage medium of claim 21, wherein the third mirror is located adjacent to the first mirror and the second mirror, and oriented in a vertical direction from a bottom surface of the hand tracking device that includes the infra-red camera.

### Claim 26 (depends on 21)

The non-transitory machine-readable storage medium of claim 21, wherein the display device is located on an aircraft, integrated with an In-flight Entertainment (IFE) System, and the first operation and the second operation are associated with displaying content using the IFE system.

### Claim 27 (depends on 21)

The non-transitory machine-readable storage medium of claim 21, wherein the display device is part of a seat device on a transportation vehicle, and the hand tracking device is integrated with an armrest of a passenger seat on the transportation vehicle.

### Claim 28 (depends on 21)

The non-transitory machine-readable storage medium of claim 21, wherein a media file is played to instruct a user to move the user hand, when the captured plurality of images do not indicate a gesture recognized by the trained model.
