# US 2015/0177843 A1 · Device and method for displaying user interface of virtual input device based on motion recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0177843 A1
- Document ID: US20150177843A1
- Status: Published application
- Kind: A1
- Application number: 14/317,993
- Filed: 2014-06-27
- Published: 2015-06-25
- Earliest claimed: 2013-12-23
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Oh-yun KWON
- Web page: https://patentgrep.com/patent/US20150177843A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/03547: {Touch pads, in which fingers can move on a surface}
- H04N 21/42204: {User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor (constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205)}
- G06V 40/19: Sensors therefor
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- H04N 21/42222: {Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A display device includes a user interface (UI) generator configured to generate a UI of the virtual input device; a camera configured to photograph a user's gesture; and a controller configured to compare the photographed user's gesture with a user's gesture stored in the template and to control the UI generator to display a UI of a virtual input device corresponding to the matched user's gesture.

## Claims

### Claim 1

A display device comprising:
a user interface (UI) generator configured to generate a UI of a virtual input device;
a camera configured to photograph a user's gesture; and
a controller configured to compare the photographed user's gesture with the user's gestures stored in a template and to control the UI generator to display the UI of the virtual input device corresponding to the photographed user's gesture that has been matched to one of the user's gestures stored in the template.

### Claim 2 (depends on 1)

The display device of claim 1, wherein the virtual input device comprises at least one of a remote controller, a touch remote controller, a keyboard, and a mouse.

### Claim 3 (depends on 1)

The display device of claim 1, wherein the user's gesture comprises a grip gesture of a real input device.

### Claim 4 (depends on 1)

The display device of claim 1, further comprising a motion calculator configured to recognize a user's finger gesture and one of a user's facial direction and a user's gaze to operate the UI of the displayed virtual input device.

### Claim 5 (depends on 4)

The display device of claim 4, wherein the controller displays a pointer on the UI of the virtual input device based on the user's finger gesture that has been calculated by the motion calculator.

### Claim 6 (depends on 4)

The display device of claim 4, wherein the motion calculator identifies at least one of a movement direction, a movement quantity, and a location of the user's finger, and calculates a size of the pointer on the UI of the virtual input device with respect to an actual finger.

### Claim 7 (depends on 4)

The display device of claim 4, further comprising:
a storage configured to store at least one of a manipulating sound of the virtual input device according to the user's finger gesture, a voice announcing an execution of a function, and a voice announcing a name of a functional button of a virtual input device on which the user's finger is located.

### Claim 8 (depends on 4)

The display device of claim 4, wherein the motion calculator determines the user's gaze as a gaze state or a non-gaze state, with respect to a display, by comparing scatter and concentration of gaze recognition data within a time period with a threshold value.

### Claim 9 (depends on 8)

The display device of claim 8, wherein the controller resets a user's previous finger location if the user's gaze is determined to be the non-gaze state.

### Claim 10 (depends on 8)

The display device of claim 8, wherein the controller changes to a voice guiding mode if the user's gaze is determined to be the non-gaze state.

### Claim 11 (depends on 1)

The display device of claim 1, wherein the controller deletes, adds, and modifies a functional button of the UI of the virtual input device according to a user's command.

### Claim 12

A method for displaying a UI of a virtual input device, the method comprising:
storing a template containing links between user's gestures and virtual input devices;
photographing a user's gesture;
comparing the photographed user's gesture with the user's gestures registered with the template; and
displaying the UI of the virtual input device corresponding to the photographed user's gesture according to a comparison result.

### Claim 13 (depends on 12)

The method of claim 12, wherein the virtual input device comprises at least one of a remote controller, a touch remote controller, a keyboard, and a mouse.

### Claim 14 (depends on 12)

The method of claim 12, wherein the user's gesture comprises a grip gesture with respect to a real input device.

### Claim 15 (depends on 12)

The method of claim 12, further comprising recognizing a user's finger gesture and one of a user's facial direction and a user's gaze to operate the UI of the displayed virtual input device.

### Claim 16 (depends on 15)

The method of claim 15, further comprising displaying a pointer on the UI of the virtual input device based on the recognized user's finger gesture.

### Claim 17 (depends on 15)

The method of claim 15, wherein the recognizing comprises identifying at least one of a movement direction, a movement quantity, and a location of the user's finger and calculating a size of the pointer in the UI of the virtual input device with respect to an actual finger.

### Claim 18 (depends on 15)

The method of claim 15, further comprising storing at least one of a manipulating sound of a virtual input device according to the user's finger gesture, a voice announcing an execution of a function, and a voice announcing a name of a functional button of the virtual input device on which a user's finger is located.

### Claim 19 (depends on 15)

The method of claim 15, wherein the recognizing the user's gaze comprises determining a gaze state or a non-gaze state, with respect to a display, by comparing scatter and concentration of gaze recognition data within a time period with a threshold value.

### Claim 20 (depends on 19)

The method of claim 19, further comprising resetting a user's previous finger location if the user's gaze is determined to be the non-gaze state.

### Claim 21 (depends on 19)

The method of claim 19, further comprising automatically changing to a voice guiding mode if the user's gaze is determined to be the non-gaze state.

### Claim 22 (depends on 12)

The method of claim 12, further comprising at least one of deleting, adding, and modifying a functional button in the UI of the virtual input device according to a user's command.
