# US 2019/0377423 A1 · User interface controlled by hand gestures above a desktop or a keyboard

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0377423 A1
- Document ID: US20190377423A1
- Status: Published application
- Kind: A1
- Application number: 16/438,372
- Filed: 2019-06-11
- Published: 2019-12-12
- Earliest claimed: 2018-06-12
- Inventors: Gergely Marton
- Web page: https://patentgrep.com/patent/US20190377423A1

## CPC

- G06F 3/023: Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The keyboard, the mouse, the touchpad and the touchscreen are among the most commonly used devices for issuing commands to computers. While the keyboard is a very effective device for generating text and basic input commands, the mouse and the touchpad are more suitable input devices for graphical user interfaces. The mouse and the touchpad are separated from the keyboard in space, and a computer user usually has to move her hands back and forth between these devices and the keyboard. Moreover, the mouse and the touchpad require additional hardware and free space e.g. on a laptop computer or on a desktop next to the keyboard. This invention provides a user interface system for a laptop or tablet computer allowing the user to control her device via hand gestures while keeping her hand or hands above the keyboard area of a laptop or comfortably in front of a tablet.

## Claims

### Claim 1

A controller system for a computer, wherein the computer comprises a display element and an operation system, the controller system comprising:
interaction surface located in front of the display element;
one or more video cameras wherein the one or more video cameras have fields of view in which fields of view the interaction surface is included;
gesture-recognition system with the capability of interpreting one or more gestures generated by one or more users, wherein the gesture-recognition system comprises logic which determines gestures from video data provided by the one or more video cameras and which logic determines one or more gestures based on at least one of the position, movement, orientation and shape properties of one or more predetermined objects within the video data;
capability of initiating one or more predetermined events when one or more gestures are determined by the gesture-recognition system, wherein the one or more predetermined events is one or more of the following: changing mode of operation of the controller system, control of a cursor, scrolling, zooming view, panning view, pressing a virtual button, character input, virtual icon input, clicking, double-clicking, launching a computer application, providing input for a computer application, changing one or more settings of the operation system of the computer, changing one or more settings of a computer application, switching between applications, expressing emotion on a social media platform.

### Claim 2 (depends on 1)

The controller system of claim 1, wherein the interaction surface contains at least a part of a keyboard of the computer.

### Claim 3 (depends on 1)

The controller system of claim 1, wherein the computer is a tablet computer, and provided the tablet computer is placed on a desktop, the interaction surface at least partly includes the desktop.

### Claim 4 (depends on 1)

The controller system of claim 1, wherein the at least one predetermined objects comprises at least one hand of a user of the computer.

### Claim 5 (depends on 1)

The controller system of claim 1, wherein the controller system initiates an event of changing mode of operation of the controller system in response to touching one or more predetermined hand parts together by a user of the computer.

### Claim 6 (depends on 1)

The controller system of claim 1, wherein the controller system initiates an event of changing mode of operation of the controller system in response to flexing one or more predetermined fingers in one or more predetermined directions by a user of the computer.

### Claim 7 (depends on 1)

The controller system of claim 1, wherein the controller system initiates an event of clicking or double-clicking in response to a predetermined movement or series of predetermined movements of one or more predetermined fingers of a hand of a user of the computer, wherein the predetermined movement or series of predetermined movements are evaluated relative to one or more predetermined parts of the said hand of the said user of the computer.

### Claim 8 (depends on 1)

The controller system of claim 1, wherein the controller system initiates an event of scrolling in response to a predetermined movement or series of predetermined movements of one or more predetermined fingers of a hand of a user of the computer, wherein the predetermined movement or series of predetermined movements are evaluated relative to one or more predetermined parts of the said hand of the said user of the computer.

### Claim 9 (depends on 1)

The controller system of claim 1, wherein the controller system initiates an event of zooming view in response to a predetermined movement or series of predetermined movements of one or more predetermined fingers of a hand of a user of the computer, wherein the predetermined movement or series of predetermined movements are evaluated relative to one or more predetermined parts of the said hand of the said user of the computer.

### Claim 10 (depends on 1)

The controller system of claim 1, wherein:
the controller system initiates an event equivalent of a left mouse click in response to a predetermined movement or series of predetermined movements of an index finger of a hand of a user of the computer, wherein the predetermined movement or series of predetermined movements of the index finger are evaluated relative to one or more predetermined parts of the said hand;
the controller system initiates an event equivalent of a right mouse click in response to a predetermined movement or series of predetermined movements of a pinky finger of a hand of a user of the computer, wherein the predetermined movement or series of predetermined movements of the pinky finger are evaluated relative to one or more predetermined parts of the said hand;
the controller system initiates a scrolling event in response to a predetermined movement or series of predetermined movements of a predetermined fingertip of a hand of a user of the computer, wherein the predetermined movement or series of predetermined movements of the predetermined fingertip are evaluated relative to a thumb of the said hand.

### Claim 11

A method for controlling a computer comprising:
interaction surface located in front of a display element of the computer;
utilization of video data from one or more video cameras wherein the one or more video cameras have fields of view in which fields of view an interaction surface is included;
interpreting one or more gestures generated by one or more users with a gesture-recognition system, wherein the gesture-recognition system comprises logic which determines gestures from the video data provided by the one or more video cameras and which logic determines one or more gestures based on at least one of the position, movement, orientation and shape properties of one or more predetermined objects within the video data;
initiation of one or more predetermined events when one or more gestures are determined by the gesture-recognition system, wherein the one or more predetermined events is one or more of the following: changing mode of operation within the method for controlling the computer, control of a cursor, scrolling, zooming view, panning view, pressing a virtual button, character input, virtual icon input, clicking, double-clicking, launching a computer application, providing input for a computer application, changing one or more settings of the operation system of the computer, changing one or more settings of a computer application, switching between applications, expressing emotion on a social media platform.

### Claim 12 (depends on 11)

The method of claim 11, wherein the interaction surface contains at least a part of a keyboard of the computer.

### Claim 13 (depends on 11)

The method of claim 11, wherein the computer is a tablet computer, and provided the tablet computer is placed on a desktop, the interaction surface at least partly includes the desktop.

### Claim 14 (depends on 11)

The method of claim 11, wherein the at least one predetermined objects comprises at least one hand of a user of the computer.

### Claim 15 (depends on 11)

The method of claim 11, wherein an event of changing mode of operation within the method for controlling the computer is initiated in response to touching one or more predetermined hand parts together by a user of the computer.

### Claim 16 (depends on 11)

The method of claim 11, wherein an event of changing mode of operation within the method for controlling the computer is initiated in response to flexing one or more predetermined fingers in one or more predetermined directions by a user of the computer.

### Claim 17 (depends on 11)

The method of claim 11, wherein an event of clicking or double-clicking is initiated in response to a predetermined movement or series of predetermined movements of one or more predetermined fingers of a hand of a user of the computer, wherein the predetermined movement or series of predetermined movements are evaluated relative to one or more predetermined parts of the said hand of the said user of the computer.

### Claim 18 (depends on 11)

The method of claim 11, wherein an event of scrolling is initiated in response to a predetermined movement or series of predetermined movements of one or more predetermined fingers of a hand of a user of the computer, wherein the predetermined movement or series of predetermined movements are evaluated relative to one or more predetermined parts of the said hand of the said user of the computer.

### Claim 19 (depends on 11)

The method of claim 11, wherein an event of zooming view is initiated in response to a predetermined movement or series of predetermined movements of one or more predetermined fingers of a hand of a user of the computer, wherein the predetermined movement or series of predetermined movements are evaluated relative to one or more predetermined parts of the said hand of the said user of the computer.

### Claim 20 (depends on 11)

The method of claim 11, wherein:
an event equivalent of a left mouse click is initiated in response to a predetermined movement or series of predetermined movements of an index finger of a hand of a user of the computer, wherein the predetermined movement or series of predetermined movements of the index finger are evaluated relative to one or more predetermined parts of the said hand;
an event equivalent of a right mouse click is initiated in response to a predetermined movement or series of predetermined movements of a pinky finger of a hand of a user of the computer, wherein the predetermined movement or series of predetermined movements of the pinky finger are evaluated relative to one or more predetermined parts of the said hand;
a scrolling event is initiated in response to a predetermined movement or series of predetermined movements of a predetermined fingertip of a hand of a user of the computer, wherein the predetermined movement or series of predetermined movements of the predetermined fingertip are evaluated relative to a thumb of the said hand.
