# US 9,430,039 B2 · Apparatus for controlling virtual mouse based on hand motion and method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,430,039 B2
- Document ID: US9430039B2
- Status: Granted
- Kind: B2
- Application number: 14/324,447
- Filed: 2014-07-07
- Published: 2015-01-15
- Granted: 2016-08-30
- Earliest claimed: 2013-07-15
- Assignees: Korea Electronics Technology Institute
- Applicants: Korea Electronics Technology Institute
- Inventors: Yang Keun Ahn; Kwang Mo Jung
- Web page: https://patentgrep.com/patent/US9430039B2

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46)
- G01S 17/89: for mapping or imaging
- G06F 3/005: {Input arrangements through a video camera}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

Provided are an apparatus and method for controlling a virtual mouse based on a hand motion. The apparatus includes an image processing unit configured to measure a maximum horizontal width of a hand region of a user and a width of an index finger in any one of a first depth image captured in a state in which the index finger is spread out in a vertical direction with his or her fist and a second depth image captured in a state in which the index finger is folded with the fist, a hand motion recognizing unit configured to compare the maximum horizontal width and the width of the index finger to recognize whether the index finger is folded, and a function matching unit configured to match a change in a folded state of the index finger to a pre-set first function and output a control signal for the function.

## Claims

### Claim 1

An apparatus for controlling a virtual mouse based on a hand motion, the apparatus comprising:
an image processing unit configured to measure a maximum horizontal width of a hand region of a user and a width of a finger in any one of a first depth image captured in a state in which the finger is spread out in a vertical direction with his or her fist and a second depth image captured in a state in which the finger is folded with the fist;
a hand motion recognizing unit configured to compare the maximum horizontal width and the width of the finger to recognize whether the finger is folded; and
a function matching unit configured to match a change in a folded state of the finger to a pre-set first function and output a control signal for implementing the pre-set first function,
wherein the image processing unit detects the hand region of the user using foreground/background separation from the first depth image and the second depth image, generates a horizontal histogram from a hand region image of the user, and measures the maximum horizontal width at a point having a maximum pixel number.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the image processing unit determines a region of interest by a predefined length specified from an upper end of the horizontal histogram, and determines an average width of the hand region within the region of interest, as the width of the finger.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the hand motion recognizing unit compares the width of the finger with a width corresponding to a predefined percentage of the maximum horizontal width, and when the width of the finger is less than the width corresponding to the predefined percentage, the hand motion recognizing unit recognizes the finger in a spread state, and when the width of the finger is equal to or greater than the width corresponding to the predefined percentage, the hand motion recognizing unit recognizes the finger in the folded state.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the image processing unit generates a distance-transformed image in units of pixels with respect to an image of the hand region of the user, and detects a pixel having the greatest value in the distant-transformed image, as a marker.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the hand motion recognizing unit recognizes a change in a position of the hand using the marker, and the function matching unit matches the change in the position of the hand to a pre-set second function and outputs a control signal for implementing the pre-set second function.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the image processing unit comprises:
a foreground/background separating unit configured to separate a foreground and a background on the basis of the depth information from the first depth image and the second depth image;
a horizontal hand width measuring unit configured to measure the maximum horizontal width of the hand region image of the user from which the background has been separated;
a horizontal finger width measuring unit configured to measure the finger width in the hand region image of the user; and
a marker detecting unit configured to detect a marker to be used to recognize a change in a position of the hand from the hand region image of the user.

### Claim 7

A method for controlling a virtual mouse based on a hand motion, the method comprising:
measuring a maximum horizontal width of a hand region of a user and a width of a finger in any one of a first depth image captured in a state in which the finger is spread out in a vertical direction with his or her fist and a second depth image captured in a state in which the finger is folded with the fist;
comparing the maximum horizontal width and the width of the finger to recognize whether the finger is folded; and
matching a change in a folded state of the finger to a pre-set first function and outputting a control signal for implementing the pre-set first function,
wherein the measuring of the maximum horizontal width and the width of the finger comprises:
detecting the hand region of the user using foreground/background separation from the first depth image and the second depth image, generating a horizontal histogram from a hand region image of the user, and measuring the maximum horizontal width at a point having a maximum pixel number.

### Claim 8 (depends on 7)

The method of claim 7, wherein the measuring of the maximum horizontal width and the width of the finger comprises:
determining a region of interest by a predefined length specified from an upper end of the horizontal histogram, and determining an average width of the hand region within the region of interest, as the width of the finger.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
generating a distance-transformed image in units of pixels with respect to the image of the hand region of the user, and detecting a pixel having the greatest value in the distant-transformed image, as a marker.

### Claim 10 (depends on 7)

The method of claim 7, wherein the recognizing of whether the finger is folded comprises:
comparing the width of the finger with a width corresponding to a predefined percentage of the maximum horizontal width; and
when the width of the finger is less than the width corresponding to the predefined percentage, recognizing the finger in a spread state, and when the width of the finger is equal to or greater than the width corresponding to the predefined percentage, recognizing the finger in the folded state.

### Claim 11 (depends on 9)

The method of claim 9 further comprising:
recognizing a change in a position of the hand using the marker; and
matching the change in the position of the hand to a pre-set second function and outputs a control signal for implementing the pre-set second function.