# US 2013/0010071 A1 · Methods and systems for mapping pointing device on depth map

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0010071 A1
- Document ID: US20130010071A1
- Status: Published application
- Kind: A1
- Application number: 13/541,684
- Filed: 2012-07-04
- Published: 2013-01-10
- Earliest claimed: 2011-07-04
- Assignees: 3Divi
- Inventors: Andrey Valik; Pavel Zaitsev; Dmitry Morozov; Alexander Argutin
- Web page: https://patentgrep.com/patent/US20130010071A1

## CPC

- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Disclosed are methods for determining and tracking a current location of a handheld pointing device, such as a remote control for an entertainment system, on a depth map generated by a gesture recognition control system. The methods disclosed herein enable identifying a user's hand gesture, and generating corresponding motion data. Further, the handheld pointing device may send motion, such as acceleration or velocity, and/or orientation data such as pitch, roll, and yaw angles. The motion data of user's hand gesture and motion data (orientation data) as received from the handheld pointing device are then compared, and if they correspond to each other, it is determined that the handheld pointing device is in active use by the user as it is held by a particular hand. Accordingly, a location of the handheld pointing device on the depth map can be determined.

## Claims

### Claim 1

A computer-implemented method for determining a position of a handheld pointing device, the method comprising:
determining one or more motions of one or more user hands on a depth map;
generating motion data associated with the one or more motions of the one or more user hands;
acquiring motion data of a handheld pointing device;
determining that the motion of the handheld pointing device is associated with the one or more motions of the one or more user hands;
and determining a current position of the handheld pointing device on the depth map.

### Claim 2 (depends on 1)

The method of claim 1, further comprising generating the depth map by capturing a series of images.

### Claim 3 (depends on 1)

The method of claim 1, further comprising generating a virtual skeleton of the user, the virtual skeleton comprising at least one virtual limb of the user.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
determining coordinates of the one or more user hands, the coordinates to be associated with the virtual skeleton; and
generating motion data of the one or more user hands.

### Claim 5 (depends on 4)

The method of claim 4, wherein determining that the motion of the handheld pointing device is associated with the one or more motions of the one or more user hands comprises comparing motion data of the one or more user hands and motion data of the handheld pointing device.

### Claim 6 (depends on 5)

The method of claim 5, further comprising determining which hand is holding the pointing device.

### Claim 7 (depends on 6)

The method of claim 6, further comprising selectively assigning the coordinates of the hand holding the handheld pointing device to the handheld pointing device.

### Claim 8 (depends on 7)

The method of claim 7, further comprising determining an orientation of the handheld pointing device based upon the coordinates of various virtual skeleton joints related to the hand holding the handheld pointing device.

### Claim 9 (depends on 8)

The method of claim 8, further comprising generating a vector associated with the orientation of the handheld pointing device.

### Claim 10 (depends on 1)

The method of claim 1, further comprising acquiring orientation data of the handheld pointing device, wherein the orientation data comprising one or more of the following: a pitch angle, a roll angle, and a yaw angle.

### Claim 11 (depends on 1)

The method of claim 1, wherein the motion data associated with a motion of the handheld pointing device comprises one or more of acceleration data, velocity data, and inertial data.

### Claim 12 (depends on 1)

The method of claim 1, further comprising determining that the handheld pointing device is in active use by the user.

### Claim 13 (depends on 12)

The method of claim 12, wherein the handheld pointing device is in active use by the user when the handheld pointing device is held and moved by the user and when the user is identified on the depth map.

### Claim 14 (depends on 1)

The method of claim 1, further comprising identifying the user on the depth map.

### Claim 15 (depends on 1)

The method of claim 1, further comprising tracking motions of the user hand.

### Claim 16

A system for determining a position of a handheld pointing device, the system comprising:
a depth sensing device configured to generate a depth map;
a communication module configured to acquire motion data of a handheld pointing device; and
a computing unit communicatively coupled to the depth sensing device, the computing unit configured to:
determine one or more motions of one or more user hands on a depth map;
generate motion data associated with the one or more motions of the one or more user hands;
determine that the motion of the handheld pointing device is associated with the one or more motions of the one or more user hands; and
determine a current position of the handheld pointing device on the depth map.

### Claim 17 (depends on 16)

The system of claim 16, further comprising a video camera communicatively coupled to the computing unit, the video camera being configured to facilitate generation of the depth map.

### Claim 18 (depends on 16)

The system of claim 16, wherein the computing unit is further configured to generate a virtual skeleton of the user, the virtual skeleton comprising at least one virtual hand of the user.

### Claim 19 (depends on 16)

The system of claim 16, wherein the computing unit is further configured to:
determine coordinates of the one or more user hands, the coordinates to be associated with the virtual skeleton;
generate motion data of the one or more user hands; and
compare motion data of the one or more user hands and motion data of the handheld pointing device.

### Claim 20

A processor-readable nontransitory medium having instructions stored thereon, which when executed by one or more processors, cause the one or more processors to:
determine one or more motions of one or more user hands on a depth map;
generate motion data associated with the one or more motions of the one or more user hands;
acquire motion data of a handheld pointing device;
determine that the motion of the handheld pointing device is associated with the one or more motions of the one or more user hands; and
determine a current position of the handheld pointing device on the depth map.
