# US 8,666,115 B2 · Computer vision gesture based control of a device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,666,115 B2
- Document ID: US8666115B2
- Status: Granted
- Kind: B2
- Application number: 13/499,929
- Filed: 2010-10-13
- Published: 2012-08-09
- Granted: 2014-03-04
- Earliest claimed: 2009-10-13
- Assignees: Pointgrab Ltd.
- Applicants: Haim Perski; Amir Kaplan; Eran Eilat
- Inventors: Haim Perski; Amir Kaplan; Eran Eilat
- Web page: https://patentgrep.com/patent/US8666115B2

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/107: {Static hand or arm}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system and method are provided for controlling a device based on computer vision. Embodiments of the system and method of the invention are based on receiving a sequence of images of a field of view; detecting movement of at least one object in the images; applying a shape recognition algorithm on the at least one moving object; confirming that the object is a user hand by combining information from at least two images of the object; and tracking the object to control the device.

## Claims

### Claim 1

A method for computer vision based hand gesture device control, the method comprising
receiving a sequence of images of a field of view, said images comprising at least one object;
detecting movement of the at least one object in the images, to define at least one moving object;
applying a shape recognition algorithm on the at least one moving object in each of at least two of the sequence of images, to obtain shape information;
performing a first stage determination that the at least one moving object is a user hand by determining that the movement of the at least one object is a movement in a predefined pattern;
performing a second stage determination that the at least one moving object is a user hand by combining the shape information from each of the at least two of the sequence of images to obtain a combined shape information, and determining that the combined shape information corresponds to a hand shape; and
subject to successful first and second stage determination, tracking the user hand to control the device.

### Claim 2 (depends on 1)

The method according to claim 1 comprising tracking the user hand to identify at least one control gesture.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the combined shape information comprises a shape affinity grade associated with said object in each image.

### Claim 4 (depends on 3)

The method according to claim 3 wherein the combined shape information is a function of at least two shape affinity grades from at least two images.

### Claim 5 (depends on 4)

The method according to claims 4 wherein combining information from at least two images of the object comprises calculating an average of the shape affinity grades from at least two images.

### Claim 6 (depends on 5)

The method according to claim 5 comprising comparing the average to a predetermined threshold and deeming the second stage determination positive if the average is above the threshold.

### Claim 7 (depends on 1)

The method according to claim 1 comprising:
determining a shape recognition threshold;
following the determination in the first stage that the movement of the at least one object is a movement in a predefined pattern, lowering the shape recognition threshold, such that the longer the movement is in a predefined pattern, the lower the shape recognition threshold is.

### Claim 8 (depends on 2)

The method according to claim 2 wherein the movement in predefined pattern is an initializing gesture and wherein the initializing gesture and the at least one control gesture are not the same.

### Claim 9 (depends on 8)

The method according to claim 8 wherein the initializing gesture is a wave-like movement of the hand.

### Claim 10 (depends on 2)

The method according to claim 2 wherein the at least one control gesture is to switch a system to cursor control mode.

### Claim 11 (depends on 1)

The method according to claim 1 wherein the combined shape information corresponds to a hand shape only if it is a hand with extended fingers.

### Claim 12 (depends on 1)

The method according to claim 1 comprising:
receiving images of a field of view;
tracking the movement of more than one object in the images; and
comparing the movement of at least one of the objects to a predefined movement pattern; and
if at least one object is moving in a predefined movement pattern, applying shape recognition algorithms to confirm a shape of a hand.

### Claim 13 (depends on 1)

The method according to claim 1 wherein tracking comprises selecting points of interest within one image in the sequence of images, detecting the location of the point in another image and identifying points with similar movement and location parameters.

### Claim 14 (depends on 1)

The method according to claim 1 comprising applying an edge detection algorithm to confirm the object is a user hand.

### Claim 15 (depends on 1)

The method according to claim 1 comprising deriving 3D information from the received sequence of images and combining the 3D information with the information combined from at least two images.

### Claim 16

A system for user-device interaction, the system comprising:
an electronic device, an image sensor and a processor, the processor to:
receive a sequence of images of a field of view;
detect a movement of at least one moving object in the images, to define at least one moving object;
apply a shape recognition algorithm on the at least one moving object in each of at least two of the sequence of images, to obtain shape information;
perform a first stage determination that the at least one moving object is a user hand by determining that the movement of the at least one object is a movement in a predefined pattern;
perform a confirmatory second stage determination that the at least one moving object is a user hand by combining the shape information from each of the at least two of the sequence of images to obtain a combined shape information dataset, and comparing the shape information dataset against a reference hand shape information dataset; and
subject to successful first and second stage determination, track the object to control the device.

### Claim 17 (depends on 16)

The system according to claim 16 wherein the processor is to track the object to identify at least one control gesture and to generate a command to the device based on the at least one control gesture.

### Claim 18 (depends on 16)

The system according to claim 16 wherein the image sensor is a 3D image sensor.

### Claim 19 (depends on 16)

The system according to claim 16 comprising one or more 2D image sensors.
