# US 2009/0268945 A1 · Architecture for controlling a computer using hand gestures

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2009/0268945 A1
- Document ID: US20090268945A1
- Status: Published application
- Kind: A1
- Application number: 12/494,303
- Filed: 2009-06-30
- Published: 2009-10-29
- Earliest claimed: 2003-03-25
- Assignees: Microsoft Corporation
- Inventors: Andrew D. Wilson; Nuria M. Oliver
- Web page: https://patentgrep.com/patent/US20090268945A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06F 3/023: Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
- G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06F 13/105: {where the program performs an input/output emulation function}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Architecture for implementing a perceptual user interface. The architecture comprises alternative modalities for controlling computer application programs and manipulating on-screen objects through hand gestures or a combination of hand gestures and verbal commands. The perceptual user interface system includes a tracking component that detects object characteristics of at least one of a plurality of objects within a scene, and tracks the respective object. Detection of object characteristics is based at least in part upon image comparison of a plurality of images relative to a course mapping of the images. A seeding component iteratively seeds the tracking component with object hypotheses based upon the presence of the object characteristics and the image comparison. A filtering component selectively removes the tracked object from the object hypotheses and/or at least one object hypothesis from the set of object hypotheses based upon predetermined removal criteria.

## Claims

### Claim 1

A method of determining a command, comprising:
capturing an image of an object with a camera;
determining a gesture based at least partly upon the image;
detecting an audio input; and
determining, at one or more processors, the command based at least partly upon the gesture and the audio input.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining a depth of the object; and
determining the command based at least partly upon the depth of the object.

### Claim 3 (depends on 2)

The method of claim 2, wherein determining the depth of the object includes capturing a second image of the object with a second camera.

### Claim 4 (depends on 1)

The method of claim 1, wherein the camera is a video camera.

### Claim 5 (depends on 1)

The method of claim 1, wherein the camera detects visible light.

### Claim 6 (depends on 1)

The method of claim 1, wherein determining the gesture includes capturing a second image of the object with the camera and comparing the image with the second image.

### Claim 7

A computer-readable medium having instruction that cause a processor to execute steps, the steps comprising:
capturing an image of an object with a camera;
determining a gesture based at least partly upon the image;
detecting an audio input; and
determining, at one or more processors, a command based at least partly upon the gesture and the audio input.

### Claim 8 (depends on 7)

The computer-readable medium of claim 7, the steps further comprising:
determining a depth of the object; and
determining the command based at least partly upon the depth of the object.

### Claim 9 (depends on 8)

The computer-readable medium of claim 8, wherein determining the depth of the object includes capturing a second image of the object with a second camera.

### Claim 10 (depends on 7)

The computer-readable medium of claim 7, wherein the camera is a video camera.

### Claim 11 (depends on 7)

The computer-readable medium of claim 7, wherein the camera detects visible light.

### Claim 12 (depends on 7)

The computer-readable medium of claim 7, wherein determining the gesture includes capturing a second image of the object with the camera and comparing the image with the second image.

### Claim 13

A command determining system, comprising:
a camera configured to capture an image of an object;
a first determiner configured to determine a gesture based at least partly upon the image;
an audio detection unit configured to detect an audio input; and
a second determiner configured to determine the command based at least partly upon the gesture and the audio input.

### Claim 14 (depends on 13)

The command determining system of claim 13, further comprising:
a third determiner configured to determine a depth of the object, wherein the second determiner is further configured to determine the command based at least partly upon the depth of the object.

### Claim 15 (depends on 14)

The command determining system of claim 14, wherein determining the depth of the object includes capturing a second image of the object with a second camera.

### Claim 16 (depends on 13)

The command determining system of claim 13, wherein the camera is a video camera.

### Claim 17 (depends on 13)

The command determining system of claim 13, wherein the camera detects visible light.

### Claim 18 (depends on 13)

The command determining system of claim 13, wherein determining the gesture includes capturing a second image of the object with the camera and comparing the image with the second image.
