# US 2012/0254810 A1 · Combined Activation for Natural User Interface Systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0254810 A1
- Document ID: US20120254810A1
- Status: Published application
- Kind: A1
- Application number: 13/077,368
- Filed: 2011-03-31
- Published: 2012-10-04
- Earliest claimed: 2011-03-31
- Assignees: Microsoft Corporation
- Inventors: Larry Paul Heck; Madhusudan Chinthakunta; David Mitby; Lisa Stifelman
- Web page: https://patentgrep.com/patent/US20120254810A1

## CPC

- G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0487: using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A user interaction activation may be provided. A plurality of signals received from a user may be evaluated to determine whether the plurality of signals are associated with a visual display. If so, the plurality of signals may be translated into an agent action and a context associated with the visual display may be retrieved. The agent action may be performed according to the retrieved context and a result associated with the performed agent action may be displayed to the user.

## Claims

### Claim 1

A method for providing a user-interaction activation, the method comprising:
receiving a plurality of signals from a user;
determining whether the plurality of signals are associated with a visual display; and
in response to determining that the plurality of signals are associated with the visual display:
translating the plurality of signals into an agent action,
retrieving a context associated with the visual display,
performing the agent action according to the retrieved context, and
displaying a result associated with the performed agent action to the user.

### Claim 2 (depends on 1)

The method of claim 1, wherein the visual display comprises at least one of the following: a video and a still image.

### Claim 3 (depends on 1)

The method of claim 1, wherein the visual display is associated with at least one of the following: a television show, a movie, and a game.

### Claim 4 (depends on 1)

The method of claim 1, wherein the plurality of signals comprises at least one of the following: a keyword and an activation gesture.

### Claim 5 (depends on 1)

The method of claim 1, wherein performing the agent action according to the retrieved context comprises narrowing a plurality of results to a subset of the plurality of results according to the retrieved context.

### Claim 6 (depends on 5)

The method of claim 5, further comprising displaying the subset of the plurality of results to the user.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
receiving a gesture from the user, wherein the gesture comprises at least one of the plurality of signals;
updating the retrieved context according to the gesture; and
performing the agent action according to the updated context.

### Claim 8 (depends on 7)

The method of claim 7, wherein the gesture comprises a pointing received from the user.

### Claim 9 (depends on 8)

The method of claim 8, wherein updating the retrieved context according to the pointing gesture comprises identifying an element of the visual display indicated by the pointing gesture.

### Claim 10

A computer-readable medium which stores a set of instructions which when executed performs a method for providing a user-interaction activation, the method executed by the set of instructions comprising:
receiving a request from a user, wherein the request comprises a speech signal;
retrieving a context associated with a visual display;
identifying a gesture received from the user;
performing an action associated with the request according to the retrieved context and the identified gesture; and
providing a result associated with the performed action to the user.

### Claim 11 (depends on 10)

The computer-readable medium of claim 10, wherein the natural language phrase comprises a spoken phrase.

### Claim 12 (depends on 10)

The computer-readable medium of claim 10, wherein the visual display comprises at least one of the following a still image displayed to the user and a video image displayed to the user.

### Claim 13 (depends on 10)

The computer-readable medium of claim 10, wherein the gesture and the speech signal are received from the user contemporaneously.

### Claim 14 (depends on 10)

The computer-readable medium of claim 10, wherein the visual display comprises an image captured by a recording device associated with the user.

### Claim 15 (depends on 10)

The computer-readable medium of claim 10, wherein the gesture comprises an activation gesture.

### Claim 16 (depends on 10)

The computer-readable medium of claim 10, wherein the gesture is identified via a plurality of cameras.

### Claim 17 (depends on 10)

The computer-readable medium of claim 10, wherein performing the action associated with the request according to the identified gesture comprises selecting a subset of the visual display to be associated with the request.

### Claim 18 (depends on 10)

The computer-readable medium of claim 10, wherein the context associated with the visual display is retrieved from a plurality of metadata associated with the visual display.

### Claim 19 (depends on 10)

The computer-readable medium of claim 10, wherein the context associated with the visual display is retrieved from a database.

### Claim 20

A system for providing a user-interaction activation, the system comprising:
at least one camera;
a memory storage; and
a processing unit coupled to the memory storage and the camera, wherein the processing unit is operative to:
receive a plurality of contemporaneous signals from a user, wherein at least one first signal comprises a speech signal received via at least one microphone and at least one second signal comprises a gesture received via the at least one camera,
determine whether the plurality of signals are directed to the system, and
in response to determining that the plurality of signals are directed to the system:
retrieve a context associated with a visual display;
translate the plurality of signals into at least one agent action associated with the visual display, wherein the gesture comprises a pointing gesture operative to select a subset of the visual display;
perform the agent action according to the retrieved context and the gesture; and
display a result associated with the performed agent action to the user.
