# US 2008/0313575 A1 · System and process for controlling electronic components in a ubiquitous computing environment using multimodal integration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2008/0313575 A1
- Document ID: US20080313575A1
- Status: Published application
- Kind: A1
- Application number: 12/191,883
- Filed: 2008-08-14
- Published: 2008-12-18
- Earliest claimed: 2002-02-07
- Assignees: Microsoft Corporation
- Inventors: ANDREW WILSON
- Web page: https://patentgrep.com/patent/US20080313575A1

## CPC

- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G08C 17/00: Arrangements for transmitting signals characterised by the use of a wireless electrical link
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G08C 2201/31: Voice input
- G08C 2201/32: Remote control based on movements, attitude of remote control device
- G08C 2201/41: Remote control of gateways
- G08C 2201/50: Receiving or transmitting feedback, e.g. replies, status updates, acknowledgements, from the controlled devices
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G08C: TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS (fluid pressure transmitting systems F15B; mechanical means for transferring the output of a sensing member into a different variable G01D5/00; mechanical control systems G05G)

## Abstract

The present invention is directed toward a system and process that controls a group of networked electronic components using a multimodal integration scheme in which inputs from a speech recognition subsystem, gesture recognition subsystem employing a wireless pointing device and pointing analysis subsystem also employing the pointing device, are combined to determine what component a user wants to control and what control action is desired. In this multimodal integration scheme, the desired action concerning an electronic component is decomposed into a command and a referent pair. The referent can be identified using the pointing device to identify the component by pointing at the component or an object associated with it, by using speech recognition, or both. The command may be specified by pressing a button on the pointing device, by a gesture performed with the pointing device, by a speech recognition event, or by any combination of these inputs.

## Claims

### Claim 1

A computer-implemented process for controlling a user-selected electronic component within an environment using a pointing device, comprising:
a waiting step for waiting for an orientation message to be received;
a receiving step for receiving orientation messages transmitted by the pointing device, said orientation messages comprising at least one orientation sensor reading generated by at least one orientation sensor of the pointing device;
a switch state determining step for determining whether a switch state indicates that a pointing device switch has been activated by the user to indicate that a gesture is being performed, said switch state determining step being executed whenever an orientation message is received;
a repeating step for repeating the waiting step, the receiving step, and the switch state determining step whenever it is determined that the switch state does not indicate that the pointing device switch is activated;
a recording step for recording a prescribed one or ones of the pointing device sensor outputs taken from the orientation message received, in accordance with a determination that the switch state indicates that the pointing device switch is activated;
a determining step for determining whether a threshold if just one, or all the thresholds if more than one, of a gesture threshold definition under consideration are exceeded by the recorded sensor output associated with the same sensor output as the threshold for the gesture threshold definition assigned to the selected electronic component;
a designating step for designating that the user has performed the gesture associated with the gesture threshold definition whenever it is determined that the threshold if just one, or all the thresholds if more than one, of the gesture threshold definition are exceeded by the recorded sensor output associated with the same sensor output; and
an implementing step for implementing the control action represented by the gesture that the user was designated to have performed.

### Claim 2 (depends on 1)

The process of claim 1, wherein prior to performing the determining step, performing an establishing step for establishing, for the gesture representing the control action for the selected electronic component, the gesture threshold definition, said gesture threshold definitions comprising either (i) a prescribed threshold applicable to a particular single sensor output or (ii) a set of thresholds applicable to a particular group of sensor outputs, which are indicative of the pointing device being moved in a particular direction from a starting point wherein the pointing device is pointed at the object in the environment corresponding to or associated with the selected electronic component.

### Claim 3 (depends on 1)

The process of claim 1, wherein the at least one sensor of the pointing device comprises at least one accelerometer.

### Claim 4 (depends on 1)

The process of claim 1, wherein the at least one sensor of the pointing device comprises at least one gyroscope.

### Claim 5 (depends on 1)

The process of claim 1, wherein the at least one sensor of the pointing device comprises at least one accelerometer and at least one gyroscope.

### Claim 6 (depends on 1)

The process of claim 1, wherein the at least one orientation sensor reading comprises an acceleration value.

### Claim 7 (depends on 1)

The process of claim 1, wherein the environment includes more than one electronic component.

### Claim 8 (depends on 1)

The process of claim 1, wherein the prescribed one or ones of the pointing device sensor outputs taken from the orientation message received are recorded only when it is determined that the switch state indicates that the pointing device switch is activated.
