# US 2013/0159350 A1 · Sensor Fusion Interface for Multiple Sensor Input

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0159350 A1
- Document ID: US20130159350A1
- Status: Published application
- Kind: A1
- Application number: 13/329,777
- Filed: 2011-12-19
- Published: 2013-06-20
- Earliest claimed: 2011-12-19
- Assignees: Microsoft Corporation
- Inventors: Aditya Sankar; William Lawrence Portnoy
- Web page: https://patentgrep.com/patent/US20130159350A1

## CPC

- G06F 16/258: {Data format conversion from or to a database}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06F 16/213: {with details for schema evolution support}
- H04L 67/125: involving control of end-device applications over a network
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

The subject disclosure is directed towards a sensor fusion interface that enables interaction between one or more entities of a physical environment and a computerized device component. A plurality of sensor modules generate multiple sensor input data associated with one or more entities in an environment and store such data in a shared library in accordance with a uniform and common schema. The multiple sensor input data is refined until a certain level of accuracy is achieved. Using the sensor fusion interface, entity state data is extracted from the shared library and exposed to the computerized device component.

## Claims

### Claim 1

In a computing environment, a method performed at least in part on at least one processor, comprising, processing requests received via a sensor fusion interface that enables interaction between one or more entities and an computerized device component, including, accessing a shared library that is coupled to a plurality of sensor modules, wherein the shared library comprises multiple sensor input data associated with one or more entities, extracting entity state data from the shared library, and providing the computerized device component with access to the entity state data.

### Claim 2 (depends on 1)

The method of claim 1, wherein providing the computerized device component with access to the entity state data further comprises communicating an event corresponding to a change in entity pose.

### Claim 3 (depends on 1)

The method of claim 1, wherein providing the computerized device component with access to the entity state data further comprises communicating one or more commands for the computerized device component from the one or more entities.

### Claim 4 (depends on 1)

The method of claim 1, wherein providing the computerized device component with access to the entity state data further comprises generating control data based on the entity state data that corresponds to different points-in-time.

### Claim 5 (depends on 1)

The method of claim 1, wherein extracting the entity state data further comprising determining a position and an orientation of a device being manipulated by the one or more entities.

### Claim 6 (depends on 1)

The method of claim 1, wherein extracting the entity state data further comprises removing unwanted perturbations in the multiple sensor input data.

### Claim 7 (depends on 1)

The method of claim 1, wherein accessing the shared library further comprises configuring the plurality of sensor modules to refine the multiple sensor input data.

### Claim 8 (depends on 1)

The method of claim 1, wherein extracting the entity state data further comprises defining symbolic activity that translates into commands for the computerized device component.

### Claim 9 (depends on 1)

The method of claim 1 further comprising updating the entity state data in response to entity movement.

### Claim 10 (depends on 1)

The method of claim 1, wherein extracting the entity state data further comprises producing an environment model associated with the one or more entities.

### Claim 11 (depends on 1)

The method of claim 1, wherein extracting the entity state data further comprises extracting a first portion of the entity state data associated with a first entity and a second portion of the entity state data associated with a second entity, and correlating the first portion with the second portion to determine entity interactions.

### Claim 12

In a computing environment, a system, comprising, an interaction mechanism configured to provide a sensor fusion interface to an computerized device component, wherein, for each entity in an environment, the sensor fusion interface is configured to generate a profile comprising entity state data, to expose the profile to the computerized device component, and to instruct a common knowledge data component to update the profile in response to entity activity, wherein the common knowledge data component is configured to integrate multiple sensor input data from a plurality of sensor modules according to a schema, and to identify the entity activity based on the multiple sensor input data.

### Claim 13 (depends on 12)

The system of claim 12, wherein the interaction mechanism is further configured to operate a computerized device in the physical environment based on the entity activity.

### Claim 14 (depends on 12)

The system of claim 12, wherein the interaction mechanism is further configured to control a virtual character on a virtual environment that is generated by the computerized device component.

### Claim 15 (depends on 12)

The system of claim 12, wherein the interaction mechanism is further configured to initialize the plurality of sensor modules in response to the computerized device component.

### Claim 16 (depends on 12)

The system of claim 12, wherein the interaction mechanism is further configured to identify a conflict between two or more entities.

### Claim 17 (depends on 12)

The system of claim 12, wherein the interaction mechanism is further configured to expose an event that indicates entity movement out of a range of a device.

### Claim 18 (depends on 12)

The system of claim 12, wherein the plurality of sensor modules include a camera, a microphone, a digitizer, a motion processor, a gaming controller, a speech recognition module, a gesture recognition module, a facial recognition module, a mobile phone or a pointing device, or any combination of a camera, a microphone, a digitizer, a motion processor, a gaming controller, a speech recognition module, a gesture recognition module, a facial recognition module, a mobile phone or a pointing device.

### Claim 19

One or more computer-readable media having computer-executable instructions, which when executed perform steps, comprising:
monitoring a physical environment having an entity;
processing a request for entity state data associated with the entity;
selecting at least one input mode among a plurality of input modes for producing the entity state data based on configuration data from a computerized device component; and
communicating the entity state data to the computerized device component.

### Claim 20 (depends on 19)

The one or more computer-readable media of claim 19 having further computer-executable instructions comprising:
examining entity interactions associated with a plurality of entities; and
communicating events to the computerized device component indicating conflicts corresponding to the entity interactions within the physical environment.
