# US 2015/0105159 A1 · Boolean/float controller and gesture recognition system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0105159 A1
- Document ID: US20150105159A1
- Status: Published application
- Kind: A1
- Application number: 14/053,417
- Filed: 2013-10-14
- Published: 2015-04-16
- Earliest claimed: 2013-10-14
- Assignees: Microsoft Corporation
- Applicants: Microsoft Corporation
- Inventors: Mike Palotas
- Web page: https://patentgrep.com/patent/US20150105159A1

## CPC

- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/215: comprising means for detecting acoustic signals, e.g. using a microphone
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0227: {Cooperation and interconnection of the input arrangement with other functional units of a computer (G06F3/023 - G06F3/037 take precedence)}
- G06F 3/033: Pointing devices displaced or positioned by the user {, e.g. mice, trackballs, pens or joysticks}; Accessories therefor (digitisers characterised by the transducing means G06F3/041)
- G06F 9/541: {via adapters, e.g. between incompatible applications}
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system and method are disclosed for providing input to a software application such as a gaming application. The inputs may be received from one or both of a hand-held controller and a NUI system accepting gestural inputs. In an example, a gesture and controller API receives skeletal pose data from the NUI system and converts that data to Boolean or floating point data. In this way, data from both the controller and the NUI system may be exposed to the software application as easily consumable data in the same format.

## Claims

### Claim 1

A content presentation and interaction system, comprising:
a controller providing tactile control input and communicatively coupled to the content presentation and interaction system, the controller including a plurality of tactile input mechanisms and providing a first set of control inputs to the content presentation and interaction system;
a target recognition and tracking system communicatively coupled to the content presentation and interaction system, the target recognition and tracking system detecting pose information including skeletal data having absolute and relative positions of body joints; and
a gesture and controller application programming interface for receiving pose information from the target recognition and tracking system and processing the pose information to identify gestural input in a format compatible with a format of the tactile control input.

### Claim 2 (depends on 1)

The content presentation and interaction system of claim 1, wherein the gesture and controller application programming interface receives skeletal data and processes the skeletal data to identify gestural input as a Boolean value.

### Claim 3 (depends on 2)

The content presentation and interaction system of claim 2, wherein the skeletal data processed to identify gestural input as a Boolean value includes a first body part contacting, or not contacting, a second body part.

### Claim 4 (depends on 1)

The content presentation and interaction system of claim 1, wherein the gesture and controller application programming interface receives skeletal data and processes the skeletal data to identify gestural input as a float value.

### Claim 5 (depends on 4)

The content presentation and interaction system of claim 4, wherein the skeletal data processed to identify gestural input as a float value includes one or more body parts leaning between a neutral position and a maximum extent.

### Claim 6 (depends on 4)

The content presentation and interaction system of claim 4, wherein the skeletal data processed to identify gestural input as a float value includes one of the following:
a. measuring a degree of body tilt,
b. measuring a degree of body twist,
c. measuring a degree of bend from the waste,
d. measuring a degree to which an arm or leg is raised,
e. measuring speech, and
f. head rotation about one or more of three perpendicular axes.

### Claim 7 (depends on 1)

The content presentation and interaction system of claim 1, wherein the gesture and controller application programming interface receives skeletal data and processes the skeletal data to identify gestural input as an action gesture.

### Claim 8 (depends on 7)

The content presentation and interaction system of claim 7, wherein the skeletal data processed to identify gestural input as an action gesture includes measuring positions of one or more body parts over multiple instances of time.

### Claim 9 (depends on 7)

The content presentation and interaction system of claim 7, wherein the skeletal data processed to identify gestural input as an action gesture includes detecting vector movement of a body part.

### Claim 10 (depends on 1)

The content presentation and interaction system of claim 1, wherein the gesture and controller application programming interface includes a library of stored rules defining body positions and/or movements that qualify as predefined gestures.

### Claim 11

A method of providing input to a content presentation and interaction system, the method comprising:
a. receiving tactile controller input from a hand-held controller, the input including Boolean and float values;
b. receiving gestural input from a target recognition and tracking system, the gestural input including skeletal data having absolute and relative positions of body joints;
c. receiving a Boolean or float value based on identification of a predefined gesture from the skeletal data; and
d. providing the Boolean and/or float values from said steps a and/or c as inputs to a software application running on the content presentation and interaction system.

### Claim 12 (depends on 11)

The method of claim 11, said step a of receiving tactile controller input from a hand-held controller comprises receiving tactile controller input from one or more of a thumbstick, a directional pad, a buttons, or a trigger on the controller.

### Claim 13 (depends on 11)

The method of claim 11, said step c comprises analyzing the skeletal data to discern touching of a hand to a head, and assigning a Boolean value of a 1 or a 0 when touching of a hand to a head is detected.

### Claim 14 (depends on 11)

The method of claim 11, said step c comprises analyzing the skeletal data to discern a body position within a range of body positions, and assigning a float value indicative of the body position within the range of body positions.

### Claim 15 (depends on 11)

The method of claim 11, further comprising receiving speech input from the target recognition and tracking system, the speech input analyzed and assigned one of a Boolean value or float value depending on the received speech.

### Claim 16 (depends on 11)

The method of claim 11, said step c. comprises the step of comparing skeletal data to a library of stored rules defining the predefined gestures.

### Claim 17

A computer readable medium for programming a processor to perform a method of providing input to a gaming console, the method comprising:
a. receiving tactile controller input from a hand-held controller, the input including Boolean and float values;
b. receiving gestural input from a target recognition and tracking system, the gestural input including skeletal data having absolute and relative positions of body joints;
c. receiving speech input via a microphone of the target recognition and tracking system;
d. analyzing via a processor of the content presentation and interaction system the speech input and skeletal data to discern predefined gestures defined by the speech input, or positions and/or movements of body parts;
e. assigning a Boolean or float value to a predefined gesture discerned in said step d; and
f. providing the Boolean and/or float values from said steps a and/or e as inputs to a gaming application running on the gaming console.

### Claim 18 (depends on 17)

The computer readable medium of claim 17, said step e comprising the step of assigning a Boolean value to a predefined gesture defined by the speech input, the Boolean value being a 1 or a 0 depending on whether or not speech input was received or not received.

### Claim 19 (depends on 17)

The computer readable medium of claim 17, wherein at least two of the tactile control input, gestural input and speech input are received at the same time.

### Claim 20 (depends on 17)

The computer readable medium of claim 17, further comprising the step of analyzing the skeletal data to discern an action gesture which is performed over time, the action gesture used as an input to the gaming application running on the gaming console.
