# US 2017/0285757 A1 · Control system using aesthetically guided gesture recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0285757 A1
- Document ID: US20170285757A1
- Status: Published application
- Kind: A1
- Application number: 15/086,561
- Filed: 2016-03-31
- Published: 2017-10-05
- Earliest claimed: 2016-03-31
- Assignees: Disney Enterprises, Inc.
- Applicants: Disney Enterprises, Inc.
- Inventors: JAMES R. ROBERTSON; RAYMOND J. SCANLON; MICHAEL R. HONECK; ANGELA JAY; CORY J. ROUSE; HOLGER IRMLER
- Web page: https://patentgrep.com/patent/US20170285757A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves}
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06F 2218/08: Feature extraction
- G06F 2218/12: Classification; Matching
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for facilitating and enhancing computer-based authoring of gesture definitions that are useful in controlling a walk-around character and other systems using gesture controls. The method includes, during performance of a gesture by a performer, collecting sets of raw sensor data each corresponding to differing parameters of the performance of the gesture. The method includes displaying a graphical user interface with a graphical plot of each of the sets of raw sensor data. The method includes receiving user input identifying which of the parameters to include in a gesture definition. The method includes, for the graphical plots associated with the chosen parameters receiving user input defining a starting position, an ending position, a maximum value, and a minimum value. The gesture may involve movement of a performer's arms, legs, hands, head, eyes, and so on in a particular manner.

## Claims

### Claim 1

A method for authoring gesture definitions for use in gesture-based control systems, comprising:
during performance of a gesture by a performer, collecting a plurality of sets of raw sensor data each corresponding to differing parameters of the performance of the gesture;
displaying a graphical user interface (GUI) on a display device, wherein the GUI includes a graphical plot of each of the sets of raw sensor data;
receiving and storing in memory user input identifying a subset of the parameters to include in a gesture definition; and
for each of the graphical plots for the subset of the parameters in the gesture definition, receiving and storing, in the memory in the gesture definition, user input defining at least one of a starting position, an ending position, a maximum value, and a minimum value.

### Claim 2 (depends on 1)

The method of claim 1, wherein the user input defining the starting position, the ending position, the maximum value, and the minimum value includes positioning a set of bounding boxes in each of the graphical plots of the sets of the raw sensor data for the subset of the parameters and sizing each of the bounding boxes.

### Claim 3 (depends on 1)

The method of claim 1, further including receiving user input defining for one of the graphical plots associated with the subset of the parameters used in the gesture definition a must pass window defining an area a plot of sensor data must pass for gesture recognition.

### Claim 4 (depends on 1)

The method of claim 1, further including receiving user input defining for one of the graphical plots associated with the subset of the parameters used in the gesture definition a must not pass window defining an area a plot of sensor data must not pass for gesture recognition.

### Claim 5 (depends on 1)

The method of claim 1, further including receiving additional user input modifying the stored definition of at least one of the starting position, the ending position, the maximum value, and the minimum value, whereby the additional user input adjusts the gesture definition based on aesthetic or other guidelines.

### Claim 6 (depends on 1)

The method of claim 1, further comprising performing training of an additional performer comprising:
receiving an additional sets of raw sensor data each corresponding to the differing parameters of the performance of the gesture by the additional performer; and
comparing the additional sets of the raw sensor data with each of the graphical plots for the subset of the parameters in the gesture definition to verify at least one of the starting position, the ending position, the maximum value, and the minimum value are matched.

### Claim 7 (depends on 6)

The method of claim 6, wherein the performing training of an additional performer further comprises providing visual or audio feedback to the additional performer indicating whether the performance of the gesture satisfied the gesture definition.

### Claim 8 (depends on 7)

The method of claim 7, wherein the providing of the visual or audio feedback comprises a plotting of the additional sets of raw sensor data relative to the graphical plots of each of the sets of raw sensor data used to create the gesture definition.

### Claim 9 (depends on 1)

The method of claim 1, further comprising defining a length of time for performing the gesture and wherein the length of time is stored with the gesture definition.

### Claim 10 (depends on 1)

The method of claim 1, wherein the sets of raw sensor data include data in a number of channels ranging from 1 to 13 channels, whereby the differing parameters of the performance of the gesture number from 1 to 13.

### Claim 11

A system for authoring gestures for use by a gesture-based controller, comprising:
a computer with a processor managing a display and a memory device;
a gesture monitoring assembly including a set of sensors sampling data during performance of a gesture by a performer; and
a gesture authoring tool run by the processor to:
during the performance of the gesture by the performer, collect sets of sensor data from each of the sensors;
display a graphical user interface (GUI) on a display device, wherein the GUI includes a graphical plot of each of the sets of sensor data;
receiving and storing in the memory device user input identifying a subset of the sensors to include in a gesture definition; and
for each of the graphical plots for the subset of the sensors included in the gesture definition, receiving and storing, in the memory device as part of the gesture definition, user input defining at least one of a starting position, an ending position, a maximum value, and a minimum value.

### Claim 12 (depends on 11)

The system of claim 11, wherein the user input defining the starting position, the ending position, the maximum value, and the minimum value includes positioning a set of bounding boxes in each of the graphical plots of the sets of the raw sensor data for the subset of the parameters and sizing each of the bounding boxes.

### Claim 13 (depends on 11)

The system of claim 11, wherein the gesture authoring tool further acts to receive user input defining for one of the graphical plots associated with the subset of the parameters used in the gesture definition at least one of a must pass window and a must not pass window defining, respectively, an area a plot of sensor data must pass for gesture recognition and an area of a plot of sensor data must not pass for gesture recognition.

### Claim 14 (depends on 11)

The system of claim 11, wherein the gesture authoring tool receives additional user input modifying the stored definition of at least one of the starting position, the ending position, the maximum value, and the minimum value, whereby the additional user input adjusts the gesture definition based on aesthetic or other guidelines.

### Claim 15 (depends on 11)

The system of claim 11, wherein the gesture authoring tool performs regression analysis including comparing the gesture definition with at least one previously generated gesture definition to determine whether the gesturing monitoring assembly can accurately distinguish performances of a corresponding pair of gestures and, when not, to flag the gesture definition and the at least one previously generated gesture definition for modification.

### Claim 16

A gesture authoring method, comprising:
collecting a set of data from a sensor during a data collection window;
receiving first user input defining a gesture performance time period with a start time and an end time for performance of gesture by a performer during the collecting;
creating a first gesture definition including the set of data from the sensor during the gesture performance time period;
receiving second user input marking in the first gesture definition at least one of a start position, an end position, a maximum value, and a minimum value; and
creating a second gesture definition comprising a modified version of the first gesture definition including the marking from the second user input.

### Claim 17 (depends on 16)

The method of claim 16, further comprising, prior to the receiving of the second user input, generating a graphical user input including a graphical plot of the set of data from the sensor and displaying the graphical user input on a monitor of a computer.

### Claim 18 (depends on 17)

The method of claim 17, wherein the receiving of second user input providing the marking includes displaying bounding boxes for each of the start position, the end position, the maximum value and the minimum value and receiving user input placing each of the bounding boxes relative to the graphical plot and sizing each of the bounding boxes.

### Claim 19 (depends on 17)

The method of claim 17, further comprising receiving third user input modifying the marking from the second user input and, in response, creating a third gesture definition comprising a modified version of the second gesture definition.

### Claim 20 (depends on 16)

The method of claim 16, further comprising using the second gesture definition for training a performer including collecting additional data from the sensor and comparing the additional data to the graphical plot to verify at least the marked subset of the start position, the end position, the maximum value and the minimum value are matched.