# US 8,280,732 B2 · System and method for multidimensional gesture analysis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,280,732 B2
- Document ID: US8280732B2
- Status: Granted
- Kind: B2
- Application number: 12/411,514
- Filed: 2009-03-26
- Published: 2010-03-11
- Granted: 2012-10-02
- Earliest claimed: 2008-03-27
- Inventors: Wolfgang Richter; Roland Aubauer
- Web page: https://patentgrep.com/patent/US8280732B2

## CPC

- G10L 15/24: Speech recognition using non-acoustical features
- G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Hand gestures are translated by first detecting the hand gestures with an electronic sensor and converting the detected gestures into respective electrical transfer signals in a frequency band corresponding to that of speech. These transfer signals are inputted in the audible-sound frequency band into a speech-recognition system where they are analyzed.

## Claims

### Claim 1

A method of translating hand gestures, the method comprising the steps of:
detecting the hand gestures with an electronic sensor and converting the detected gestures into respective electrical transfer signals in a frequency band corresponding to that of speech;
inputting the transfer signals in the audible-sound frequency band to a speech-recognition system; and
analyzing the inputted transfer signals with the speech-recognition system.

### Claim 2 (depends on 1)

The method according to claim 1 wherein the input signals are generated by a tone-generator system.

### Claim 3 (depends on 1)

The method according to claim 1 wherein the input signals are generated by a terminal circuit.

### Claim 4 (depends on 1)

The method according to claim 1 wherein the transfer signals are generated in such a way that they map the spatial movement of the gesture.

### Claim 5 (depends on 1)

The method according to claim 1 wherein the transfer signals are composed of three tones, with the composition of the individual tones changing due to the movement of a gesturing hand or a finger according to a movement-mapping rule.

### Claim 6 (depends on 1)

The method according to claim 1 wherein, on movement away from a detection point, the pitch of the transfer signal for this reference point increases.

### Claim 7 (depends on 1)

The method according to claim 1 wherein, on movement toward a detection point, the pitch of the transfer signal for this reference point decreases.

### Claim 8 (depends on 1)

The method according to claim 1 wherein the transfer signals are structured such that they describe the change in distance from the gesturing hand relative to a reference system by means of at least three reference points.

### Claim 9 (depends on 1)

The method according to claim 1 wherein the gestures are sensed by at least three e-field sensors producing respective sensor output signals and only those sensor output signals are recognized as gestures in which at least one sensor signal exceeds a predetermined level.

### Claim 10 (depends on 9)

The method according to claim 9 wherein only those sensor signals that have a minimum alteration dynamic are recognized as gestures.

### Claim 11 (depends on 10)

The method according to claim 10, further comprising the step of
determining mathematical derivatives of the sensor signals.

### Claim 12 (depends on 11)

The method according to claim 11, further comprising the steps of
standardizing sensor signals by subtraction of an average time value or direct signal portion of all sensor signals from the respective individual sensor signals.

### Claim 13 (depends on 9)

The method according to claim 9, further comprising the step of
conducting a coordinate transformation of the sensor signals such that the corresponding gestures are mapped in a spatial gesture reference plane.

### Claim 14 (depends on 13)

The method according to claim 13 wherein the gesture reference plane is established such that it runs essentially parallel to a sensor-electrode surfaces and transverse to a sensor axis such that the corresponding gestures are always mapped at a certain distance from the sensors.

### Claim 15 (depends on 14)

The method according to claim 14 wherein a center of a surface spanned by the gesture lies essentially on the axis of the sensor.

### Claim 16 (depends on 9)

The method according to claim 9 wherein the sensor signals are converted to tone sequences that lie in a speech range of 50 Hz-3400 Hz and the various sensor signals are combined into one single gesture signal to be analyzed by the speech-recognition system.

### Claim 17 (depends on 1)

The method according to claim 1 wherein the sensor signals are captured via a time multiplexer.

### Claim 18 (depends on 1)

The method according to claim 1 wherein the sensor signals are transformed by means of VCO conversion into different frequency bands in the speech frequency range.

### Claim 19 (depends on 1)

The method according to claim 1 wherein the gesture recognition is managed using a dynamic time warping pattern recognizer for recognizing total gestures, analogously to word recognition in speech recognition.

### Claim 20 (depends on 1)

The method according to claim 1 wherein the gesture recognition is managed using a Hidden Markov Model recognizer known from speech recognition.

### Claim 21 (depends on 1)

The method according to claim 1 wherein the gestures to be recognized are trained and stored as reference patterns.

### Claim 22 (depends on 1)

The method according to claim 1 wherein gestures that are recognized during a training as being too different are rejected as unreliable.

### Claim 23 (depends on 1)

The method according to claim 1 wherein similar gestures that are not rejected are combined by averaging the corresponding reference signals.

### Claim 24 (depends on 1)

The method according to claim 1 wherein a gesture training process is supported by visual parameters or visual feedback.

### Claim 25

A system for translating hand gestures, the system comprising:
for a gesture detection unit operable to detect the hand gestures and to convert the detected hand gestures into respective electrical transfer signals in a frequency band corresponding to that of speech;
a speech-recognition system
wherein the electrical transfer signals are fed to the speech-recognition system for analysis of the inputted transfer signals by the speech-recognition system.

### Claim 26 (depends on 25)

The system according to claim 25 wherein the sensor array is integrated into a mouse, mouse pad, headset, keyboard, or monitor.

### Claim 27 (depends on 25)

The system according to claim 25 wherein the sensor array is integrated into a household appliance or a machine for performing input operations.

### Claim 28 (depends on 25)

The system according to claim 25 wherein the sensor array is integrated into a mobile communications device.

### Claim 29 (depends on 25)

The system according to claim 25 wherein the sensor array is integrated into a motor-vehicle navigation system.

### Claim 30 (depends on 25)

The system according to claim 25 wherein the sensor array is integrated into an electronic toy.

### Claim 31

A system for translating hand gestures, the system comprising:
means including a sensor array for detecting the hand gestures and converting the detected gestures into respective electrical transfer signals in a frequency band corresponding to that of speech;
a speech-recognition system; and
means for inputting the transfer signals in the audible-sound frequency band to the speech-recognition system for analysis of the inputted transfer signals by the speech-recognition system.
