# US 2010/0063794 A1 · Method and apparatus for translating hand gestures

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2010/0063794 A1
- Document ID: US20100063794A1
- Status: Published application
- Kind: A1
- Application number: 12/506,677
- Filed: 2009-07-21
- Published: 2010-03-11
- Earliest claimed: 2003-08-28
- Assignees: The George Washington University
- Inventors: Jose L. HERNANDEZ-REBOLLAR
- Web page: https://patentgrep.com/patent/US20100063794A1

## CPC

- G09B 21/009: {Teaching or communicating with deaf persons (G09B21/04 takes precedence)}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G10L 13/00: Speech synthesis; Text to speech systems
- G06F 2218/12: Classification; Matching
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A sign language recognition apparatus and method is provided for translating hand gestures into speech or written text. The apparatus includes a number of sensors on the hand, arm and shoulder to measure dynamic and static gestures. The sensors are connected to a microprocessor to search a library of gestures and generate output signals that can then be used to produce a synthesized voice or written text. The apparatus includes sensors such as accelerometers on the fingers and thumb and two accelerometers on the back of the hand to detect motion and orientation of the hand. Sensors are also provided on the back of the hand or wrist to detect forearm rotation, an angle sensor to detect flexing of the elbow, two sensors on the upper arm to detect arm elevation and rotation, and a sensor on the upper arm to detect arm twist. The sensors transmit the data to the microprocessor to determine the shape, position and orientation of the hand relative to the body of the user.

## Claims

### Claim 1

A gesture recognition apparatus for use by a user, comprising:
an input assembly detecting movement and position of each finger with respect to the user's hand, and movement and position of the user's hand with respect to the user's body, said input assembly generating at least one value corresponding to one or more phonemes; and
a word storage device for storing words, each word associated with a stored value corresponding to a sequence of one or more phonemes, receiving said at least one value from said input assembly, matching said at least one received value with the stored value, and producing an output value corresponding to the word associated with the matched stored value.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein said input assembly comprises at least one sensor on each of the user's finger and thumb, for detecting motion and position of each of the user's finger and thumb.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein said sensors each comprise an accelerometer.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein said input assembly comprises two accelerometers on a distal phalanx of the user's thumb.

### Claim 5 (depends on 3)

The apparatus of claim 3, wherein said input assembly further comprises two accelerometers on a back of the user's hand to detect orientation and movement of the user's hand.

### Claim 6 (depends on 2)

The apparatus of claim 2, wherein said input assembly further includes a sensor to determine orientation and movement of the user's arm with respect to the user's body.

### Claim 7 (depends on 5)

The apparatus of claim 5, wherein said input assembly includes a first angular sensor positioned to measure arm rotation with respect to the user's shoulder.

### Claim 8 (depends on 7)

The apparatus of claim 7, wherein said input assembly further comprises a second angular sensor positioned to measure flexing of the user's forearm about the user's elbow.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein said input assembly further comprises an accelerometer positioned to measure elevation and rotation of the user's arm with respect to the user's shoulder.

### Claim 10 (depends on 1)

The apparatus of claim 1, said input assembly comprising:
a glove to be worn by the user, said glove having sensors for detecting dynamic hand movements of each of the user's finger and the user's thumb;
an elbow sensor for detecting and measuring flexing and positioning of the user's forearm about the user's elbow;
an arm sensor for detecting movement and orientation of the user's arm with respect to the user's shoulder; and
a shoulder sensor for detecting movement and position of the user's arm with respect to the user's shoulder.

### Claim 11

A gesture recognition apparatus comprising:
an input assembly comprising a glove having hand sensors for detecting hand movement, an elbow sensor for detecting forearm movement, an arm sensor for detecting arm orientation, and a shoulder sensor for detecting arm rotation,
said input assembly further comprising a frame having an upper arm section and a forearm section, said upper arm section and said forearm section coupled together by a hinge, and
a computer connected to said input assembly and generating an output signal for producing a visual or audible output corresponding to the detected hand movement, forearm movement, or arm rotation.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein said shoulder sensor is coupled
to said upper arm section.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein said shoulder sensor comprises a first sensor for detecting arm twisting.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein said first sensor of said shoulder sensor
comprises a resistive angular sensor.

### Claim 15 (depends on 13)

The apparatus of claim 13, wherein said shoulder sensor further comprises an accelerometer for detecting motion, elevation and position of the user's upper arm with respect to the user's shoulder.

### Claim 16 (depends on 11)

The apparatus of claim 11, wherein said glove includes a finger accelerometer, a thumb accelerometer, a first back hand accelerometer on a back of said glove to detect vertical orientation and movement of said glove, and a second back hand accelerometer on the back of said glove to detect axial orientation and movement of said glove with respect to the forearm sensor.

### Claim 17

A method for translating a user's gesture of a signed language composed of an initial pose, movement, hand location, and a final pose, the method comprising:
determining the initial pose, the hand location and the final pose of the gesture, and a movement of the gesture, the movement occurring between the initial pose and the final pose;
matching the determined initial pose with one or more initial poses of all known gestures, and defining a first list of candidate gestures as those whose pose matches the determined initial pose or, if there is only one match, returning a first most likely gesture corresponding to the match;
matching the determined hand location with one or more hand locations of the first list of candidate gestures, and defining a second list of candidate gestures as those whose hand locations match the determined hand location, or, if there is only one match, returning a second most likely gesture corresponding to the match;
matching the determined movement with one or more movements of the second list of candidate gestures, and defining a third list of candidate gestures as those more than one gestures whose movements match the determined movement, or, if there is only one match, returning a third most likely gesture corresponding to the match;
matching the determined final pose with one or more poses of the third list of candidate gestures, and defining a fourth list of candidate gestures as those more than one gestures whose final pose matches the determined final pose, or, if there is only one match, returning a fourth most likely gesture corresponding
to the match;
matching the determined final hand location of the gesture with a hand location of the fourth list of candidate signs, and returning a fifth most likely gesture corresponding to the match; and
converting the first, second, third, fourth or fifth gesture into a stream of ASCII characters to be displayed as text and/ or to a voice synthesizer to be reproduced as speech.

### Claim 18 (depends on 17)

The method for translating a sign of claim 17, wherein said method transmits said most likely gesture as a stream of ASCII characters to be displayed as text or synthesized as voice in any language.

### Claim 19 (depends on 17)

The method for translating signs of claim 17, wherein only the first step is
used to translate postures as letters of finger-spelled words.

### Claim 20 (depends on 17)

The method of claim 17, wherein hand posture is the combination of hand orientation and hand shape, and hand pose is the combination of hand posture and hand location with respect to the user's body, movement is the trajectory defined by the hand when going from one pose to the next, static phoneme is a pose, dynamic phoneme is movement, and a gesture is a sequence of static and dynamic phonemes.

### Claim 21 (depends on 15)

The apparatus of claim 15, wherein said elbow sensor is coupled to said arm section.

### Claim 22 (depends on 21)

The apparatus of claim 21, wherein said elbow sensor further comprises a resistive angular sensor.
