# US 2021/0174034 A1 · Computer vision based sign language interpreter

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174034 A1
- Document ID: US20210174034A1
- Status: Published application
- Kind: A1
- Application number: 16/762,302
- Filed: 2018-11-08
- Published: 2021-06-10
- Earliest claimed: 2017-11-08
- Assignees: Signall Technologies Zrt
- Applicants: Signall Technologies Zrt
- Inventors: David RETEK; David PALHAZI; Marton KAJTAR; Attila ALVAREZ; Peter POSCSI; Andras NEMETH; Matyas TROSZTEL; Zsolt ROBOTKA; Janos ROVNYAI
- Web page: https://patentgrep.com/patent/US20210174034A1

## CPC

- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 40/205: Parsing
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G09B 21/009: {Teaching or communicating with deaf persons (G09B21/04 takes precedence)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

A system and method for translating sign language utterances into a target language, including: receiving motion capture data; producing phonemes/sign fragments from the received motion capture data; producing a plurality of sign sequences from the phonemes/sign fragments; parsing these sign sequences to produce grammatically parsed sign utterances; translating the grammatically parsed sign utterances into grammatical representations in the target language; and generating output utterances in the target language based upon the grammatical representations.

## Claims

### Claim 1

A method of translating sign language utterances into a target language, comprising:
receiving motion capture data;
producing phonemes/sign fragments from the received motion capture data;
producing a plurality of sign sequences from the phonemes/sign fragments;
parsing these sign sequences to produce grammatically parsed sign utterances;
translating the grammatically parsed sign utterances into grammatical representations in the target language; and
generating output utterances in the target language based upon the grammatical representations.

### Claim 2 (depends on 1)

The method of claim 1, wherein confidence values are produced for each generated output utterance.

### Claim 3 (depends on 1)

The method of claim 1, wherein producing phonemes/sign fragments from motion capture data includes producing a confidence value for each produced phoneme/sign fragment.

### Claim 4 (depends on 1)

The method of claim 1, wherein producing phonemes/sign fragments includes producing a plurality of segments as time intervals matching the phonemes/sign fragments, where these intervals of the segments may overlap.

### Claim 5 (depends on 4)

The method of claim 4, wherein producing phonemes/sign fragments and their intervals includes determining of a set of possible succeeding phoneme/sign fragment for each phoneme/sign fragment.

### Claim 6 (depends on 1)

The method of claim 1, wherein producing sign sequences from the phonemes/sign fragments includes matching potential paths in a graph of phonemes/sign fragments to each sign in each sign sequence.

### Claim 7 (depends on 1)

The method of claim 1, wherein producing grammatically parsed sign utterances includes producing a grammatical context and using the grammatical context of previous utterances.

### Claim 8 (depends on 1)

The method of claim 1, wherein producing grammatically parsed sign utterances includes producing a confidence value based on the confidences of the signs, the confidence of the parsing and confidence of the parse matching a grammatical context for each parse of each sign sequence.

### Claim 9 (depends on 1)

The method of claim 1, further comprising
generating a plurality of output utterances in the target language based upon the plurality of sign sequences;
displaying the plurality output utterances to a user; and
receiving an indication from the user selecting one of the plurality of displayed output utterances as the correct translation.

### Claim 10 (depends on 1)

The method of claim 1, further comprising detecting the end of a sign language utterance before parsing the sign sequence to produce a grammatically parsed sign utterance.

### Claim 11 (depends on 1)

The method of claim 1, wherein producing phonemes/sign fragments from the extracted features further includes extracting features from the motion capture data.

### Claim 12 (depends on 1)

The method of claim 1, wherein the motion capture data includes data captured using marked gloves used by a user to produce the sign language utterance

### Claim 13 (depends on 1)

The method of claim 1, wherein user specific parameter data is used for one of:
producing phonemes/sign fragments;
producing a plurality of sign sequences;
parsing these sign sequences; and
translating the grammatically parsed sign utterances.

### Claim 14 (depends on 1)

The method of claim 1, further comprising:
detecting that the user is using fingerspelling, after producing phonemes/sign fragments;
translating the fingerspelling phoneme/sign fragments to letters in the target language;
generating an output to the user showing translated letters to the user; and
receiving an input from the user indicating the correctness of the translated letters.

### Claim 15

A system configured to translate sign language utterances into a target language, comprising:
an input interface configured to receive motion capture data;
a memory; and
a processor in communication with the input interface and the memory, the processor being configured to:
produce phonemes/sign fragments from the extracted features;
produce a sign sequence from the phonemes/sign fragments;
parse the sign sequence to produce a grammatically parsed sign utterance;
translate the grammatically parsed sign utterance into a grammatical representation in the target language; and
generate an output utterance in the target language based upon the grammatical representation.

### Claim 16 (depends on 15)

The system of claim 15, wherein confidence values are produced for each generated output utterance.

### Claim 17 (depends on 15)

The system of claim 15, wherein producing phonemes/sign fragments from motion capture data includes producing a confidence value for each produced phoneme/sign fragment.

### Claim 18 (depends on 15)

The system of claim 15, wherein producing phonemes/sign fragments includes producing a plurality of segments as time intervals matching the phonemes/sign fragments, where these intervals of the segments may overlap.

### Claim 19 (depends on 18)

The system of claim 18, wherein producing phonemes/sign fragments and their intervals includes determining of a set of possible succeeding phoneme/sign fragment for each phoneme/sign fragment.

### Claim 20 (depends on 15)

The system of claim 15, wherein producing sign sequences from the phonemes/sign fragments includes matching potential paths in a graph of phonemes/sign fragments to each sign in each sign sequence.

### Claim 21 (depends on 15)

The system of claim 15, wherein producing grammatically parsed sign utterances includes producing a grammatical context and using the grammatical context of previous utterances.

### Claim 22 (depends on 15)

The system of claim 15, wherein producing grammatically parsed sign utterances includes producing a confidence value based on the confidences of the signs, the confidence of the parsing and confidence of the parse matching a grammatical context for each parse of each sign sequence.

### Claim 23 (depends on 15)

The system of claim 15, wherein the processor is further configured to:
generate a plurality of output utterances in the target language based upon the plurality of sign sequences;
display the plurality output utterances to a user; and
receive an indication from the user selecting one of the plurality of displayed output utterances as the correct translation.

### Claim 24 (depends on 15)

The system of claim 15, wherein the processor is further configured to detect the end of a sign language utterance before parsing the sign sequence to produce a grammatically parsed sign utterance.

### Claim 25 (depends on 15)

The system of claim 15, wherein producing phonemes/sign fragments from the extracted features further includes extracting features from the motion capture data.

### Claim 26 (depends on 15)

The system of claim 15, wherein the motion capture data includes data captured using marked gloves used by a user to produce the sign language utterance

### Claim 27 (depends on 15)

The system of claim 15, wherein user specific parameter data is used for one of:
producing phonemes/sign fragments;
producing a plurality of sign sequences;
parsing these sign sequences; and
translating the grammatically parsed sign utterances.

### Claim 28 (depends on 15)

The system of claim 15, wherein the processor is further configured to:
detect that the user is using fingerspelling, after producing phonemes/sign fragments;
translate the fingerspelling phoneme/sign fragments to letters in the target language;
generate an output to the user showing translated letters to the user; and
receive an input from the user indicating the correctness of the translated letters.

### Claim 29 (depends on 15)

The system of claim 15, further comprising sensors producing motion capture data.

### Claim 30 (depends on 15)

The system of claim 15, further comprising marked gloves used by a user to produce the sign language utterance.

### Claim 31 (depends on 15)

The system of claim 15, wherein the input interface further receives communication input from a second user to facilitate a conversation between the user and the second user.

### Claim at position 32

-45. (canceled)
