# US 12,183,123 B2 · Automated sign language translation and communication using multiple input and output modalities

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,183,123 B2
- Document ID: US12183123B2
- Status: Granted
- Kind: B2
- Application number: 18/155,683
- Filed: 2023-01-17
- Published: 2023-11-23
- Granted: 2024-12-31
- Earliest claimed: 2018-02-12
- Applicants: Avodah, Inc.
- Inventors: Michael Menefee; Dallas Nash; Trevor Chandler
- Web page: https://patentgrep.com/patent/US12183123B2

## CPC

- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/73: using feature-based methods
- G06V 40/165: {using facial parts and geometric relationships}
- G06V 40/176: {Dynamic expression}
- G09B 21/00: Teaching, or communicating with, the blind, deaf or mute (audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B)
- G09B 21/009: {Teaching or communicating with deaf persons (G09B21/04 takes precedence)}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/24: Speech recognition using non-acoustical features
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06N 20/00: Machine learning
- G06T 3/4046: using neural networks
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G10L 13/00: Speech synthesis; Text to speech systems
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Methods, apparatus and systems for recognizing sign language movements using multiple input and output modalities. One example method includes capturing a movement associated with the sign language using a set of visual sensing devices, the set of visual sensing devices comprising multiple apertures oriented with respect to the subject to receive optical signals corresponding to the movement from multiple angles, generating digital information corresponding to the movement based on the optical signals from the multiple angles, collecting depth information corresponding to the movement in one or more planes perpendicular to an image plane captured by the set of visual sensing devices, producing a reduced set of digital information by removing at least some of the digital information based on the depth information, generating a composite digital representation by aligning at least a portion of the reduced set of digital information, and recognizing the movement based on the composite digital representation.

## Claims

### Claim 1

An apparatus for recognizing a sign language communicated by a subject, comprising:
a plurality of sensors for capturing movement and depth information corresponding to at least one movement associated with the sign language;
a processor; and
a memory including instructions stored thereupon, the instruction upon execution by the processor cause the processor to:
remove at least some digital information corresponding to the at least one movement based on the depth information;
generate a composite digital representation based on the digital information that is removed; and
identify, using a neural network engine, the at least one movement based on the composite digital representation.

### Claim 2 (depends on 1)

The apparatus of claim 1, further comprising a depth sensor for collecting the depth information, wherein the depth sensor is a structured-light depth sensor or a time-of-flight depth sensor.

### Claim 3 (depends on 1)

The apparatus of claim 1,
wherein at least some of the plurality of sensors are further configured to:
capture one or more of an eyebrow movement, a gaze direction or one or more blinks of the subject that are associated with the at least one movement, and
wherein the instruction upon execution by the processor further cause the processor to:
identify the at least one movement using information associated with the captured eyebrow movement, the gaze direction or the one or more blinks.

### Claim 4 (depends on 3)

The apparatus of claim 3, wherein the processor is further configured to:
use a facial recognition algorithm to generate the information associated with the eyebrow movement, the gaze direction or the one or more blinks.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein at least some of the plurality of sensors comprise one or more of an RGB color camera, a monochrome camera, a 3D stereo camera, a structured light emitter/receiver, or a time-of-flight emitter/receiver.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the processor is further configured to receive one or more of a Global Positioning System (GPS) 1 pulse-per-second (PPS) signal, a networking time protocol (NTP) or an SMPTE timecode to temporally align part of the digital information.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the apparatus is a handheld device that includes at least one display device, the at least one display device allowing a sign language rendition of a captured voice, or a textual representation of a sign language gesture including the at least one movement.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein the apparatus is implemented as a multi-component device, and wherein at least some of the plurality of sensors include three visual sensing devices, a first of the three visual sensing devices is positioned directly in front of the subject, a second of the three visual sensing devices is positioned substantially 90° to one side of the subject, and a third of the three visual sensing devices is positioned substantially 45° to another side of the subject.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the first, second and third of the three visual sensing devices are in a single horizontal spatial plane.

### Claim 10 (depends on 8)

The apparatus of claim 8, wherein the first and second of the three visual sensing devices are in a single horizontal spatial plane, and wherein the third of the three visual sensing devices is in an elevated position approximately 25° to 30° above the single horizontal spatial plane.

### Claim 11 (depends on 1)

The apparatus of claim 1, wherein at least some of the plurality of sensors further capture external information indicated by the subject, wherein the external information is associated with the at least one movement, and wherein identification of the at least one movement comprises using the external information to improve identification of the at least one movement.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the external information comprises a portion of a text, an object in a vicinity of the subject, or another person.

### Claim 13 (depends on 11)

The apparatus of claim 11, wherein the external information comprises environmental factors.

### Claim 14

A method for recognizing a sign language communicated by a subject, comprising:
capturing movement and depth information corresponding at least one movement associated with the sign language;
producing a reduced set of digital information by removing at least digital information corresponding to the at least one movement based on the depth information;
generating a composite digital representation based on the digital information that is removed; and
identifying, using a neural network engine, the at least one movement based on the composite digital representation.

### Claim 15 (depends on 14)

The method of claim 14, wherein the depth information is captured using a structured-light depth sensor or a time-of-flight depth sensor.

### Claim 16 (depends on 14)

The method of claim 14, further comprising:
capturing, one or more of an eyebrow movement, a gaze direction or one or more blinks of the subject that are associated with the at least one movement, and wherein identifying the at least one movement comprises using information associated with the captured eyebrow movement, the gaze direction or the one or more blinks to improve recognition of the at least one movement.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
using a facial recognition algorithm to generate the information associated with the eyebrow movement, the gaze direction or the one or more blinks.

### Claim 18 (depends on 14)

The method of claim 14, wherein generating the composite digital representation based on the digital information that is removed includes using one or more of a Global Positioning System (GPS) 1 pulse-per-second (PPS) signal, a networking time protocol (NTP) or an SMPTE timecode to temporally align part of the reduced set of digital information.

### Claim 19 (depends on 14)

The method of claim 14, wherein the composite digital representation comprises a point-cloud or a multi-frame three-dimensional model.