# US 2025/0029275 A1 · Automated gesture identification using neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0029275 A1
- Document ID: US20250029275A1
- Status: Published application
- Kind: A1
- Application number: 18/648,050
- Filed: 2024-04-26
- Published: 2025-01-23
- Earliest claimed: 2018-02-12
- Assignees: Avodah, Inc.
- Applicants: Avodah, Inc.
- Inventors: Trevor Chandler; Dallas Nash; Michael Menefee
- Web page: https://patentgrep.com/patent/US20250029275A1

## CPC

- G06T 7/73: using feature-based methods
- G06V 10/82: using neural networks
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 18/256: {of results relating to different input data, e.g. multimodal recognition}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/90: Determination of colour characteristics
- G06V 40/165: {using facial parts and geometric relationships}
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Disclosed are methods, apparatus and systems for gesture recognition based on neural network processing. One exemplary method for identifying a gesture communicated by a subject includes receiving a plurality of images associated with the gesture, providing the plurality of images to a first 3-dimensional convolutional neural network (3D CNN) and a second 3D CNN, where the first 3D CNN is operable to produce motion information, where the second 3D CNN is operable to produce pose and color information, and where the first 3D CNN is operable to implement an optical flow algorithm to detect the gesture, fusing the motion information and the pose and color information to produce an identification of the gesture, and determining whether the identification corresponds to a singular gesture across the plurality of images using a recurrent neural network that comprises one or more long short-term memory units.

## Claims

### Claim 1

A neural network system adapted for processing images associated with a gesture, comprising:
one or more three-dimensional convolution neural networks (3D CNNs), each of the 3D CNNs comprising:
an input to receive a plurality of input images, and
an output to provide recognition information produced by each of the 3D CNNs;
at least one recurrent neural network (RNN) comprising:
an input to receive a second type of recognition information, and
an output that is coupled to the input of the at least one RNN to provide a feedback connection; and
at least one processor coupled to a non-transitory memory with instructions stored thereon, the instructions upon execution by the at least one processor, causing the at least one processor to:
receive a plurality of captured images at a pre-processing module, perform pose estimation on each of the plurality of captured images, and overlay pose estimation pixels onto the plurality of captured images to generate the plurality of input images for consumption by the one or more 3D CNNs, and
receive the recognition information produced by each of the one or more 3D CNNs at a fusion module, and aggregate the received recognition information to generate the second type of recognition information for consumption by the at least one RNN,
wherein each of the one or more 3D CNNs is operable to produce the recognition information comprising at least one characteristic associated with the gesture in each of the plurality of images, and provide the recognition information to the fusion module, the at least one characteristic comprising a pose, a color or a gesture type, and
wherein the at least one RNN is operable to determine whether the recognition information produced by the one or more 3D CNNs corresponds to a singular gesture across the plurality of input images.

### Claim 2 (depends on 1)

The system of claim 1, wherein a first of the one or more 3D CNNs is operable to use an optical flow algorithm to detect the gesture.

### Claim 3 (depends on 2)

The system of claim 2, wherein the optical flow algorithm comprises sharpening, line, edge, corner and shape enhancements.

### Claim 4 (depends on 1)

The system of claim 1, wherein the pose estimation identifies the body, fingers and face of a subject performing the gesture.

### Claim 5 (depends on 1)

The system of claim 1, wherein the one or more 3D CNNs and the at least one RNN are implemented on the at least one processor.

### Claim 6 (depends on 1)

The system of claim 1, wherein the instructions upon execution by the at least one processor further cause the at least one processor, prior to performing pose estimation, to:
process the plurality of captured images to extract pixels corresponding to a subject, a foreground and a background in each of the plurality of captured images;
perform spatial filtering, upon determining that the background or the foreground includes no information that pertains to the gesture being identified, to remove the pixels corresponding to the background or the foreground.

### Claim 7 (depends on 1)

The system of claim 1, wherein a first 3D CNN of the one or more 3D CNNs is operable to use an optical flow algorithm to detect the gesture, and wherein the at least one characteristic identified by the first 3D CNN comprises the gesture type.

### Claim 8 (depends on 7)

The system of claim 7, wherein a second 3D CNN of the one or more 3D CNNs is operable to process the pose estimation pixels to associate a pose with the gesture, and wherein the at least one characteristic identified by the second 3D CNN comprises the pose.

### Claim 9 (depends on 8)

The system of claim 8, wherein a third 3D CNN of the one or more 3D CNNs is operable to process an expert analysis of the gesture, and wherein the at least one characteristic identified by the third 3D CNN comprises the gesture type.

### Claim 10 (depends on 1)

The system of claim 1, wherein the at least one RNN comprises one or more long short-term memory (LTSM) units.

### Claim 11 (depends on 1)

The system of claim 1, wherein the fusion module is operable to concurrently process at least one output node of each of the one or more 3D CNNs.

### Claim 12

An artificial intelligence system for identifying a gesture communicated by a subject, the system comprising:
at least one processor; and
non-transitory memory including processor executable code, wherein the processor executable code upon execution by the at least one processor causes the at least one processor to:
receive a plurality of images associated with the gesture;
provide the plurality of images to a first 3-dimensional convolutional neural network (3D CNN) and a second 3D CNN, wherein the first 3D CNN is operable to produce motion information, wherein the second 3D CNN is operable to produce pose and color information, wherein the first 3D CNN is operable to implement an optical flow algorithm to detect the gesture;
fuse the motion information and the pose and color information to produce an identification of the gesture; and
determine whether the identification corresponds to a singular gesture across the plurality of images using a recurrent neural network (RNN) that comprises one or more long short-term memory (LTSM) units.

### Claim 13 (depends on 10)

The system of claim 10, wherein the processor executable code upon execution by the at least one processor further configures the at least one processor to:
pre-process a plurality of captured images to generate the plurality of images,
wherein the processor executable code upon execution by the at least one processor configures the at least one processor, as part of pre-processing the plurality of captured images, to:
apply a pose estimation algorithm to pixels of each of the plurality of images to produce overlay pixels corresponding to body, fingers and face of the subject;
extract color and spatial information from each of the plurality of images; and
combine the overlay pixels and the color and spatial information to produce the plurality of images.

### Claim 14 (depends on 11)

The system of claim 11, wherein the processor executable code upon execution by the at least one processor further configures the at least one processor to:
identify, based on the overlay pixels corresponding to the face of the subject, lip movements of the subject; and
produce, based on the lip movements of the subject, textual information.

### Claim 15 (depends on 11)

The system of claim 11, wherein the overlay pixels comprise pixels with different colors for each finger of the subject.

### Claim 16 (depends on 10)

The system of claim 10, wherein the optical flow algorithm comprises sharpening, line, edge, corner and shape enhancements.

### Claim 17 (depends on 10)

The system of claim 10, wherein the first 3D CNN and the second 3D CNN process the plurality of processed images concurrently.

### Claim 18 (depends on 10)

The system of claim 10, wherein the processor executable code upon execution by the at least one processor further configures the at least one processor to:
receive, from one or more additional sources, information associated with the sign language gesture, the information being different from the plurality of images; and
process the information using one or more additional 3D CNNs to produce processed information, wherein the identification of the sign language gesture is based on fusing the processed information with the motion information and the pose and color information,
wherein the one or more additional sources comprise an audio sensor, an environmental sensor or an expert analysis of the sign language gesture.

### Claim 19

A method for identifying gestures communicated by a subject, comprising:
receiving a plurality of images associated with the gesture;
providing the plurality of images to a first 3-dimensional convolutional neural network (3D CNN) and a second 3D CNN, wherein the first 3D CNN is operable to produce motion information, wherein the second 3D CNN is operable to produce pose and color information, wherein the first 3D CNN is operable to implement an optical flow algorithm to detect the gesture;
fusing the motion information and the pose and color information to produce an identification of the gesture; and
determining whether the identification corresponds to a singular gesture across the plurality of images using a recurrent neural network (RNN) that comprises one or more long short-term memory (LTSM) units.

### Claim 20 (depends on 10)

The method of claim 10, further comprising:
pre-processing a plurality of captured images to generate the plurality of images,
wherein the pre-processing the plurality of captured images comprises:
applying a pose estimation algorithm to pixels of each of the plurality of images to produce overlay pixels corresponding to body, fingers and face of the subject;
extracting color and spatial information from each of the plurality of images; and
combining the overlay pixels and the color and spatial information to produce the plurality of images.

### Claim 21 (depends on 18)

The method of claim 18, further comprising:
identifying, based on the overlay pixels corresponding to the face of the subject, lip movements of the subject; and
producing, based on the lip movements of the subject, textual information.

### Claim 22 (depends on 18)

The method of claim 18, wherein the overlay pixels comprise pixels with different colors for each finger of the subject.
