# US 2021/0279453 A1 · Methods and systems for computerized recognition of hand gestures

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279453 A1
- Document ID: US20210279453A1
- Status: Published application
- Kind: A1
- Application number: 16/809,939
- Filed: 2020-03-05
- Published: 2021-09-09
- Earliest claimed: 2020-03-05
- Assignees: King Fahd University of Petroleum and Minerals
- Applicants: King Fahd University of Petroleum and Minerals
- Inventors: Hamzah Luqman; El-Sayed M. El-Alfy; Galal M. Binmakhashen
- Web page: https://patentgrep.com/patent/US20210279453A1

## CPC

- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/2431: Multiple classes
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and computer readable media for recognizing one or more hand gestures of a hand-based signal or conversation are described. Some implementations include obtaining one or more unprocessed images of the hand-based signal or conversation including images of at least one of the one or more hand gestures extracting one or more spectral features from the one or more unprocessed images using a Gabor filter bank, receiving the one or more unprocessed images of the one or more hand gestures and the extracted one or more spectral features by a Convolution Neural Network (CNN), and outputting a classification for the at least one of the one or more hand gestures using the Convolution Neural Network (CNN). In some implementations, at least one of the extracted one or more spectral features and at least one of the one or more unprocessed images of the one or more hand gestures are concatenated and input to the Convolution Neural Network (CNN).

## Claims

### Claim 1

A computerized method for recognizing one or more hand gestures of a hand-based signal or conversation, the method comprising:
obtaining one or more unprocessed images of the hand-based signal or conversation, wherein the one or more unprocessed images include images of at least one of the one or more hand gestures, and wherein the one or more unprocessed images are being processed to determine a meaning of the hand-based signal or conversation;
extracting one or more spectral features from the one or more unprocessed images using a Gabor filter bank;
receiving the one or more unprocessed images of the one or more hand gestures and the extracted one or more spectral features by a Convolution Neural Network (CNN); and
outputting a classification for the at least one of the one or more hand gestures using the Convolution Neural Network (CNN), wherein the classification is selected by the CNN from among a plurality of candidate classifications to determine the meaning of the hand-based signal or conversation.

### Claim 2 (depends on 1)

The method of claim 1, wherein the Gabor filter bank performs Gabor filtering with a single scale and orientation to the one or more unprocessed images of the at least one of the one or more hand gestures, and wherein one or more of imaginary, phase, magnitude, or real Gabor responses are extracted as the extracted one or more spectral features for the one or more unprocessed images.

### Claim 3 (depends on 1)

The method of claim 1, wherein the Gabor filter bank performs Gabor filtering with four scales and four orientations, and wherein one or more of imaginary, phase, magnitude, or real Gabor responses are extracted as the extracted one or more spectral features for the one or more unprocessed images.

### Claim 4 (depends on 1)

The method of claim 1, wherein at least one of the extracted one or more spectral features and at least one of the one or more unprocessed images of the one or more hand gestures are concatenated and input to the Convolution Neural Network (CNN).

### Claim 5 (depends on 1)

The method of claim 1, wherein the Convolution Neural Network (CNN) has an architecture that includes a convolution layer with 128 kernels of size 5×5 with a stride of 1 pixel.

### Claim 6 (depends on 1)

The method of claim 1, wherein the hand-based signal or conversation includes a sign language that comprises at least one of an American MNIST replacement sign language alphabet (MNIST ASL), an Arabic sign language alphabet (ArSL), or a Massey University ASL dataset (MUASL).

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
creating a dataset from the sign language, wherein the dataset includes at least one of cropped hand gesture images, hand gesture images with noise added thereto, or the cropped hand gesture images with the noise added thereto.

### Claim 8

A system for recognizing one or more hand gestures of a hand-based signal or conversation, the system comprising:
one or more processors and a non-transitory computer readable storage having software instructions stored thereon configured to cause the one or more processors to:
obtain one or more unprocessed images of the hand-based signal or conversation, wherein the one or more unprocessed images include images of at least one of the one or more hand gestures, and wherein the one or more unprocessed images are being processed to determine a meaning of the hand-based signal or conversation;
extract one or more spectral features from the one or more unprocessed images using a Gabor filter bank;
receive the one or more unprocessed images of the one or more hand gestures and the extracted one or more spectral features by a Convolution Neural Network (CNN); and
output a classification for the at least one of the one or more hand gestures using the Convolution Neural Network (CNN), wherein the classification is selected by the CNN from among a plurality of candidate classifications to determine the meaning of the hand-based signal or conversation.

### Claim 9 (depends on 8)

The system of claim 8, wherein the Gabor filter bank performs Gabor filtering with a single scale and orientation to the one or more unprocessed images of the at least one of the one or more hand gestures, and wherein one or more of imaginary, phase, magnitude, or real Gabor responses are extracted as the extracted one or more spectral features for the one or more unprocessed images.

### Claim 10 (depends on 8)

The system of claim 8, wherein the Gabor filter bank performs Gabor filtering with four scales and four orientations, and wherein one or more of imaginary, phase, magnitude, or real Gabor responses are extracted as the extracted one or more spectral features for the one or more unprocessed images.

### Claim 11 (depends on 8)

The system of claim 8, wherein at least one of the extracted one or more spectral features and at least one of the one or more unprocessed images of the one or more hand gestures are concatenated and input to the Convolution Neural Network (CNN).

### Claim 12 (depends on 8)

The system of claim 8, wherein the Convolution Neural Network (CNN) has an architecture that includes a convolution layer with 128 kernels of size 5×5 with a stride of 1 pixel.

### Claim 13

A non-transitory computer readable medium having instructions stored therein that, when executed by one or more processors, cause the one or more processors to perform a method for recognizing one or more hand gestures of a hand-based signal or conversation, the method comprising:
obtaining one or more unprocessed images of the hand-based signal or conversation, wherein the one or more unprocessed images include images of at least one of the one or more hand gestures, and wherein the one or more unprocessed images are being processed to determine a meaning of the hand-based signal or conversation;
extracting one or more spectral features from the one or more unprocessed images using a Gabor filter bank;
receiving the one or more unprocessed images of the one or more hand gestures and the extracted one or more spectral features by a Convolution Neural Network (CNN); and
outputting a classification for the at least one of the one or more hand gestures using the Convolution Neural Network (CNN), wherein the classification is selected by the CNN from among a plurality of candidate classifications to determine the meaning of the hand-based signal or conversation.

### Claim 14 (depends on 13)

The non-transitory computer readable medium of claim 13, wherein the Gabor filter bank performs Gabor filtering with a single scale and orientation to the one or more unprocessed images of the at least one of the one or more hand gestures, and wherein one or more of imaginary, phase, magnitude, or real Gabor responses are extracted as the extracted one or more spectral features for the one or more unprocessed images.

### Claim 15 (depends on 13)

The non-transitory computer readable medium of claim 13, wherein the Gabor filter bank performs Gabor filtering with four scales and four orientations, and wherein one or more of imaginary, phase, magnitude, or real Gabor responses are extracted as the extracted one or more spectral features for the one or more unprocessed images.

### Claim 16 (depends on 13)

The non-transitory computer readable medium of claim 13, wherein at least one of the extracted one or more spectral features and at least one of the one or more unprocessed images of the one or more hand gestures are concatenated and input to the Convolution Neural Network (CNN).

### Claim 17 (depends on 13)

The non-transitory computer readable medium of claim 13, wherein the Convolution Neural Network (CNN) has an architecture that includes a convolution layer with 128 kernels of size 5×5 with a stride of 1 pixel.

### Claim 18 (depends on 13)

The non-transitory computer readable medium of claim 13, wherein the hand-based signal or conversation includes a sign language that comprises at least one of an American MNIST replacement sign language (MNIST) alphabet, an Arabic sign language (ArSL) alphabet, or a Massey University ASL (MUASL) dataset.

### Claim 19 (depends on 18)

The non-transitory computer readable medium of claim 18, further comprising:
creating a dataset from the sign language, wherein the dataset includes at least one of cropped hand gesture images, hand gesture images with noise added thereto, or the cropped hand gesture images with the noise added thereto.

### Claim 20 (depends on 13)

The non-transitory computer readable medium of claim 13, wherein the Convolution Neural Network (CNN) includes at least 11 layers followed by a Softmax classifier, and wherein one or more of the 11 layers is followed by one or more of an ReLU activation function, max-pooling, batch normalization, dropout, or one or more fully connected layers.
