# US 2021/0334524 A1 · Gesture recognition method and terminal device and computer readable storage medium using the same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334524 A1
- Document ID: US20210334524A1
- Status: Published application
- Kind: A1
- Application number: 17/118,578
- Filed: 2020-12-10
- Published: 2021-10-28
- Earliest claimed: 2020-04-22
- Assignees: Ubtech Robotics Corp Ltd
- Applicants: Ubtech Robotics Corp Ltd
- Inventors: Miaochen Guo; Jingtao Zhang; Shuping Hu; Dong Wang; Zaiwang Gu; Jianxin Pang; Youjun Xiong
- Web page: https://patentgrep.com/patent/US20210334524A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/0485: Scrolling or panning
- G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 5/70: Denoising; Smoothing
- G06T 7/64: of convexity or concavity
- G06T 7/73: using feature-based methods
- G06T 7/90: Determination of colour characteristics
- G06V 10/82: using neural networks
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- H04N 1/6075: {Corrections to the hue}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20084: Artificial neural networks [ANN]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The present disclosure provides a gesture recognition method as well as a terminal device and a computer-readable storage medium using the same. The method includes: obtaining a video stream collected by an image recording device in real time; performing a hand recognition on the video stream to determine static gesture information of a recognized hand in each video frame of the video stream; encoding the static gesture information in the video frames of the video stream in sequence to obtain an encoded information sequence of the recognized hands; and performing a slide detection on the encoded information sequence using a preset sliding window to determine a dynamic gesture category of each recognized hand. In this manner, static gesture recognition and dynamic gesture recognition are effectively integrated in the same process. The dynamic gesture recognition is realized through the slide detection of the sliding window without complex network calculations.

## Claims

### Claim 1

A computer-implemented gesture recognition method, comprising steps of:
obtaining a video stream collected by an image recording device;
performing a hand recognition on the video stream to determine static gesture information of at least a recognized hand in each video frame of the video stream;
encoding the static gesture information in the video frames of the video stream in sequence to obtain an encoded information sequence of the recognized hands; and
performing a slide detection on the encoded information sequence using a preset sliding window to determine a dynamic gesture category of each recognized hand.

### Claim 2 (depends on 1)

The method of claim 1, wherein the static gesture information comprises a static gesture category and a hand centroid position, and the step of performing the hand recognition on the video stream to determine the static gesture information of the recognized hand in each video frame of the video stream comprises:
performing the hand recognition on a target video frame through a neural network classifier to obtain the static gesture category of the recognized hand in the target video frame, in response to the hand recognition being in a short-distance recognition mode, wherein the target video frame is any one of the video frames of the video stream; and
calculating the hand centroid position in the target video frame by performing an image processing on the target video frame.

### Claim 3 (depends on 2)

The method of claim 2, wherein the step of calculating the hand centroid position in the target video frame by performing the image processing on the target video frame comprises:
performing a preprocessing on the target video frame to obtain a hand mask image in the target video frame;
binarizing the hand mask image to obtain a binarized hand image;
extracting a contour from the binarized hand image, and selecting the contour with the largest area from the extracted contour as a hand contour; and
performing a distance transformation on the hand contour to calculate the hand centroid position in the target video frame.

### Claim 4 (depends on 3)

The method of claim 3, wherein the step of performing a preprocessing on the target video frame to obtain the hand mask image in the target video frame comprises:
smoothing the target video frame using Gaussian filtering to obtain a smooth image;
converting the smooth image from the RGB color space to the HSV color space to obtain a space-converted image;
performing a feature extraction on the space-converted image using a preset elliptical skin color model to obtain a feature image; and
filtering out an impurity area in the feature image through a morphological opening and closing operation to obtain the hand mask image.

### Claim 5 (depends on 1)

The method of claim 1, wherein the static gesture information comprises a static gesture category and a hand centroid position, and the step of performing the hand recognition on the video stream to determine the static gesture information of the recognized hand in each video frame of the video stream comprises:
performing a hand recognition on each video frame of the video stream through a neural network target detector to determine the static gesture category and the hand centroid position of each recognized hand in each video frame of the video stream, in response to the hand recognition being in a long-distance recognition mode; and
determining a matchingness of each recognized hand in each video frame, and building a tracker corresponding to each recognized hand.

### Claim 6 (depends on 5)

The method of claim 5, wherein the step of determining the matchingness of each recognized hand in each video frame, and building the tracker corresponding to each recognized hand comprises:
calculating a predicted detection frame in the next video frame according to a hand bounding box in the current video frame using a Kalman filter, and determining a hand bounding box in the next video frame according to the predicted detection frame;
performing a Hungarian matching between the hand bounding box in the current video frame and the hand bounding box in the next video frame to determine the matchingness of each recognized hand in the current video frame and the next video frame; and
building the tracker corresponding to the matched recognized hand, in response to the matched recognized hand being successfully matched in a plurality of the video frames consecutive in the video stream.

### Claim 7 (depends on 1)

The method of claim 1, wherein the step of performing the slide detection on the encoded information sequence using the preset sliding window to determine the dynamic gesture category of each recognized hand comprises:
detecting a key frame in the encoded information sequence within the current sliding window, and determining the dynamic gesture category corresponding to the encoded information sequence in response to the detected key frame meeting a preset pattern characteristic; and
sliding the sliding window with one frame backward in the encoded information sequence, and returning to the step of detecting the key frame in the encoded information sequence within the current sliding window until a gesture recognition process is terminated.

### Claim 8

A terminal device, comprising:
an image recording device;
a memory;
a processor; and
one or more computer programs stored in the memory and executable on the processor, wherein the one or more computer programs comprise:
instructions for obtaining a video stream collected by the image recording device;
instructions for performing a hand recognition on the video stream to determine static gesture information of at least a recognized hand in each video frame of the video stream;
instructions for encoding the static gesture information in the video frames of the video stream in sequence to obtain an encoded information sequence of the recognized hands; and
instructions for performing a slide detection on the encoded information sequence using a preset sliding window to determine a dynamic gesture category of each recognized hand.

### Claim 9 (depends on 8)

The terminal device of claim 8, wherein the static gesture information comprises a static gesture category and a hand centroid position, and the instructions for performing the hand recognition on the video stream to determine the static gesture information of the recognized hand in each video frame of the video stream comprise:
instructions for performing the hand recognition on a target video frame through a neural network classifier to obtain the static gesture category of the recognized hand in the target video frame, in response to the hand recognition being in a short-distance recognition mode, wherein the target video frame is any one of the video frames of the video stream; and
instructions for calculating the hand centroid position in the target video frame by performing an image processing on the target video frame.

### Claim 10 (depends on 9)

The terminal device of claim 9, wherein the instructions for calculating the hand centroid position in the target video frame by performing the image processing on the target video frame comprise:
instructions for performing a preprocessing on the target video frame to obtain a hand mask image in the target video frame;
instructions for binarizing the hand mask image to obtain a binarized hand image;
instructions for extracting a contour from the binarized hand image, and selecting the contour with the largest area from the extracted contour as a hand contour; and
instructions for performing a distance transformation on the hand contour to calculate the hand centroid position in the target video frame.

### Claim 11 (depends on 10)

The terminal device of claim 10, wherein the instructions for performing a preprocessing on the target video frame to obtain the hand mask image in the target video frame comprise:
instructions for smoothing the target video frame using Gaussian filtering to obtain a smooth image;
instructions for converting the smooth image from the RGB color space to the HSV color space to obtain a space-converted image;
instructions for performing a feature extraction on the space-converted image using a preset elliptical skin color model to obtain a feature image; and
instructions for filtering out an impurity area in the feature image through a morphological opening and closing operation to obtain the hand mask image.

### Claim 12 (depends on 8)

The terminal device of claim 8, wherein the static gesture information comprises a static gesture category and a hand centroid position, and the instructions for performing the hand recognition on the video stream to determine the static gesture information of the recognized hand in each video frame of the video stream comprise:
instructions for performing a hand recognition on each video frame of the video stream through a neural network target detector to determine the static gesture category and the hand centroid position of each recognized hand in each video frame of the video stream, in response to the hand recognition being in a long-distance recognition mode; and
instructions for determining a matchingness of each recognized hand in each video frame, and building a tracker corresponding to each recognized hand.

### Claim 13 (depends on 12)

The terminal device of claim 12, wherein the instructions for determining the matchingness of each recognized hand in each video frame, and building the tracker corresponding to each recognized hand comprise:
instructions for calculating a predicted detection frame in the next video frame according to a hand bounding box in the current video frame using a Kalman filter, and determining a hand bounding box in the next video frame according to the predicted detection frame;
instructions for performing a Hungarian matching between the hand bounding box in the current video frame and the hand bounding box in the next video frame to determine the matchingness of each recognized hand in the current video frame and the next video frame; and
instructions for building the tracker corresponding to the matched recognized hand, in response to the matched recognized hand being successfully matched in a plurality of the video frames consecutive in the video stream.

### Claim 14 (depends on 8)

The terminal device of claim 8, wherein the instructions for performing the slide detection on the encoded information sequence using the preset sliding window to determine the dynamic gesture category of each recognized hand comprise:
instructions for detecting a key frame in the encoded information sequence within the current sliding window, and determining the dynamic gesture category corresponding to the encoded information sequence in response to the detected key frame meeting a preset pattern characteristic; and
instructions for sliding the sliding window with one frame backward in the encoded information sequence, and returning to detect the key frame in the encoded information sequence within the current sliding window until a gesture recognition process is terminated.

### Claim 15

A non-transitory computer readable storage medium for storing one or more computer programs, wherein the one or more computer programs comprise:
instructions for obtaining a video stream collected by an image recording device;
instructions for performing a hand recognition on the video stream to determine static gesture information of at least a recognized hand in each video frame of the video stream;
instructions for encoding the static gesture information in the video frames of the video stream in sequence to obtain an encoded information sequence of the recognized hands; and
instructions for performing a slide detection on the encoded information sequence using a preset sliding window to determine a dynamic gesture category of each recognized hand.

### Claim 16 (depends on 15)

The storage medium of claim 15, wherein the static gesture information comprises a static gesture category and a hand centroid position, and the instructions for performing the hand recognition on the video stream to determine the static gesture information of the recognized hand in each video frame of the video stream comprise:
instructions for performing the hand recognition on a target video frame through a neural network classifier to obtain the static gesture category of the recognized hand in the target video frame, in response to the hand recognition being in a short-distance recognition mode, wherein the target video frame is any one of the video frames of the video stream; and
instructions for calculating the hand centroid position in the target video frame by performing an image processing on the target video frame.

### Claim 17 (depends on 16)

The storage medium of claim 16, wherein the instructions for calculating the hand centroid position in the target video frame by performing the image processing on the target video frame comprise:
instructions for performing a preprocessing on the target video frame to obtain a hand mask image in the target video frame;
instructions for binarizing the hand mask image to obtain a binarized hand image;
instructions for extracting a contour from the binarized hand image, and selecting the contour with the largest area from the extracted contour as a hand contour; and
instructions for performing a distance transformation on the hand contour to calculate the hand centroid position in the target video frame.

### Claim 18 (depends on 17)

The storage medium of claim 17, wherein the instructions for performing a preprocessing on the target video frame to obtain the hand mask image in the target video frame comprise:
instructions for smoothing the target video frame using Gaussian filtering to obtain a smooth image;
instructions for converting the smooth image from the RGB color space to the HSV color space to obtain a space-converted image;
instructions for performing a feature extraction on the space-converted image using a preset elliptical skin color model to obtain a feature image; and
instructions for filtering out an impurity area in the feature image through a morphological opening and closing operation to obtain the hand mask image.

### Claim 19 (depends on 15)

The storage medium of claim 15, wherein the static gesture information comprises a static gesture category and a hand centroid position, and the instructions for performing the hand recognition on the video stream to determine the static gesture information of the recognized hand in each video frame of the video stream comprise:
instructions for performing a hand recognition on each video frame of the video stream through a neural network target detector to determine the static gesture category and the hand centroid position of each recognized hand in each video frame of the video stream, in response to the hand recognition being in a long-distance recognition mode; and
instructions for determining a matchingness of each recognized hand in each video frame, and building a tracker corresponding to each recognized hand.

### Claim 20 (depends on 19)

The storage medium of claim 19, wherein the instructions for determining the matchingness of each recognized hand in each video frame, and building the tracker corresponding to each recognized hand comprise:
instructions for calculating a predicted detection frame in the next video frame according to a hand bounding box in the current video frame using a Kalman filter, and determining a hand bounding box in the next video frame according to the predicted detection frame;
instructions for performing a Hungarian matching between the hand bounding box in the current video frame and the hand bounding box in the next video frame to determine the matchingness of each recognized hand in the current video frame and the next video frame; and
instructions for building the tracker corresponding to the matched recognized hand, in response to the matched recognized hand being successfully matched in a plurality of the video frames consecutive in the video stream.
