# US 11,307,668 B2 · Gesture recognition method and apparatus, electronic device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,307,668 B2
- Document ID: US11307668B2
- Status: Granted
- Kind: B2
- Application number: 16/901,111
- Filed: 2020-06-15
- Published: 2021-06-24
- Granted: 2022-04-19
- Earliest claimed: 2019-12-23
- Assignees: Sensetime International Pte. Ltd.
- Applicants: Sensetime International Pte. Ltd.
- Inventors: Haiyu Zhao; Kaige Chen; Daming Niu
- Web page: https://patentgrep.com/patent/US11307668B2

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 40/107: {Static hand or arm}
- G06V 40/113: {Recognition of static hand signs}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G07F 17/3239: {Tracking of individual players}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07F: COIN-FREED OR LIKE APPARATUS (coin sorting G07D3/00; coin testing G07D5/00; {handling coins or paper currencies apart from payment activated apparatus G07D; payment architectures, schemes or protocols G06Q20/00})

## Abstract

Disclosed are a gesture recognition method and apparatus, an electronic device, and a storage medium. The method includes: obtaining a hand detection result by detecting a to-be-processed image; in response to the hand detection result including a bounding box of hand, determining that a hand pose in the bounding box of hand conforms to a hand pose in a target gesture; determining that the hand is located in a pre-defined area of the to-be-processed image; and determining that the hand makes the target gesture.

## Claims

### Claim 1

A gesture recognition method, comprising:
obtaining a hand detection result by detecting a to-be-processed image, wherein the to-be-processed image is an image obtained by capturing a target area;
in response to the hand detection result comprising a bounding box of a hand, determining that a hand pose in the bounding box of the hand conforms to a hand pose in a target gesture;
determining that the hand is located in a pre-defined area of the to-be-processed image, wherein the pre-defined area in the to-be-processed image comprises two set sub-areas corresponding to two sub-areas in the target area respectively; and
determining that the hand in the to-be-processed image makes the target gesture,
wherein determining that the hand is located in the pre-defined area of the to-be-processed image comprises:
determining that center points of two bounding boxes of the hand are located in the two set sub-areas of the to-be-processed image, respectively.

### Claim 2 (depends on 1)

The method according to claim 1, wherein in response to the hand detection result comprising the bounding box of the hand, determining that the hand pose in the bounding box of the hand conforms to the hand pose in the target gesture comprises:
in response to the hand detection result comprising the bounding box of the hand, cropping an image located in the bounding box of the hand in the to-be-processed image; and
determining, according to the cropped image, that the hand pose in the cropped image conforms to the hand pose in the target gesture.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the hand pose in the target gesture is two hands both facing up; and
determining that the hand pose in the bounding box of the hand conforms to the hand pose in the target gesture comprises:
inputting an image in the bounding box of the hand into a hand classification neural network to obtain a hand classification result, the hand classification result comprising the hand facing up or the hand with other poses; and
when hand classification results of two bounding boxes of a hand are hands facing up, determining that hand poses in the two bounding boxes of the hand conform to the hand pose in the target gesture.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the hand classification neural network is obtained by training with hand image samples, a hand image sample comprising an image sample for the hand facing up, and an annotation information of the image sample for the hand facing up being a palm of the hand.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the to-be-processed image is an image of the target area captured at a top view.

### Claim 6

An electronic device, comprising:
a processor, and
a memory for storing computer instructions executable by the processor,
wherein the processor is configured to:
obtain a hand detection result by detecting a to-be-processed image, wherein the to-be-processed image is an image obtained by capturing a target area;
in response to the hand detection result comprising a bounding box of a hand, determine that a hand pose in the bounding box of the hand conforms to a hand pose in a target gesture;
determine that the hand is located in a pre-defined area of the to-be-processed image, wherein the pre-defined area in the to-be-processed image comprises two set sub-areas corresponding to two sub-areas in the target area respectively; and
determine that the hand in the to-be-processed image makes the target gesture,
wherein the processor is configured to determine that the hand is located in the pre-defined area of the to-be-processed image by:
determining that center points of two bounding boxes of the hand are located in the two set sub-areas of the to-be-processed image, respectively.

### Claim 7 (depends on 6)

The electronic device according to claim 6, wherein in response to the hand detection result comprising the bounding box of the hand, determining that the hand pose in the bounding box of the hand conforms to the hand pose in the target gesture comprises:
in response to the hand detection result comprising the bounding box of the hand, cropping an image located in the bounding box of the hand in the to-be-processed image; and
determining, according to the cropped image, that the hand pose in the cropped image conforms to the hand pose in the target gesture.

### Claim 8 (depends on 6)

The electronic device according to claim 6, wherein the hand pose in the target gesture is two hands both facing up; and
determining that the hand pose in the bounding box of the hand conforms to the hand pose in the target gesture comprises:
inputting an image in the bounding box of the hand into a hand classification neural network to obtain a hand classification result, the hand classification result comprising the hand facing up or the hand with other poses; and
when hand classification results of two bounding boxes of a hand are hands facing up, determining that hand poses in the two bounding boxes of the hand conform to the hand pose in the target gesture.

### Claim 9 (depends on 8)

The electronic device according to claim 8, wherein the hand classification neural network is obtained by training with hand image samples, a hand image sample comprising an image sample for the hand facing up, and an annotation information of the image sample for the hand facing up being a palm of the hand.

### Claim 10 (depends on 6)

The electronic device according to claim 6, wherein the to-be-processed image is an image of the target area captured at a top view.

### Claim 11

A non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, cause the processor to:
obtain a hand detection result by detecting a to-be-processed image, wherein the to-be-processed image is an image obtained by capturing a target area;
in response to the hand detection result comprising a bounding box of a hand, determine that a hand pose in the bounding box of the hand conforms to a hand pose in a target gesture;
determine that the hand is located in a pre-defined area of the to-be-processed image, wherein the pre-defined area in the to-be-processed image comprises two set sub-areas corresponding to two sub-areas in the target area respectively; and
determine that the hand in the to-be-processed image makes the target gesture;
wherein the computer program causes the processor to determine that the hand is located in the pre-defined area of the to-be-processed image by:
determining that center points of two bounding boxes of the hand are located in the two set sub-areas of the to-be-processed image, respectively.