# US 12,217,496 B2 · Hand gesture detection method involves acquiring initial depth image using backbone and apparatus, and non-transitory computer-readable storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,496 B2 - Document ID: US12217496B2 - Status: Granted - Kind: B2 - Application number: 17/748,907 - Filed: 2022-05-19 - Published: 2022-09-01 - Granted: 2025-02-04 - Earliest claimed: 2019-11-20 - Assignees: Guangdong Oppo Mobile Telecommunications Corp., Ltd. - Applicants: Guangdong Oppo Mobile Telecommunications Corp., Ltd. - Inventors: Yang Zhou; Jie Liu - Web page: https://patentgrep.com/patent/US12217496B2 ## CPC - G06V 10/82: using neural networks - G06T 7/50: Depth or shape recovery - G06T 7/75: {involving models} - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20081: Training; Learning - G06T 2207/20132: Image cropping - G06V 10/803: {of input or preprocessed data} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Provided are a hand gesture detection method and device, and a computer storage medium. The method includes: obtaining an initial depth image including a hand to be detected, and performing detection processing on the initial depth image by using a backbone feature extractor and a bounding box detection model to obtain initial bounding boxes and a first feature map corresponding to the hand to be detected; determining one of the initial bounding boxes as a target bounding box; cropping, based on the target bounding box, the first feature map by using RoIAlign feature extractor, to obtain a second feature map corresponding to the hand to be detected; and performing, based on the second feature map, a three-dimensional gesture estimation processing on the hand to be detected by using a gesture estimation model to obtain a gesture detection result of the hand to be detected. ## Claims ### Claim 1 A hand gesture detection method, comprising: obtaining an initial depth image comprising a hand to be detected, and performing detection processing on the initial depth image by using a backbone feature extractor and a bounding box detection model, to obtain initial bounding boxes and a first feature map corresponding to the hand to be detected; determining a target bounding box based on the initial bounding boxes, the target bounding box being one of the initial bounding boxes; cropping, based on the target bounding box, the first feature map by using an RoIAlign feature extractor, to obtain a second feature map corresponding to the hand to be detected; and performing, based on the second feature map, a three-dimensional gesture estimation processing on the hand to be detected by using a gesture estimation model to obtain a gesture detection result of the hand to be detected. ### Claim 2 (depends on 1) The method according to claim 1, wherein said performing the detection processing on the initial depth image by using the backbone feature extractor and the bounding box detection model, to obtain the initial bounding boxes and the first feature map corresponding to the hand to be detected comprises: inputting the initial depth image into the backbone feature extractor, and outputting the first feature map; and obtaining, based on the first feature map, the initial bounding boxes by using the bounding box detection model. ### Claim 3 (depends on 2) The method according to claim 2, wherein said obtaining, based on the first feature map, the initial bounding boxes by using the bounding box detection model comprises: inputting the first feature map into the bounding box detection model, and outputting a plurality of bounding boxes and a plurality of confidences corresponding to the plurality of bounding boxes in one-to-one correspondence; and determining, based on the plurality of confidences, a part of the plurality of bounding boxes in the plurality of bounding boxes as the initial bounding boxes. ### Claim 4 (depends on 2) The method according to claim 2, wherein said determining the target bounding box based on the initial bounding boxes comprises: determining, as the target bounding box, an initial bounding box corresponding to a maximum confidence among the plurality of confidences corresponding to the initial bounding boxes. ### Claim 5 (depends on 3) The method of claim 3, wherein said determining the target bounding box based on the initial bounding boxes comprises: determining an intersection parameter between the initial bounding boxes; down-sampling, when the intersection parameter is greater than a predetermined intersection threshold, the initial bounding boxes to obtain spare bounding boxes; and determining, as the target bounding box, a bounding box corresponding to a maximum confidence among the plurality of confidences corresponding to the spare bounding boxes. ### Claim 6 (depends on 1) The method according to claim 1, wherein said cropping, based on the target bounding box, the first feature map by using the RoIAlign feature extractor to obtain the second feature map corresponding to the hand to be detected comprises: inputting the target bounding box and the first feature map into the RoIAlign feature extractor, and outputting the second feature map. ### Claim 7 (depends on 6) The method according to claim 6, wherein said cropping, based on the target bounding box, the first feature map by using the RoIAlign feature extractor to obtain the second feature map corresponding to the hand to be detected comprises: determining a cropping region based on the target bounding box; and cropping, based on the cropping region, the first feature map by using the RoIAlign feature extractor to obtain the second feature map. ### Claim 8 (depends on 1) The method according to claim 1, wherein said performing, based on the second feature map, the three-dimensional gesture estimation processing on the hand to be detected by using the gesture estimation model to obtain the gesture detection result of the hand to be detected comprises: inputting the second feature map into the gesture estimation model, and outputting the gesture detection result. ### Claim 9 (depends on 1) The method according to claim 1, further comprising: building a detection model and an estimation model; based on each of a plurality of training images comprised in a training sample set, performing model training on the detection model by using a first predetermined loss function, and performing model training on the estimation model by using a second predetermined loss function; determining, when a loss value of the first predetermined loss function is within a first predetermined interval, a trained detection model as the bounding box detection model; and determining, when a loss value of the second predetermined loss function is within a second predetermined interval, a trained estimation model as the gesture estimation model. ### Claim 10 A hand gesture detection apparatus, comprising: a processor; and a memory having instructions stored thereon and executable by the processor, wherein the instructions, when executed by the processor, implement a hand gesture detection method comprising: obtaining an initial depth image comprising a hand to be detected, and performing detection processing on the initial depth image by using a backbone feature extractor and a bounding box detection model, to obtain initial bounding boxes and a first feature map corresponding to the hand to be detected; determining a target bounding box based on the initial bounding boxes, the target bounding box being one of the initial bounding boxes; cropping, based on the target bounding box, the first feature map by using an RoIAlign feature extractor, to obtain a second feature map corresponding to the hand to be detected; and performing, based on the second feature map, a three-dimensional gesture estimation processing on the hand to be detected by using a gesture estimation model to obtain a gesture detection result of the hand to be detected. ### Claim 11 (depends on 10) The apparatus according to claim 10, wherein the instructions, when executed by the processor, implement: inputting the initial depth image into the backbone feature extractor, and outputting the first feature map; and obtaining, based on the first feature map, the initial bounding boxes by using the bounding box detection model. ### Claim 12 (depends on 11) The apparatus according to claim 11, wherein the instructions, when executed by the processor, implement: inputting the first feature map into the bounding box detection model, and outputting a plurality of bounding boxes and a plurality of confidences corresponding to the plurality of bounding boxes in one-to-one correspondence; and determining, based on the plurality of confidences, a part of the plurality of bounding boxes in the plurality of bounding boxes as the initial bounding boxes. ### Claim 13 (depends on 11) The apparatus according to claim 11, wherein the instructions, when executed by the processor, implement: determining, as the target bounding box, an initial bounding box corresponding to a maximum confidence among the plurality of confidences corresponding to the initial bounding boxes. ### Claim 14 (depends on 13) The apparatus according to claim 13, wherein the instructions, when executed by the processor, implement: determining an intersection parameter between the initial bounding boxes; down-sampling, when the intersection parameter is greater than a predetermined intersection threshold, the initial bounding boxes to obtain spare bounding boxes; and determining, as the target bounding box, a bounding box corresponding to a maximum confidence among the plurality of confidences corresponding to the spare bounding boxes. ### Claim 15 (depends on 10) The apparatus according to claim 10, wherein the instructions, when executed by the processor, implement: inputting the target bounding box and the first feature map into the RoIAlign feature extractor, and outputting the second feature map. ### Claim 16 (depends on 15) The apparatus according to claim 15, wherein the instructions, when executed by the processor, implement: determining a cropping region based on the target bounding box; and cropping, based on the cropping region, the first feature map by using the RoIAlign feature extractor to obtain the second feature map. ### Claim 17 (depends on 10) The apparatus according to claim 10, wherein the instructions, when executed by the processor, implement: inputting the second feature map into the gesture estimation model, and outputting the gesture detection result. ### Claim 18 (depends on 10) The apparatus according to claim 10, wherein the instructions, when executed by the processor, further implement: building a detection model and an estimation model; based on each of a plurality of training images comprised in a training sample set, performing model training on the detection model by using a first predetermined loss function, and performing model training on the estimation model by using a second predetermined loss function; determining, when a loss value of the first predetermined loss function is within a first predetermined interval, a trained detection model as the bounding box detection model; and determining, when a loss value of the second predetermined loss function is within a second predetermined interval, a trained estimation model as the gesture estimation model. ### Claim 19 A non-transitory computer-readable storage medium, having a program stored thereon and applied to a hand gesture detection apparatus, wherein the program, when executed by a processor, implements a hand gesture detection method comprising: obtaining an initial depth image comprising a hand to be detected, and performing detection processing on the initial depth image by using a backbone feature extractor and a bounding box detection model, to obtain initial bounding boxes and a first feature map corresponding to the hand to be detected; determining a target bounding box based on the initial bounding boxes, the target bounding box being one of the initial bounding boxes; cropping, based on the target bounding box, the first feature map by using an RoIAlign feature extractor, to obtain a second feature map corresponding to the hand to be detected; and performing, based on the second feature map, a three-dimensional gesture estimation processing on the hand to be detected by using a gesture estimation model to obtain a gesture detection result of the hand to be detected.