# US 12,001,613 B2 · Methods and systems for hand gesture-based control of a device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,001,613 B2 - Document ID: US12001613B2 - Status: Granted - Kind: B2 - Application number: 17/827,939 - Filed: 2022-05-30 - Published: 2022-09-15 - Granted: 2024-06-04 - Earliest claimed: 2020-03-20 - Assignees: Huawei Technologies Co., Ltd. - Applicants: Juwei Lu; Sayem Mohammad Siam; Wei Zhou; Peng Dai; Xiaofei Wu; Songcen Xu - Inventors: Juwei Lu; Sayem Mohammad Siam; Wei Zhou; Peng Dai; Xiaofei Wu; Songcen Xu - Web page: https://patentgrep.com/patent/US12001613B2 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/82: using neural networks - G06V 40/16: Human faces, e.g. facial parts, sketches or expressions - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Methods and systems for gesture-based control of a device are described. A virtual gesture-space is determined in a received input frame. The virtual gesture-space is associated with a primary user from a ranked user list of users. The received input frame is processed in only the virtual gesture-space, to detect and track a hand. Using a hand bounding box generated by detecting and tracking the hand, gesture classification is performed to determine a gesture input associated with the hand. A command input associated with the determined gesture input is processed. The device may be a smart television, a smart phone, a tablet, etc. ## Claims ### Claim 1 A method for processing gesture input, the method comprising: determining a virtual gesture-space defined in a received input frame of a sequence of frames, the virtual gesture-space being associated with a primary user from a ranked user list of one or more users and the virtual gesture-space being smaller than a total field of view captured in the received input frame; processing only the virtual gesture-space in one or more frames in the sequence of frames to detect and track a hand; using a hand bounding box generated by detecting and tracking the hand, performing gesture classification to determine a gesture input associated with the hand; and outputting the determined gesture input to cause processing of a command input associated with the determined gesture input. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the virtual gesture space comprises: processing the input frame to detect the one or more users; generating the ranked user list based on the detected one or more users, the primary user being identified as a highest ranked user on the ranked user list; and generating the virtual gesture-space based on a detected anatomical feature of the primary user. ### Claim 3 (depends on 2) The method of claim 2, wherein processing the input frame comprises: selecting a region of interest (ROI) for processing the input frame, the ROI defining an area smaller than a total area of the input frame; wherein the ROI is selected from a defined ROI sequence, the ROI sequence defining a plurality of ROIs for processing a respective plurality of sequentially received input frames. ### Claim 4 (depends on 1) The method of claim 1, wherein determining the gesture input comprises: determining, in at least one subsequent frame, an invalid gesture input associated with the hand detected in the virtual gesture-space associated with the primary user; selecting a next highest ranked user in the ranked user list as a new primary user; and repeating the method using the new primary user. ### Claim 5 (depends on 1) The method of claim 1, wherein processing the input frame one or more frames in only the virtual gesture-space comprises: determining a low-light condition associated with at least one frame of the one or more frames; and automatically performing image adjustment to adjust pixel values of the at least one frame in response to the low-light condition. ### Claim 6 (depends on 1) The method of claim 1, wherein the one or more frames are processed using a trained joint neural network to detect and track the hand, and wherein the trained joint neural network includes a trained gesture classification convolutional neural network. ### Claim 7 (depends on 1) The method of claim 1, wherein performing gesture classification comprises: identifying a gesture class associated with the detected hand; determining a state transition from a previous gesture state to a current gesture state, the state transition being determined based on the identified gesture class; and determining the gesture input associated with the current gesture state. ### Claim 8 (depends on 1) The method of claim 1, wherein the one or more frames are received and processed at a frequency lower than a frame capture frequency of an image capturing device used to capture the sequence of frames. ### Claim 9 An apparatus comprising: a processing device coupled to a memory storing machine-executable instructions thereon, wherein the instructions, when executed by the processing device, cause the apparatus to: determining a virtual gesture-space defined in a received input frame of a sequence of frames, the virtual gesture-space being associated with a primary user from a ranked user list of one or more users and the virtual gesture-space being smaller than a total field of view captured in the received input frame; processing only the virtual gesture-space in one or more frames in the sequence of frames to detect and track a hand; using a hand bounding box generated by detecting and tracking the hand, perform gesture classification to determine a gesture input associated with the hand; wherein the determined gesture input causes processing of a command input associated with the determined gesture input. ### Claim 10 (depends on 9) The apparatus of claim 9, wherein the instructions further cause the apparatus to determine the virtual gesture space by: processing the input frame to detect the one or more users; generating the ranked user list based on the detected one or more users, the primary user being identified as a highest ranked user on the ranked user list; and generating the virtual gesture-space based on a detected anatomical feature of the primary user. ### Claim 11 (depends on 10) The apparatus of claim 10, wherein the instructions further cause the apparatus to process the input frame by: selecting a region of interest (ROI) for processing the input frame, the ROI defining an area smaller than a total area of the input frame; wherein the ROI is selected from a defined ROI sequence, the ROI sequence defining a plurality of ROIs for processing a respective plurality of sequentially received input frames. ### Claim 12 (depends on 9) The apparatus of claim 9, wherein the instructions further cause the apparatus to determine the gesture input by: determining, in at least one subsequent frame, an invalid gesture input associated with the hand detected in the virtual gesture-space associated with the primary user; selecting a next highest ranked user in the ranked user list as a new primary user; and repeating the method using the new primary user. ### Claim 13 (depends on 9) The apparatus of claim 9, wherein the instructions further cause the apparatus to process the one or more frames in only the virtual gesture-space by: determining a low-light condition associated with at least one frame of the one or more frames; and automatically performing image adjustment to adjust pixel values of the at least one frame in response to the low-light condition. ### Claim 14 (depends on 9) The apparatus of claim 9, where the one or more frames are processed using a trained joint neural network to detect and track the hand, and wherein the trained joint neural network includes a trained gesture classification convolutional neural network. ### Claim 15 (depends on 9) The apparatus of claim 9, wherein the instructions further cause the apparatus to perform gesture classification by: identifying a gesture class associated with the detected hand; determining a state transition from a previous gesture state to a current gesture state, the state transition being determined based on the identified gesture class; and determining the gesture input associated with the current gesture state. ### Claim 16 (depends on 9) The apparatus of claim 9, wherein the one or more frames are received and processed at a frequency lower than a frame capture frequency of an image capturing device used to capture the sequence of frames. ### Claim 17 (depends on 9) The apparatus of claim 9, wherein the apparatus is a gesture-controlled device. ### Claim 18 (depends on 17) The apparatus of claim 17, further comprising a camera for capturing the sequence of frames. ### Claim 19 (depends on 17) The apparatus of claim 17, wherein the gesture-controlled device is one of: a television; a smartphone; a tablet; a vehicle-coupled device; an internet of things device; an artificial reality device; or a virtual reality device. ### Claim 20 A non-transitory computer-readable medium having machine-executable instructions stored thereon, the instructions, when executed by a processing device of an apparatus, cause the apparatus to: determine a virtual gesture-space defined in a received input frame of a sequence of frames, the virtual gesture-space being associated with a primary user from a ranked user list of one or more users and the virtual gesture-space being smaller than a total field of view captured in the received input frame; process only the virtual gesture-space in one or more frames in the sequence of frames to detect and track a hand; using a hand bounding box generated by detecting and tracking the hand, perform gesture classification to determine a gesture input associated with the hand; and output the determined gesture input to cause processing of a command input associated with the determined gesture input.