# US 2024/0201788 A1 · System and Method for Long-Distance Recognition and Personalization of Gestures > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0201788 A1 - Document ID: US20240201788A1 - Status: Published application - Kind: A1 - Application number: 18/068,118 - Filed: 2022-12-19 - Published: 2024-06-20 - Earliest claimed: 2022-12-19 - Assignees: Robert Bosch GmbH - Applicants: Robert Bosch GmbH - Inventors: Sharath Gopal; Shubhang Bhatnagar; Liu Ren - Web page: https://patentgrep.com/patent/US20240201788A1 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A computer-implemented system and method relate to gesture recognition. A machine learning model includes a first subnetwork, a second subnetwork, and a third subnetwork. The first subnetwork generates feature data based on sensor data, which includes a gesture. The feature data is divided into a set of patches. The second subnetwork selects a target patch of feature data from among the set of patches. The third subnetwork generates gesture data based on the target patch of feature data. The gesture data identifies the gesture of the sensor data. Command data is generated based on the gesture data. A device is controlled based on the command data. ## Claims ### Claim 1 A computer-implemented method for gesture recognition comprising: receiving sensor data from a sensor, the sensor data including a gesture; generating, via a first subnetwork, feature data upon receiving the sensor data; dividing the feature data into a set of patches: selecting, via a second subnetwork, a target patch of feature data from among the set of patches; generating, via a third subnetwork, gesture data based on the target patch of feature data, the gesture data being indicative of the gesture of the sensor data; generating command data based on the gesture data; and controlling an actuator based on the command data, wherein a machine learning model comprises the first subnetwork, the second subnetwork, and the third subnetwork. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein: the second subnetwork includes a binary classifier, the binary classifier classifying each patch of feature data as being indicative of (i) subject data or (ii) non-subject data; the target patch is selected upon being classified as the subject data; and the third subnetwork is not applied to other patches from the set of patches in which each of the other patches is classified as the non-subject data. ### Claim 3 (depends on 2) The computer-implemented method of claim 2. wherein the subject data refers to the feature data that includes a gesturing part of the gesturer. ### Claim 4 (depends on 1) The computer-implemented method of claim 1. wherein: the first subnetwork includes a first set of artificial neural network layers, the first set of artificial neural network layers being convolutional neural network (CNN) layers; the second subnetwork includes a second set of artificial neural network layers, the second set of artificial neural network layers being CNN layers, recurrent neural network (RNN) layers, or transformer neural network; and the third subnetwork includes a third set of artificial neural network layers, the third set of artificial neural network layers including CNN layers, RNN layers, or a transformer. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein: the third subnetwork includes a classifier that classifies the target patch of feature data into a gesture class; and the third subnetwork generates the gesture data indicative of the gesture class that identifies the gesture of the sensor data. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, further comprising: generating an embedding vector upon classifying the target patch of feature data, the embedding vector being indicative of the gesture class; generating a gesture prediction vector based on the embedding vector; and generating the gesture data based on the gesture prediction vector. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the sensor data includes (a) digital image data, (b) digital video data. (c) digital image data and depth data, or (d) digital video data and depth data. ### Claim 8 A system for gesture recognition, the system comprising: a processor; and a non-transitory computer readable medium in data communication with the processor. the non-transitory computer readable medium having computer readable data including instructions stored thereon that when executed by the processor is configured to cause the processor to perform a method that comprises: receiving sensor data from a sensor, the sensor data including a gesture; generating, via a first subnetwork, feature data upon receiving the sensor data; dividing the feature data into a set of patches; selecting, via a second subnetwork, a target patch of feature data from among the set of patches; generating, via a third subnetwork, gesture data by classifying the feature data of the target patch, the gesture data being indicative of the gesture of the sensor data; generating command data based on the gesture data; and controlling an actuator based on the command data, wherein a machine learning model comprises the first subnetwork, the second subnetwork, and the third subnetwork. ### Claim 9 (depends on 8) The system of claim 8, wherein: the second subnetwork includes a binary classifier, the binary classifier classifying each patch of feature data as being indicative of (i) subject data or (ii) non-subject data; the target patch is selected upon being classified as the subject data; and the third subnetwork is not applied to other patches from the set of patches in which each of the other patches is classified as the non-subject data. ### Claim 10 (depends on 9) The system of claim 9, wherein the subject data refers to the feature data that includes a gesturing part of the gesturer. ### Claim 11 (depends on 8) The system of claim 8, further comprising: the first subnetwork includes a first set of artificial neural network layers, the first set of artificial neural network layers being convolutional neural network (CNN) layers; the second subnetwork includes a second set of artificial neural network layers, the second set of artificial neural network layers being CNN layers, recurrent neural network (RNN) layers, or a transformer; and the third subnetwork includes a third set of artificial neural network layers, the third set of artificial neural network layers including CNN layers, RNN layers, or a transformer. ### Claim 12 (depends on 8) The system of claim 8, wherein: the third subnetwork includes a classifier that classifies the target patch of feature data into a gesture class; and the third subnetwork generates the gesture data indicative of the gesture class that identifies the gesture. ### Claim 13 (depends on 12) The system of claim 12, further comprising: generating an embedding vector upon classifying the target patch of feature data, the embedding vector being indicative of the gesture class; generating a gesture prediction vector based on the embedding vector; and generating the gesture data based on the gesture prediction vector. ### Claim 14 (depends on 8) The system of claim 8, wherein the sensor data includes (a) digital image data, (b) digital video data. (c) digital image data and depth data, or (d) digital video data and depth data. ### Claim 15 A non-transitory computer readable medium having computer readable data including instructions stored thereon that when executed by a processor is configured to cause the processor to perform a method that comprises: receiving sensor data from a sensor, the sensor data including a gesture; generating, via a first subnetwork, feature data upon receiving the sensor data; dividing the feature data into a set of patches; selecting, via a second subnetwork, a target patch of feature data from among the set of patches: generating, via a third subnetwork, gesture data by classifying the feature data of the target patch, the gesture data being indicative of the gesture of the sensor data; generating command data based on the gesture data; and controlling an actuator based on the command data, wherein a machine learning model comprises the first subnetwork, the second subnetwork, and the third subnetwork. ### Claim 16 (depends on 15) The non-transitory computer readable medium of claim 15. wherein: the second subnetwork includes a binary classifier, the binary classifier classifying each patch of feature data as being indicative of (i) subject data or (ii) non-subject data; the target patch is selected upon being classified as the subject data; and the third subnetwork is not applied to other patches from the set of patches in which each of the other patches is classified as the non-subject data. ### Claim 17 (depends on 16) The non-transitory computer readable medium of claim 16, wherein the subject data refers to the feature data that includes a gesturing part of the gesturer. ### Claim 18 (depends on 15) The non-transitory computer readable medium of claim 15, wherein: the first subnetwork includes a first set of artificial neural network layers, the first set of artificial neural network layers being convolutional neural network (CNN) layers; the second subnetwork includes a second set of artificial neural network layers, the second set of artificial neural network layers being CNN layers, recurrent neural network (RNN) layers, or a transformer neural network; and the third subnetwork includes a third set of artificial neural network layers, the third set of artificial neural network layers including CNN layers, RNN layers, or a transformer neural network. ### Claim 19 (depends on 15) The non-transitory computer readable medium of claim 15, wherein: the third subnetwork includes a classifier that classifies the target patch of feature data into a gesture class; and the third subnetwork generates the gesture data indicative of the gesture class that identifies the gesture. ### Claim 20 (depends on 15) The non-transitory computer readable medium of claim 15, wherein the sensor data includes (a) digital image data, (b) digital video data, (c) digital image data and depth data, or (d) digital video data and depth data.