# US 2024/0203104 A1 · System and Method for Learning Long-Distance Recognition and Personalization of Gestures > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0203104 A1 - Document ID: US20240203104A1 - Status: Published application - Kind: A1 - Application number: 18/068,091 - Filed: 2022-12-19 - Published: 2024-06-20 - Earliest claimed: 2022-12-19 - Assignees: Robert Bosch GmbH - Applicants: Robert Bosch GmBH - Inventors: Sharath Gopal; Shubhang Bhatnagar; Liu Ren - Web page: https://patentgrep.com/patent/US20240203104A1 ## CPC - G06V 10/776: Validation; Performance evaluation - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 10/764: using classification, e.g. of video objects - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A computer-implemented system and method relate to gesture recognition. A machine learning system is trained using a training dataset of sensor data that include a set of gestures. The training dataset includes at least a first subset that displays a first gesture. Loss data is generated based on a first loss function that includes a first cross entropy loss and a second cross entropy loss. Parameters of the machine learning system are updated based on the loss data. The machine learning system is outputted and configured for gesture recognition of the set of gestures. The machine learning system includes (i) a first subnetwork to generate feature data based on the sensor data, (ii) a second subnetwork to extract a selected patch of the feature data, and (iii) a third subnetwork to generate gesture data based on a classification of the corresponding feature data of the selected patch. The first cross entropy loss is based on a first performance of the second subnetwork in relation to the training dataset. The second cross entropy loss is based on a second performance of third subnetwork in relation to the training dataset. ## Claims ### Claim 1 A computer-implemented method for a machine learning system to learn to recognize gestures, the computer-implemented method comprising: training the machine learning system using a training dataset of sensor data that include a set of gestures, the training dataset including at least a first subset that displays a first gesture and a second subset that displays a second gesture; generating loss data based on a first loss function that includes a first cross entropy loss and a second cross entropy loss; updating parameters of the machine learning system based on the loss data; and outputting the machine learning system for gesture recognition of the set of gestures, wherein, the machine learning system includes (i) a first subnetwork to generate feature data based on the sensor data, (ii) a second subnetwork to extract a selected patch of the feature data, and (iii) a third subnetwork to generate gesture data based on a classification of the corresponding feature data of the selected patch, the first cross entropy loss is based on a first performance of the second subnetwork in relation to the training dataset, and the second cross entropy loss is based on a second performance of third subnetwork in relation to the training dataset. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the first subnetwork, the second subnetwork, and the third subnetwork form an artificial neural network model that is trained end-to-end. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: receiving additional sensor data that include samples of a new gesture, the additional sensor data being received after the machine learning system has been trained on the training dataset; training the machine learning system with the samples of the new gesture; generating additional loss data via a second loss function based on the training of the machine learning system with respect to at least the samples of the new gesture, the second loss function being different from the first loss function; and updating the parameters of the machine learning system based on the additional loss data. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein the second loss function is optimized such that the new gesture is associated with new embeddings that form a new cluster in an embedding space. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the second loss function is optimized such that the new cluster of the new gesture is spaced away from at least (i) a first cluster of embeddings of the first gesture and (ii) a second cluster of embeddings of the second gesture. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, further comprising: receiving additional sensor data that include samples of the first gesture being performed by a new gesturer; training the machine learning system using the samples to adapt the machine learning system to the new gesturer; generating additional loss data via another loss function; and updating affine parameters of the machine learning system based on the additional loss data. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, further comprising: generating, via the machine learning system, embeddings based on the samples; generating output by performing affine transformations on the embeddings using the affine parameters when the machine learning system is being trained with the samples; and generating the additional loss data based on the output, wherein the another loss function is a Shannon entropy loss function. ### Claim 8 A system for gesture recognition comprising: a processor; and a non-transitory computer readable medium in data communication with the processor, the non-transitory computer readable medium having computer readable data including instructions stored thereon that when executed by the processor is configured to cause the processor to perform a method that comprises: training a machine learning system using a training dataset of sensor data that include a set of gestures, the training dataset including at least a first subset that displays a first gesture and a second subset that displays a second gesture; generating loss data based on a first loss function that includes a first cross entropy loss and a second cross entropy loss; updating parameters of the machine learning system based on the loss data; and outputting the machine learning system for gesture recognition of the set of gestures, wherein, the machine learning system includes (i) a first subnetwork to generate feature data based on the sensor data, (ii) a second subnetwork to extract a selected patch of the feature data, and (iii) a third subnetwork to generate gesture data based on a classification of the corresponding feature data of the selected patch, the first cross entropy loss is based on a first performance of the second subnetwork in relation to the training dataset, and the second cross entropy loss is based on a second performance of third subnetwork in relation to the training dataset. ### Claim 9 (depends on 8) The system of claim 8, wherein the first subnetwork, the second subnetwork, and the third subnetwork form an artificial neural network model that is trained end-to-end. ### Claim 10 (depends on 8) The system of claim 8, wherein the method further comprises: receiving additional sensor data that include samples of a new gesture, the additional sensor data being received after the machine learning system has been trained on the training dataset: training the machine learning system with the samples of the new gesture; generating additional loss data via a second loss function based on the training of the machine learning system with respect to at least the samples of the new gesture, the second loss function being different from the first loss function; and updating the parameters of the machine learning system based on the additional loss data. ### Claim 11 (depends on 10) The system of claim 10, wherein the second loss function is optimized such that the new gesture is associated with new embeddings that form a new cluster in an embedding space. ### Claim 12 (depends on 11) The system of claim 11, wherein the second loss function is optimized such that the new cluster of the new embeddings of the new gesture is spaced away from at least (i) a first cluster of embeddings of the first gesture and (ii) a second cluster of embeddings of the second gesture. ### Claim 13 (depends on 10) The system of claim 10, further comprising: receiving additional sensor data that include samples of the first gesture being performed by a new gesturer; training the machine learning system using the samples to adapt the machine learning system to the new gesturer; generating additional loss data via another loss function; and updating affine parameters of the machine learning system based on the additional loss data. ### Claim 14 (depends on 13) The system of claim 13, further comprising: generating, via the machine learning system, embeddings based on the samples; generating output by performing affine transformations on the embeddings using the affine parameters when the machine learning system is being trained with the samples; and generating the additional loss data based on the output, wherein the another loss function is a Shannon entropy loss function. ### Claim 15 A non-transitory computer readable medium having computer readable data including instructions stored thereon that, when executed by a processor, is configured to cause the processor to perform a method that comprises: training a machine learning system using a training dataset of sensor data that include a set of gestures, the training dataset including at least a first subset that displays a first gesture; generating loss data based on a first loss function that includes a first cross entropy loss and a second cross entropy loss; updating parameters of the machine learning system based on the loss data; and outputting the machine learning system for gesture recognition of the set of gestures, wherein, the machine learning system includes (i) a first subnetwork to generate feature data based on the sensor data, (ii) a second subnetwork to extract a selected patch of the feature data, and (iii) a third subnetwork to generate gesture data based on a classification of the corresponding feature data of the selected patch, the first cross entropy loss is based on a first performance of the second subnetwork in relation to the training dataset, and the second cross entropy loss is based on a second performance of third subnetwork in relation to the training dataset. ### Claim 16 (depends on 15) The non-transitory computer readable medium of claim 15, wherein the first subnetwork, the second subnetwork, and the third subnetwork form an artificial neural network model that is trained end-to-end. ### Claim 17 (depends on 15) The non-transitory computer readable medium of claim 15, further comprising: receiving additional sensor data that include samples of a new gesture, the additional sensor data being received after the machine learning system has been trained on the training dataset; training the machine learning system with the samples of the new gesture; generating additional loss data via a second loss function based on the training of the machine learning system with respect to at least the samples of the new gesture, the second loss function being different from the first loss function; and updating the parameters of the machine learning system based on the additional loss data. ### Claim 18 (depends on 17) The non-transitory computer readable medium of claim 17, wherein the second loss function is optimized such that the new gesture is associated with new embeddings that form a new cluster in an embedding space that is spaced away from at least a first cluster of embeddings of the first gesture. ### Claim 19 (depends on 15) The non-transitory computer readable medium of claim 15, further comprising: receiving additional sensor data that include samples of the first gesture being performed by a new gesturer; training the machine learning system using the samples to adapt the machine learning system to the new gesturer; generating additional loss data via another loss function; and updating affine parameters of the machine learning system based on the additional loss data. ### Claim 20 (depends on 19) The non-transitory computer readable medium of claim 19, further comprising: generating, via the machine learning system, embeddings based on the samples; generating output by performing affine transformations on the embeddings using the affine parameters when the machine learning system is being trained with the samples; and generating the additional loss data based on the output, wherein the another loss function is Shannon entropy loss function.