# US 12,118,443 B2 · Machine-learning based gesture recognition using multiple sensors > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,443 B2 - Document ID: US12118443B2 - Status: Granted - Kind: B2 - Application number: 18/202,857 - Filed: 2023-05-26 - Published: 2023-10-12 - Granted: 2024-10-15 - Earliest claimed: 2019-11-08 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Charles Maalouf; Shawn R. Scully; Christopher B. Fleizach; Tu K. Nguyen; Lilian H. Liang; Warren J. Seto; Julian Quintana; Michael J. Beyhs; Hojjat Seyed Mousavi; Behrooz Shahsavari - Web page: https://patentgrep.com/patent/US12118443B2 ## CPC - G06N 20/00: Machine learning - G06F 1/163: {Wearable computers, e.g. on a belt} - G06F 1/169: {the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes (G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033)} - G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling} - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06N 3/098: Distributed learning, e.g. federated learning - G06V 10/82: using neural networks - G06V 40/15: {Biometric patterns based on physiological signals, e.g. heartbeat, blood flow} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06N 3/08: Learning methods - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A device implementing a system for machine-learning based gesture recognition includes at least one processor configured to, receive, from a first sensor of the device, first sensor output of a first type, and receive, from a second sensor of the device, second sensor output of a second type that differs from the first type. The at least one processor is further configured to provide the first sensor output and the second sensor output as inputs to a machine learning model, the machine learning model having been trained to output a predicted gesture based on sensor output of the first type and sensor output of the second type. The at least one processor is further configured to determine the predicted gesture based on an output from the machine learning model, and to perform, in response to determining the predicted gesture, a predetermined action on the device. ## Claims ### Claim 1 A method, comprising: receiving sensor data from a sensor of a device; providing the sensor data as input to a machine learning model, the machine learning model having been trained to output, for each of a plurality of prediction windows that each include a portion of the sensor data, a gesture label and a no-gesture label; determining a predicted gesture based on an output from the machine learning model; and performing, in response to determining the predicted gesture, a predetermined action on the device. ### Claim 2 (depends on 1) The method of claim 1, wherein, for each of the plurality of prediction windows, the gesture label and the no-gesture label each comprise a smoothed gesture label having a respective value between a maximum value for the smoothed gesture label and a minimum value for the smoothed gesture label. ### Claim 3 (depends on 2) The method of claim 2, wherein the respective value of the smoothed gesture label is a value on a linear curve, a sigmoid curve, or an exponential curve between the minimum value and the maximum value of the smoothed gesture label. ### Claim 4 (depends on 1) The method of claim 1, wherein, for each of the plurality of prediction windows, the gesture label comprises a probability that a gesture is occurring in that prediction window, and the no-gesture label comprises a probability that no gesture is occurring in that window. ### Claim 5 (depends on 4) The method of claim 4, wherein, for each of the plurality of prediction windows, the output of the machine learning model comprises the gesture label, the no-gesture label, and a prediction of an amount of the predicted gesture that overlaps with a current portion of the sensor data. ### Claim 6 (depends on 4) The method of claim 4, wherein determining the predicted gesture based on the output from the machine learning model comprises determining the predicted gesture based on a gesture predicted for the portion of the sensor data corresponding to the prediction window having a highest gesture label and a lowest no-gesture label. ### Claim 7 (depends on 4) The method of claim 4, wherein determining the predicted gesture based on the output from the machine learning model comprises: determining, based on the gesture label and the no-gesture label for each of the plurality of prediction windows, a start time and an end time of the predicted gesture; and performing a gesture prediction operation for a set of the sensor data between the start time and the end time. ### Claim 8 (depends on 1) The method of claim 1, wherein the predicted gesture comprises one of: a finger-based gesture or a wrist-based gesture. ### Claim 9 (depends on 1) The method of claim 1, wherein performing the predetermined action on the device comprises providing a user input based on the predicted gesture to an application running at the device, the application including one of: a media player application, a workout application, an augmented reality application, a virtual reality application, or a health related application. ### Claim 10 A device, comprising: a sensor; and at least one processor configured to: receive sensor data from the sensor; provide the sensor data as input to a machine learning model, the machine learning model having been trained to output, for each of a plurality of prediction windows that each include a portion of the sensor data, a gesture label and a no-gesture label; determine a predicted gesture based on an output from the machine learning model; and perform, in response to determining the predicted gesture, a predetermined action on the device. ### Claim 11 (depends on 10) The device of claim 10, wherein, for each of the plurality of prediction windows, the gesture label and the no-gesture label each comprise a smoothed gesture label having a respective value between a maximum value for the smoothed gesture label and a minimum value for the smoothed gesture label. ### Claim 12 (depends on 11) The device of claim 11, wherein the respective value of the smoothed gesture label is a value on a linear curve, a sigmoid curve, or an exponential curve between the minimum value and the maximum value of the smoothed gesture label. ### Claim 13 (depends on 10) The device of claim 10, wherein, for each of the plurality of prediction windows, the gesture label comprises a probability that a gesture is occurring in that prediction window, and the no-gesture label comprises a probability that no gesture is occurring in that window. ### Claim 14 (depends on 13) The device of claim 13, wherein, for each of the plurality of prediction windows, the output of the machine learning model comprises the gesture label, the no-gesture label, and a prediction of an amount of the predicted gesture that overlaps with a current portion of the sensor data. ### Claim 15 (depends on 13) The device of claim 13, wherein the at least one processor is configured to determine the predicted gesture based on the output from the machine learning model based on a gesture predicted for the portion of the sensor data corresponding to the prediction window having a highest gesture label and a lowest no-gesture label. ### Claim 16 (depends on 15) The device of claim 15 wherein the at least one processor is configured to determine the predicted gesture by: determining, based on the gesture label and the no-gesture label for each of the plurality of prediction windows, a start time and an end time of the predicted gesture; and performing a gesture prediction operation for a set of the sensor data between the start time and the end time. ### Claim 17 A non-transitory computer readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving sensor data from a sensor of a device; providing the sensor data as input to a machine learning model, the machine learning model having been trained to output, for each of a plurality of prediction windows that each include a portion of the sensor data, a gesture label and a no-gesture label; determining a predicted gesture based on an output from the machine learning model; and performing, in response to determining the predicted gesture, a predetermined action on the device. ### Claim 18 (depends on 17) The non-transitory computer readable medium of claim 17, wherein, for each of the plurality of prediction windows, the gesture label and the no-gesture label each comprise a smoothed gesture label having a respective value between a maximum value for the smoothed gesture label and a minimum value for the smoothed gesture label. ### Claim 19 (depends on 18) The non-transitory computer readable medium of claim 18, wherein the respective value of the smoothed gesture label is a value on a linear curve, a sigmoid curve, or an exponential curve between the minimum value and the maximum value of the smoothed gesture label. ### Claim 20 (depends on 17) The non-transitory computer readable medium of claim 17, wherein, for each of the plurality of prediction windows, the gesture label comprises a probability that a gesture is occurring in that prediction window, and the no-gesture label comprises a probability that no gesture is occurring in that window.