# US 12,216,832 B2 · Gesture recognition based on likelihood of interaction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,216,832 B2 - Document ID: US12216832B2 - Status: Granted - Kind: B2 - Application number: 18/463,906 - Filed: 2023-09-08 - Published: 2023-12-28 - Granted: 2025-02-04 - Earliest claimed: 2022-02-01 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Julia Schwarz; Bugra Tekin; Sophie Stellmach; Erian Vazquez; Casey Leon Meekhof; Fabian Gobel - Web page: https://patentgrep.com/patent/US12216832B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06N 3/08: Learning methods - G06V 10/82: using neural networks - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method for evaluating gesture input comprises receiving input data for sequential data frames, including hand tracking data for hands of a user. A first neural network is trained to recognize features indicative of subsequent gesture interactions and configured to evaluate input data for a sequence of data frames and to output an indication of a likelihood of the user performing gesture interactions during a predetermined window of data frames. A second neural network is trained to recognize features indicative of whether the user is currently performing one or more gesture interactions and configured to adjust parameters for gesture interaction recognition during the predetermined window based on the indicated likelihood. The second neural network evaluates the predetermined window for performed gesture interactions based on the adjusted parameters, and outputs a signal as to whether the user is performing one or more gesture interactions during the predetermined window. ## Claims ### Claim 1 A method for evaluating gesture input, comprising: receiving input data for sequential data frames, the input data including at least hand tracking data for one or more hands of a user; at a first neural network trained to recognize features indicative of a plurality of subsequent gesture interactions: evaluating input data for a sequence of two or more data frames; and outputting an indication of a likelihood of the user performing one or more subsequent gesture interactions with a user interface during a predetermined window of one or more data frames; and at a second neural network trained to recognize features indicative of whether the user is currently performing one or more of the plurality of subsequent gesture interactions: adjusting parameters for gesture interaction recognition during the predetermined window based on the indicated likelihood of the user performing one or more of the plurality of subsequent gesture interactions and based at least on one or more heuristic evaluations of the received input data for the predetermined window; evaluating at least the predetermined window for performed gesture interactions based on the adjusted parameters; and outputting a signal as to whether the user is performing one or more gesture interactions with the user interface during the predetermined window. ### Claim 2 (depends on 1) The method of claim 1, further comprising: adjusting parameters of the first neural network based on one or more heuristic evaluations of the received input data. ### Claim 3 (depends on 1) The method of claim 1, wherein the heuristic evaluations include heuristic evaluations of hand tracking data for one or more hands of the user and an extracted surface mesh for an environment to determine a distance between a hand of the user and a nearest surface. ### Claim 4 (depends on 3) The method of claim 3, wherein the heuristic evaluations further include an indication as to whether the hand of the user is within a threshold distance from the nearest surface. ### Claim 5 (depends on 1) The method of claim 1, wherein the heuristic evaluations further include a binary indication as to whether a hand of the user is holding an object. ### Claim 6 (depends on 5) The method of claim 5, wherein determining the binary indication as to whether the hand of the user is holding an object comprises: extracting a first surface mesh for a region of interest including the hand of the user and any associated objects from a depth image of an environment; generating a second surface mesh for the hand of the user based on the hand tracking data; and subtracting the second surface mesh from the first surface mesh to generate an object surface mesh. ### Claim 7 (depends on 6) The method of claim 6, wherein an indication as to whether a total number of voxels of the object surface mesh is less than a threshold is provided as an input to the first neural network. ### Claim 8 (depends on 1) The method of claim 1, wherein the heuristic evaluations comprise: evaluating hand tracking data for one or more hands of the user for two or more feature points of the hand of the user; evaluating the velocity of each feature point in an environment; evaluating distances between feature points in the environment; and tracking relative velocities and distances between feature points in the environment over one or more data frames. ### Claim 9 (depends on 8) The method of claim 8, wherein the heuristic evaluations comprise evaluating a likelihood of the hand of the user being in a conformation to provide gesture input. ### Claim 10 (depends on 8) The method of claim 8, wherein the heuristic evaluations comprise evaluating a likelihood that the hand of the user is engaged in an activity. ### Claim 11 A system for a head-mounted display, comprising: a display device configured to present a user interface to a user wearing the head-mounted display; a sensor suite including at least one or more outward-facing image sensors; one or more data pre-processing machines configured to: receive data output from the sensor suite and to sequentially output data frames; a first neural network trained to: receive input data for sequential data frames, the input data including at least hand tracking data for one or more hands of a user; and evaluate input data for a sequence of two or more data frames; and output an indication of a likelihood of the user performing one or more subsequent gesture interactions with a user interface during a predetermined window of one or more data frames; and a second neural network trained to: recognize features indicative of whether the user is currently performing one or more of the plurality of subsequent gesture interactions; adjust parameters for gesture interaction recognition during the predetermined window based on the indicated likelihood of the user performing one or more of the plurality of subsequent gesture interactions and based at least on one or more heuristic evaluations of the received input data for the predetermined window; evaluate at least the predetermined window for performed gesture interactions based on the adjusted parameters; and output a signal as to whether the user is performing one or more gesture interactions with the user interface during the predetermined window. ### Claim 12 (depends on 11) The system for a head-mounted display of claim 11, wherein the parameters of the first neural network are adjusted based on one or more heuristic evaluations of the received input data. ### Claim 13 (depends on 11) The system for a head-mounted display of claim 11, wherein the heuristic evaluations include heuristic evaluations of hand tracking data for one or more hands of the user and an extracted surface mesh for an environment to determine a distance between a hand of the user and a nearest surface. ### Claim 14 (depends on 11) The system for a head-mounted display of claim 11, wherein the heuristic evaluations further include a binary indication as to whether a hand of the user is holding an object. ### Claim 15 (depends on 11) The system for a head-mounted display of claim 11, wherein the heuristic evaluations comprise: evaluating hand tracking data for one or more hands of the user for two or more feature points of the hand of the user; evaluating the velocity of each feature point in an environment; evaluating distances between feature points in the environment; and tracking relative velocities and distances between feature points in the environment over one or more data frames. ### Claim 16 A computer-readable storage machine comprising computer-executable instructions that, when executed, cause one or more logic subsystems of a computing device to: receive input data for sequential data frames, the input data including at least hand tracking data for one or more hands of a user; at a first neural network trained to recognize features indicative of a plurality of subsequent gesture interactions: evaluate input data for a sequence of two or more data frames; and output an indication of a likelihood of the user performing one or more subsequent gesture interactions with a user interface during a predetermined window of one or more data frames; and at a second neural network trained to recognize features indicative of whether the user is currently performing one or more of the plurality of subsequent gesture interactions: adjust parameters for gesture interaction recognition during the predetermined window based on the indicated likelihood of the user performing one or more of the plurality of subsequent gesture interactions and based at least on one or more heuristic evaluations of the received input data for the predetermined window; evaluate at least the predetermined window for performed gesture interactions based on the adjusted parameters; and output a signal as to whether the user is performing one or more gesture interactions with the user interface during the predetermined window. ### Claim 17 (depends on 16) The computer-readable storage machine of claim 16, where the instructions are further executable to adjust the parameters of the first neural network based on one or more heuristic evaluations of the received input data. ### Claim 18 (depends on 16) The computer-readable storage machine of claim 16, where the instructions are further executable to perform heuristic evaluations of hand tracking data for one or more hands of the user and an extracted surface mesh for an environment to determine a distance between a hand of the user and a nearest surface. ### Claim 19 (depends on 16) The computer-readable storage machine of claim 16, where the instructions are further executable to perform heuristic evaluations that further include a binary indication as to whether a hand of the user is holding an object. ### Claim 20 (depends on 16) The computer-readable storage machine of claim 16, where the instructions are further executable to: evaluate hand tracking data for one or more hands of the user for two or more feature points of the hand of the user; evaluate the velocity of each feature point in an environment; evaluate distances between feature points in the environment; and track relative velocities and distances between feature points in the environment over one or more data frames.