# US 2024/0201794 A1 · Systems and methods of tracking moving hands and recognizing gestural interactions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0201794 A1 - Document ID: US20240201794A1 - Status: Published application - Kind: A1 - Application number: 18/587,257 - Filed: 2024-02-26 - Published: 2024-06-20 - Earliest claimed: 2014-05-14 - Assignees: Ultrahaptics IP Two Limited - Applicants: Ultrahaptics IP Two Limited - Inventors: Kevin A. HOROWITZ; Matias PEREZ; Raffi BEDIKIAN; David S. HOLZ; Gabriel A. HARE - Web page: https://patentgrep.com/patent/US20240201794A1 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G01S 3/00: Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic or electromagnetic waves, or particle emission, not having a directional significance, are being received (position-fixing by co-ordinating a plurality of determinations of direction or position lines G01S5/00) - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 7/251: {involving models} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 40/113: {Recognition of static hand signs} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - H04N 13/296: Synchronisation thereof; Control thereof - G06T 2207/10016: Video; Image sequence - G06T 2207/30196: Human being; Person - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract The technology disclosed relates to relates to providing command input to a machine under control. It further relates to gesturally interacting with the machine. The technology disclosed also relates to providing monitoring information about a process under control. The technology disclosed further relates to providing biometric information about an individual. The technology disclosed yet further relates to providing abstract features information (pose, grab strength, pinch strength, confidence, and so forth) about an individual. ## Claims ### Claim 1 A method of determining a gesture performed by the at least a portion of a hand in three dimensional (3D) sensory space, the method comprising: determining a variance between a point on a set of observation information based on a first image and a corresponding point on at least one of a set of 3D capsules fitted to another set of observation information based on a second image by: pairing point sets from points on a surface of the observation information with points on the 3D capsules or axes of the 3D capsules, wherein normal vectors to points on the set of observation information are parallel to normal vectors to points on the 3D capsules; and determining the variance comprising a reduced root mean squared deviation (RMSD) of distances between paired point sets; and determining a gesture performed by the at least a portion of a hand based on the determined variance; and providing the gesture determined. ### Claim 2 (depends on 1) The method of claim 1, wherein adjusting the 3D capsules further includes improving conformance of the 3D capsules to at least one of length, width, orientation, and arrangement of portions of the observation information. ### Claim 3 (depends on 1) The method of claim 1, further including: determining span modes of the hand, wherein the span modes include at least a finger width span mode and a palm width span mode; and using span width parameters for the finger width span mode and the palm width span mode to initialize 3D capsules of a 3D model of the hand. ### Claim 4 (depends on 1) The method of claim 1, further including: determining span modes of the hand, wherein the span modes include at least a finger width span mode, a palm width span mode, and a wrist width span mode; and using span width parameters for the finger width span mode, palm width span mode, and wrist width span mode to initialize a 3D model of the hand and corresponding arm. ### Claim 5 (depends on 1) The method of claim 1, further including interpreting the gesture as selecting one or more heterogeneous devices. ### Claim 6 (depends on 1) The method of claim 1, further including interpreting the gesture as selecting one or more heterogeneous marker images that trigger augmented illusions. ### Claim 7 (depends on 1) The method of claim 1, further including interpreting the gesture and automatically switching a machine under control from one operational mode to another in response. ### Claim 8 (depends on 1) The method of claim 1, wherein determining the variance further includes determining whether the point on another set of observation information based on the first image and the corresponding point on one of the 3D capsules fitted to the observation information defined based on the second image are within a threshold closest distance. ### Claim 9 (depends on 1) The method of claim 1, wherein determining the variance further includes: pairing point sets on an observation information of the at least a portion of a hand with points on axes of the 3D capsules, wherein points on observation information lie on vectors that are normal to points on axes; and determining a reduced root mean squared deviation (RMSD) of distances between paired point sets. ### Claim 10 (depends on 1) The method of claim 1, further including determining at least one of a velocity of a portion of a hand, a state, a pose. ### Claim 11 (depends on 10) The method of claim 10, wherein the determining a velocity further includes determining at least one of a velocity of one or more fingers, and a relative motion of a portion of the hand. ### Claim 12 (depends on 10) The method of claim 10, wherein the determining a state further includes determining at least one of a position, an orientation, and a location of a portion of the hand. ### Claim 13 (depends on 10) The method of claim 10, wherein the determining a pose further includes determining at least one of whether one or more fingers are extended or non-extended, one or more angles of bend for one or more fingers, a direction to which one or more fingers point, a configuration indicating a pinch, a grab, an outside pinch, and a pointing finger. ### Claim 14 (depends on 10) The method of claim 10, further including determining whether a tool or object is present in the hand. ### Claim 15 (depends on 1) The method of claim 1, further comprising: determining gesture features for the at least a portion of a hand based on the 3D capsules; and issuing a feature-specific command input to a machine under control based on the gesture features. ### Claim 16 (depends on 15) The method of claim 15, wherein the gesture features include edge information for at least one of fingers of the hand and palm of the hand. ### Claim 17 (depends on 15) The method of claim 15, wherein gesture features include at least one of joint angle and segment orientation information of the hand, and finger segment length information for fingers of the hand. ### Claim 18 (depends on 15) The method of claim 15, wherein the gesture features include at least one of curling of the hand during gestural motion and a pose, a grab strength, a pinch strength and a confidence of the hand. ### Claim 19 (depends on 1) The method of claim 1, further comprising: determining biometric features for the at least a portion of a hand based on the 3D capsules; authenticating the at least a portion of a hand based on the biometric features determined; determining a command input indicated by gestural motion of the at least a portion of a hand, determining whether the at least a portion of a hand is authorized to issue the command input; and issuing an authorized command input to a machine under control. ### Claim 20 (depends on 19) The method of claim 19, wherein the biometric features determined include at least one of measurements across a palm of the hand and finger width at a first knuckle of the hand. ### Claim 21 A non-transitory computer readable storage medium impressed with computer program instructions to determine a gesture performed by the at least a portion of a hand in three dimensional (3D) sensory space, which instructions, when executed on a processor, implement actions comprising: determining a variance between a point on a set of observation information based on a first image and a corresponding point on at least one of a set of 3D capsules fitted to another set of observation information based on a second image by: pairing point sets from points on a surface of the observation information with points on the 3D capsules or axes of the 3D capsules, wherein normal vectors to points on the set of observation information are parallel to normal vectors to points on the 3D capsules; and determining the variance comprising a reduced root mean squared deviation (RMSD) of distances between paired point sets; and determining a gesture performed by the at least a portion of a hand based on the determined variance; and providing as an output, the gesture as determined. ### Claim 22 A system to determine a gesture performed by the at least a portion of a hand in three dimensional (3D) sensory space, comprising: a processor and a computer readable storage medium storing computer instructions configured to cause the processor to: determine a variance between a point on a surface of another set of observation information based on a first image and a corresponding point on at least one of a set of 3D capsules fitted to another set of observation information based on a second image by: pairing point sets from points on a surface of the observation information with points on the 3D capsules or axes of the 3D capsules, wherein normal vectors to points on the set of observation information are parallel to normal vectors to points on the 3D capsules; and determining the variance comprising reduced root mean squared deviation (RMSD) of distances between paired point sets; and determine a gesture performed based on the determined variance; and provide as an output, the gesture as determined.