# US 2020/0401232 A1 · Systems and methods of interacting with a robotic tool using free-form gestures > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401232 A1 - Document ID: US20200401232A1 - Status: Published application - Kind: A1 - Application number: 17/011,993 - Filed: 2020-09-03 - Published: 2020-12-24 - Earliest claimed: 2014-08-21 - Assignees: Ultrahaptics IP Two Limited - Applicants: Ultrahaptics IP Two Limited - Inventors: Maxwell Sills; Robert S. Gordon; Paul Durdik - Web page: https://patentgrep.com/patent/US20200401232A1 ## CPC - B25J 13/00: Controls for manipulators (program controls B25J9/16) - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - A61B 34/74: {Manipulators with manual electric input means} - B25J 9/1612: {characterised by the hand, wrist, grip control} - B25J 9/1633: {compliant, force, torque control, e.g. combined with position control} - B25J 13/08: by means of sensing devices, e.g. viewing or touching devices - A61B 34/37: Leader-follower robots (A61B34/35 takes precedence) - A61B 2017/00207: {with hand gesture control or hand gesture recognition} - B25J 9/1697: {Vision controlled systems} - G05B 2219/35444: Gesture interface, controlled machine observes operator, executes commands - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract The technology disclosed relates to motion capture and gesture recognition. In particular, it calculates the exerted force implied by a human hand motion and applies the equivalent through a robotic arm to a target object. In one implementation, this is achieved by tracking the motion and contact of the human hand and generating corresponding robotic commands that replicate the motion and contact of the human hand on a workpiece through a robotic tool. ## Claims ### Claim 1 A method of using gestures to control a robotic tool to manipulate a workpiece, the method including: capturing sequential images of a hand in a three-dimensional (3D) sensory space while the hand is (i) empty and (ii) making motions directed to commanding the robotic tool; recognizing, from the captured images of the hand while empty, a gesture segment of the hand that represents a manipulation of a workpiece by the robotic tool, wherein there is no physical contact between the hand and any object during the recognized gesture segment of the hand; determining, according to the recognized gesture segment, a command to cause the robotic tool to apply a force to the workpiece without actual physical contact between the robotic tool and the hand, wherein a magnitude of the force to be applied to the workpiece is determined based on a motion of the hand during the recognized gesture segment; and issuing the determined command to the robotic tool to apply the force to the workpiece. ### Claim 2 (depends on 1) The method of claim 1, further including capturing edge information for fingers of the hand that performs the gesture segment and computing finger positions of a 3D solid hand model for the hand during the gesture segment. ### Claim 3 (depends on 2) The method of claim 2, further including: using the 3D solid hand model to capture a curling of the hand during the gesture segment; and interpreting the curling as a parameter of robotic tool translation. ### Claim 4 (depends on 3) The method of claim 3, further including: using the 3D solid hand model to detect the curling as an extreme degree of motion of the hand during the gesture segment; and responsive to detecting the curling as an extreme degree of motion of the hand, interpreting a maximum value of a parameter of robotic tool actuation. ### Claim 5 (depends on 4) The method of claim 4, wherein the maximum value of the parameter is an amplification function of the extreme degree of motion. ### Claim 6 (depends on 4) The method of claim 4, wherein the maximum value of the parameter is a polynomial function of the extreme degree of motion. ### Claim 7 (depends on 4) The method of claim 4, wherein the maximum value of the parameter is a transcendental function of the extreme degree of motion. ### Claim 8 (depends on 4) The method of claim 4, wherein the maximum value of the parameter is a step function of the extreme degree of motion. ### Claim 9 (depends on 2) The method of claim 2, further including: using the 3D solid hand model to capture a torsion of the hand during the gesture segment; and interpreting the torsion as a parameter of robotic tool actuation. ### Claim 10 (depends on 9) The method of claim 9, further including: using the 3D solid hand model to detect the torsion as an extreme degree of motion of the hand during the gesture segment; and responsive to detecting the torsion as the extreme degree of motion, interpreting a maximum value of a parameter of robotic tool actuation. ### Claim 11 (depends on 10) The method of claim 10, wherein the maximum value of the parameter is an amplification function of the extreme degree of motion. ### Claim 12 (depends on 11) The method of claim 11, wherein the maximum value of the parameter is a polynomial function of the extreme degree of motion. ### Claim 13 (depends on 1) A system comprising a processor and a memory storing computer instructions that, when executed on the processor, cause the processor to perform the method of claim 1. ### Claim 14 (depends on 1) A non-transitory computer-readable recording medium having computer instructions recorded thereon, the computer instructions, when executed on a processor, causing the processor to perform the method of claim 1. ### Claim 15 A method of using gestures to control a robotic tool to manipulate a workpiece, the method including: capturing sequential images of a hand in a three-dimensional (3D) sensory space while the hand is (i) interacting with a manipulable object and (ii) making motions directed to commanding the robotic tool; recognizing, from the captured images of the hand, a gesture segment of the hand that represents physical a manipulation of a workpiece by the robotic tool, wherein there is neither physical nor electrical connection facilitating passage of a signal between the manipulable object and the robotic tool during the recognized gesture segment; determining, according to the recognized gesture segment, a command to cause the robotic tool to apply a force to the workpiece without actual physical contact between the robotic tool and the hand, wherein a magnitude of the force to be applied to the workpiece is determined based on motion of the hand during the recognized gesture segment; and issuing the determined command to the robotic tool to apply the force to the workpiece. ### Claim 16 (depends on 15) A system comprising a processor and a memory storing computer instructions that, when executed on the processor, cause the processor to perform the method of claim 15. ### Claim 17 (depends on 15) A non-transitory computer-readable recording medium having computer instructions recorded thereon, the computer instructions, when executed on a processor, causing the processor to perform the method of claim 15. ### Claim 18 A method of a robotic tool manipulating a workpiece, the method including: receiving, by the robotic tool, a command to cause the robotic tool to apply a force to the workpiece without actual physical contact between the robotic tool and a hand of a user, the received command being determined by: capturing sequential images of a hand in a three-dimensional (3D) sensory space while the hand is (i) empty and (ii) making motions directed to commanding the robotic tool; and recognizing, from the captured images of the hand while empty, a gesture segment of the hand that represents a manipulation of a workpiece by the robotic tool, wherein there is no physical contact between the hand and any object during the recognized gesture segment; and applying the force to the workpiece, according to the received command, without actual physical contact between the robotic tool and the hand, wherein a magnitude of the force applied to the workpiece is determined based on a motion of the hand during the recognized gesture segment. ### Claim 19 (depends on 18) A system comprising a processor and a memory storing computer instructions that, when executed on the processor, cause the processor to perform the method of claim 18. ### Claim 20 (depends on 18) A non-transitory computer-readable recording medium having computer instructions recorded thereon, the computer instructions, when executed on a processor, causing the processor to perform the method of claim 18.