# US 10,768,708 B1 · Systems and methods of interacting with a robotic tool using free-form gestures > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,768,708 B1 - Document ID: US10768708B1 - Status: Granted - Kind: B1 - Application number: 14/833,016 - Filed: 2015-08-21 - Granted: 2020-09-08 - Earliest claimed: 2014-08-21 - Assignees: Ultrahaptics IP Two Limited - Applicants: Ultrahaptics IP Two Limited - Inventors: Maxwell Sills; Robert S. Gordon; Paul Durdik - Web page: https://patentgrep.com/patent/US10768708B1 ## CPC - B25J 13/00: Controls for manipulators (program controls B25J9/16) - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - A61B 34/74: {Manipulators with manual electric input means} - B25J 9/1612: {characterised by the hand, wrist, grip control} - B25J 9/1633: {compliant, force, torque control, e.g. combined with position control} - B25J 13/08: by means of sensing devices, e.g. viewing or touching devices - A61B 34/37: Leader-follower robots (A61B34/35 takes precedence) - A61B 2017/00207: {with hand gesture control or hand gesture recognition} - B25J 9/1697: {Vision controlled systems} - G05B 2219/35444: Gesture interface, controlled machine observes operator, executes commands - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract The technology disclosed relates to motion capture and gesture recognition. In particular, it calculates the exerted force implied by a human hand motion and applies the equivalent through a robotic arm to a target object. In one implementation, this is achieved by tracking the motion and contact of the human hand and generating corresponding robotic commands that replicate the motion and contact of the human hand on a workpiece through a robotic tool. ## Claims ### Claim 1 A method of using free-form gestures to manipulate a workpiece by a robotic tool, the method including: capturing free-form gestures of a hand in a three-dimensional (3D) sensory space utilizing two cameras to capture sequential images of the hand while empty making motions directed to commanding a robotic tool, analyzing the images as captured and translating the free-form gestures of the hand into robotic tool commands that produce smoothed emulating motions by a robotic tool, including: recognizing in the captured images of the hand while empty moving, a gesture segment within the free-form gestures of the hand that represents physical contact between the robotic tool and a workpiece; wherein there is no physical contact between the hand and any object during the gesture segment that represents physical contact; determining a command to the robotic tool to apply a force to the workpiece without actual physical contact between the robotic tool and the hand, wherein a magnitude of the force to be applied to the workpiece is determined based on a human hand motion during the gesture segment in which there is no physical contact with the hand as captured by the two cameras, without actual physical contact between the robotic tool and the hand, during the gesture segment associated with a detected path of the hand and a grasping motion of the hand; and issuing the command to the robotic tool to apply the force to the workpiece. ### Claim 2 (depends on 1) The method of claim 1, further including capturing edge information for fingers of the hand that performs the free-form gestures and computing finger positions of a 3D solid hand model for the hand during the free-form gestures. ### Claim 3 (depends on 1) The method of claim 1, further including capturing edge information for a palm of the hand that performs the free-form gestures and computing palm positions for a 3D solid hand model for the hand during the free-form gestures. ### Claim 4 (depends on 1) The method of claim 1, further including capturing finger segment length information for fingers of the hand that performs the free-form gestures and initializing a 3D solid hand model for the hand. ### Claim 5 (depends on 1) The method of claim 1, further including capturing joint angle and segment orientation information of the hand that performs the free-form gestures and applying the joint angle and segment orientation information to a 3D solid hand model for the hand during the free-form gestures. ### Claim 6 (depends on 2) The method of claim 2, further including: using the 3D solid hand model to capture a curling of the hand during the free-form gestures; and interpreting the curling as a parameter of robotic tool translation. ### Claim 7 (depends on 6) The method of claim 6, further including: using the 3D solid hand model to detect the curling as an extreme degree of motion of the hand during the free-form gestures; and responsive to detecting the curling as an extreme degree of motion of the hand, interpreting a maximum value of a parameter of robotic tool actuation. ### Claim 8 (depends on 7) The method of claim 7, wherein the maximum value of the parameter is an amplification function of the extreme degree of motion. ### Claim 9 (depends on 7) The method of claim 7, wherein the maximum value of the parameter is a polynomial function of the extreme degree of motion. ### Claim 10 (depends on 7) The method of claim 7, wherein the maximum value of the parameter is a transcendental function of the extreme degree of motion. ### Claim 11 (depends on 7) The method of claim 7, wherein the maximum value of the parameter is a step function of the extreme degree of motion. ### Claim 12 (depends on 2) The method of claim 2, further including: using the 3D solid hand model to capture a torsion of the hand during the free-form gestures; and interpreting the torsion as a parameter of robotic tool actuation. ### Claim 13 (depends on 12) The method of claim 12, further including: using the 3D solid hand model to detect the torsion as an extreme degree of motion of the hand during the free-form gestures; and responsive to detecting the torsion as the extreme degree of motion, interpreting a maximum value of a parameter of robotic tool actuation. ### Claim 14 (depends on 13) The method of claim 13, wherein the maximum value of the parameter is an amplification function of the extreme degree of motion. ### Claim 15 (depends on 13) The method of claim 13, wherein the maximum value of the parameter is a polynomial function of the extreme degree of motion. ### Claim 16 (depends on 13) The method of claim 13, wherein the maximum value of the parameter is a transcendental function of the extreme degree of motion. ### Claim 17 (depends on 13) The method of claim 13, wherein the maximum value of the parameter is a step function of the extreme degree of motion. ### Claim 18 (depends on 2) The method of claim 2, further including: using the 3D solid hand model to capture a translation of the hand during the free-form gestures; and interpreting the translation as a parameter of robotic tool actuation. ### Claim 19 (depends on 1) The method of claim 1, further comprising providing to a user haptographic feedback resulting from the performed gestures. ### Claim 20 A system including one or more processors coupled to memory storing computer instructions to use free-form gestures for manipulating a workpiece by a robotic tool, the computer instructions, when executed on the one or more processors, implement actions comprising: capturing free-form gestures of a hand in a three-dimensional (3D) sensory space utilizing two cameras to capture sequential images of the hand while empty making motions directed to commanding a robotic tool, analyzing the images as captured and translating the free-form gestures of the hand into robotic tool commands that produce smoothed emulating motions by a robotic tool, including: recognizing in the captured images of the hand while empty moving, a gesture segment within the free-form gestures of the hand that represents physical contact between the robotic tool and a workpiece; wherein there is no physical contact between the hand and any object during the gesture segment that represents physical contact; determining a command to the robotic tool to apply a force to the workpiece without actual physical contact between the robotic tool and the hand, wherein a magnitude of the force to be applied to the workpiece is determined based on a human hand motion during the gesture segment in which there is no physical contact with the hand as captured by the two cameras, without actual physical contact between the robotic tool and the hand, during the gesture segment associated with a detected path of the hand and a grasping motion of the hand; and issuing the command to the robotic tool to apply the force to the workpiece. ### Claim 21 A non-transitory computer readable medium storing a plurality of instructions for programming one or more processors to use free-form gestures for manipulating a workpiece by a robotic tool, the instructions, when executed on the processors, implementing actions including: capturing free-form gestures of a hand in a three-dimensional (3D) sensory space utilizing two cameras to capture sequential images of the hand while empty making motions directed to commanding a robotic tool, analyzing the images as captured and translating the free-form gestures of the hand into robotic tool commands that produce smoothed emulating motions by a robotic tool, including: recognizing in the captured images of the hand while empty moving, a gesture segment within the free-form gestures of the hand that represents physical contact between the robotic tool and a workpiece; wherein there is no physical contact between the empty hand and any object during the gesture segment that represents physical contact; determining a command to the robotic tool to apply a force to the workpiece without actual physical contact between the robotic tool and the hand, wherein a magnitude of the force to be applied to the workpiece is determined based on a human hand motion during the gesture segment in which there is no physical contact with the hand as captured by the two cameras, without actual physical contact between the robotic tool and the hand, during the gesture segment associated with a detected path of the hand and a grasping motion of the hand; and issuing the command to the robotic tool to apply the force to the workpiece. ### Claim 22 A method of using free-form gestures to manipulate a workpiece by a robotic tool, the method including: capturing free-form gestures of a hand in a three-dimensional (3D) sensory space while interacting with a manipulable object and translating the free-form gestures made by the hand interacting with the manipulable object into robotic tool commands that produce smoothed emulating motions by a robotic tool, including: recognizing a gesture segment within the free-form gestures of the hand that represents physical contact between the robotic tool and a workpiece; wherein there is neither physical nor electrical connection facilitating passage of a signal between the manipulable object and the robotic tool and sensor during the gesture segment that represents physical contact; determining a command to the robotic tool to apply a force to the workpiece without actual physical contact between the robotic tool and the hand, wherein a magnitude of the force to be applied to the workpiece is determined based on a human hand motion captured, without actual physical contact between the robotic tool and the hand and without physical nor electrical connection between the manipulable object and the robotic tool and sensor, during the gesture segment associated with a detected path of the hand and a grasping motion of the hand; and issuing the command to the robotic tool to apply the force to the workpiece.