# US 9,757,859 B1 · Tooltip stabilization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,757,859 B1 - Document ID: US9757859B1 - Status: Granted - Kind: B1 - Application number: 15/003,526 - Filed: 2016-01-21 - Granted: 2017-09-12 - Earliest claimed: 2016-01-21 - Assignees: X Development LLC - Applicants: X Development LLC - Inventors: Michael Kolb; Jeff Linnell; Anthony Sean Jules - Web page: https://patentgrep.com/patent/US9757859B1 ## CPC - B25J 9/1638: {compensation for arm bending/inertia, pay load weight/inertia} - B25J 5/007: {mounted on wheels} - B25J 9/0096: {co-operating with a working support, e.g. work-table} - B25J 9/162: {Mobile manipulator, movable base with manipulator arm mounted on it} - B25J 9/1684: {Tracking a line or surface by means of sensors} - B25J 9/1697: {Vision controlled systems} - B25J 13/089: {Determining the position of the robot with reference to its environment} - G01S 13/86: Combinations of radar systems with non-radar systems, e.g. sonar, direction finder - G01S 13/862: {Combination of radar systems with sonar systems} - G01S 13/865: {Combination of radar systems with lidar systems} - G01S 13/867: {Combination of radar systems with cameras} - G01S 13/931: of land vehicles - G01S 15/931: of land vehicles - G01S 17/06: Systems determining position data of a target - G01S 17/08: for measuring distance only (indirect measurement G01S17/46; active triangulation systems G01S17/48) - G01S 17/931: of land vehicles - G05B 2219/37423: Distance, gap between tool and surface sensor - G05B 2219/40298: Manipulator on vehicle, wheels, mobile - Y10S 901/09: Closed loop, sensor feedback controls arm movement - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - Y10S: TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS ## Abstract Robotic control systems and methods may include providing an end effector tool of a robotic device configured to perform a task on a work surface within a worksite coordinate frame. Unintended movement over time of the end effector tool with respect to the work surface and with respect to the worksite coordinate frame may be determined based on image data indicative of the work surface, first location data indicative of a first location of the end effector tool with respect to the worksite coordinate frame, and second location data indicative of a second location of the end effector tool with respect to the work surface. One or more control signals for the robotic device may be adjusted in order to counteract the unintended movements of the end effector tool with respect to the work surface and worksite coordinate frame. ## Claims ### Claim 1 A method comprising: providing an end effector tool mounted to a moveable component of a robotic device, wherein the end effector tool is configured to perform a task on a work surface in a worksite coordinate frame; receiving, from a first sensor, image data indicative of the work surface; determining, based on the image data, first movement information indicating unintended movement over time of the end effector tool with respect to the work surface; receiving, from a second sensor, location data indicating a location of the end effector tool with respect to the worksite coordinate frame; determining, based on the location data, second movement information indicating unintended movement over time of the end effector tool with respect to the worksite coordinate frame; determining, based on the first and second movement information, one or more adjustments to one or more control signals for the robotic device in order to counteract the unintended movements of the end effector tool with respect to the work surface and worksite coordinate frame; and communicating instructions to the robotic device to operate in accordance with the one or more adjustments. ### Claim 2 (depends on 1) The method of claim 1, further comprising: receiving, from a third sensor, second location data indicating a second location of the end effector tool with respect to the work surface; determining, based on the second location data, third movement information indicating unintended movement over time of the end effector tool with respect to the work surface; determining, based on the third movement information, one or more adjustments to one or more control signals for the robotic device in order to counteract the unintended movement over time of the end effector tool with respect to the work surface; and communicating instructions to the robotic device to operate in accordance with the one or more adjustments. ### Claim 3 (depends on 1) The method of claim 1, wherein the first sensor comprises an optical flow sensor. ### Claim 4 (depends on 3) The method of claim 3, further comprising receiving, from a laser sensor, distance data indicative of a distance of the work surface from the laser sensor. ### Claim 5 (depends on 4) The method of claim 4, further comprising adjusting the focal length of the optical flow sensor based on the distance data received from the laser sensor. ### Claim 6 (depends on 4) The method of claim 4, further comprising: determining, based on the distance data received from the laser sensor, fourth movement information indicating unintended movement over time of the end effector tool with respect to the work surface; determining, based on the fourth movement information, one or more adjustments to one or more control signals for the robotic device in order to counteract the unintended movement of the end effector tool; and communicating instructions to the robotic device to operate in accordance with the one or more adjustments. ### Claim 7 (depends on 1) The method of claim 1, wherein the second sensor comprises a motion capture sensor, wherein a plurality of motion capture markers are positioned in the work environment, and wherein at least one of the plurality of motion capture markers is attached to the end effector tool. ### Claim 8 (depends on 1) The method of claim 1, wherein the second sensor comprises a laser tracker configured to measure a location of one or more markers attached to the robotic device, wherein the location of the end effector tool is determined with respect to the location of the one or more markers attached to the robotic device. ### Claim 9 (depends on 8) The method of claim 8, wherein at least one of the one or more markers is attached to the end effector tool, wherein the end effector tool performing a task on the work surface interrupts a line of sight between the laser tracker and the at least one of the one or more markers attached to the end effector tool, wherein the robotic device is configured to periodically manipulate the end effector tool to restore the line of sight. ### Claim 10 (depends on 2) The method of claim 2, wherein the third sensor comprises an inertial measurement unit. ### Claim 11 A system comprising: an end effector tool mounted to a moveable component of a robotic device, wherein the end effector tool is configured to perform a task on a work surface in a worksite coordinate frame; and a control system configured to: receive, from a first sensor, image data indicative of the work surface; determine, based on the image data, first movement information indicating unintended movement over time of the end effector tool with respect to the work surface; receive, from a second sensor, location data indicating a location of the end effector tool with respect to the worksite coordinate frame; determine, based on the location data, second movement information indicating unintended movement over time of the end effector tool with respect to the worksite coordinate frame; determine, based on the first and second movement information, one or more adjustments to one or more control signals for the robotic device in order to counteract the unintended movements of the end effector tool with respect to the work surface and worksite coordinate frame; and communicate instructions to the robotic device to operate in accordance with the one or more adjustments. ### Claim 12 (depends on 11) The system of claim 11, wherein the control system is further configured to: receive, from a third sensor, second location data indicating a second location of the end effector tool with respect to the work surface; determine, based on the second location data, third movement information indicating unintended movement over time of the end effector tool with respect to the work surface; determine, based on the third movement information, one or more adjustments to one or more control signals for the robotic device in order to counteract the unintended movement over time of the end effector tool with respect to the work surface; and communicate instructions to the robotic device to operate in accordance with the one or more adjustments. ### Claim 13 (depends on 11) The system of claim 11, wherein the first sensor comprises an optical flow sensor. ### Claim 14 (depends on 13) The system of claim 13, wherein the control system is further configured to receive, from a laser sensor, distance data indicative of a distance of the work surface from the laser sensor. ### Claim 15 (depends on 14) The system of claim 14, wherein the control system is further configured to adjust the focal length of the optical flow sensor based on the distance data received from the laser sensor. ### Claim 16 (depends on 14) The system of claim 14, wherein the control system is further configured to: determine, based on the distance data received from the laser sensor, fourth movement information indicating unintended movement over time of the end effector tool with respect to the work surface; determine, based on the fourth movement information, one or more adjustments to one or more control signals for the robotic device in order to counteract the unintended movement of the end effector tool; and communicate instructions to the robotic device to operate in accordance with the one or more adjustments. ### Claim 17 (depends on 11) The system of claim 11, wherein the second sensor comprises a motion capture sensor, wherein a plurality of motion capture markers are positioned in the work environment, and wherein at least one of the plurality of motion capture markers is attached to the end effector tool. ### Claim 18 (depends on 11) The system of claim 11, wherein the second sensor comprises a laser tracker configured to measure a location of one or more markers attached to the robotic device, wherein the location of the end effector tool is determined with respect to the location of the one or more markers attached to the robotic device. ### Claim 19 (depends on 18) The system of claim 18, wherein at least one of the one or more markers is attached to the end effector tool, wherein the end effector tool performing a task on the work surface interrupts a line of sight between the laser tracker and the at least one of the one or more markers attached to the end effector tool, wherein the robotic device is configured to periodically manipulate the end effector tool to restore the line of sight. ### Claim 20 (depends on 12) The system of claim 12, wherein the third sensor comprises an inertial measurement unit.