# US 10,800,036 B1 · Tooltip stabilization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,800,036 B1 - Document ID: US10800036B1 - Status: Granted - Kind: B1 - Application number: 16/798,938 - Filed: 2020-02-24 - Granted: 2020-10-13 - Earliest claimed: 2016-01-21 - Assignees: X Development LLC - Applicants: X Development LLC - Inventors: Michael Kolb; Jeff Linnell; Anthony Sean Jules - Web page: https://patentgrep.com/patent/US10800036B1 ## CPC - B25J 9/1638: {compensation for arm bending/inertia, pay load weight/inertia} - B25J 5/007: {mounted on wheels} - B25J 9/0096: {co-operating with a working support, e.g. work-table} - B25J 9/162: {Mobile manipulator, movable base with manipulator arm mounted on it} - B25J 9/1684: {Tracking a line or surface by means of sensors} - B25J 9/1697: {Vision controlled systems} - B25J 13/089: {Determining the position of the robot with reference to its environment} - G01S 13/86: Combinations of radar systems with non-radar systems, e.g. sonar, direction finder - G01S 13/862: {Combination of radar systems with sonar systems} - G01S 13/865: {Combination of radar systems with lidar systems} - G01S 13/867: {Combination of radar systems with cameras} - G01S 13/931: of land vehicles - G01S 15/931: of land vehicles - G01S 17/06: Systems determining position data of a target - G01S 17/08: for measuring distance only (indirect measurement G01S17/46; active triangulation systems G01S17/48) - G01S 17/931: of land vehicles - G05B 2219/37423: Distance, gap between tool and surface sensor - G05B 2219/40298: Manipulator on vehicle, wheels, mobile - Y10S 901/09: Closed loop, sensor feedback controls arm movement - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - Y10S: TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS ## Abstract Robotic control systems and methods may include providing an end effector tool of a robotic device configured to perform a task on a work surface within a worksite coordinate frame. Unintended movement over time of the end effector tool with respect to the work surface and with respect to the worksite coordinate frame may be determined based on image data indicative of the work surface, first location data indicative of a first location of the end effector tool with respect to the worksite coordinate frame, and second location data indicative of a second location of the end effector tool with respect to the work surface. One or more control signals for the robotic device may be adjusted in order to counteract the unintended movements of the end effector tool with respect to the work surface and worksite coordinate frame. ## Claims ### Claim 1 A computer-implemented method comprising: receiving, from a robotic device that includes a sensor and an end effector tool, a location that is estimated for the end effector tool using sensor data that is generated by the sensor included on the robotic device; receiving, from a global sensor that is not included on the robotic device, a location that is estimated for the end effector tool using sensor data that is generated by the global sensor; determining an adjustment to apply to the end effector tool using (i) the location that is estimated for the end effector tool using the sensor data that is generated by the sensor included on the robotic device, and (ii) the location that is estimated for the end effector tool using the sensor data that is generated by the global sensor; and providing, to the robotic device, an instruction to apply the adjustment to the end effector tool. ### Claim 2 (depends on 1) The method of claim 1, wherein the sensor included on the robotic device has a higher refresh rate and a lower spatial accuracy than the global sensor. ### Claim 3 (depends on 1) The method of claim 1, wherein the location that is estimated for the end effector tool using the sensor data that is generated by the sensor included on the robotic device is estimated using dead reckoning. ### Claim 4 (depends on 1) The method of claim 1, wherein the location that is estimated for the end effector tool using the sensor data that is generated by the sensor included on the robotic device is estimated using optical flow. ### Claim 5 (depends on 1) The method of claim 1, wherein the location that is estimated for the end effector tool using sensor data that is generated by the global sensor comprises a location of the end effector tool with respect to a worksite coordinate frame. ### Claim 6 (depends on 1) The method of claim 1, wherein the location that is estimated for the end effector tool using the sensor data that is generated by the sensor included on the robotic device comprises a location of the end effector tool with respect to a work surface. ### Claim 7 (depends on 1) The method of claim 1, wherein the global sensor comprises a camera. ### Claim 8 (depends on 7) The method of claim 7, wherein the global sensor comprises a laser tracker. ### Claim 9 (depends on 1) The method of claim 1, wherein the robotic device further includes markers for use by the global sensor in generating the sensor data. ### Claim 10 A system comprising: one or more computers; and a non-transitory computer-readable medium coupled to the one or more computers having instructions stored thereon which, when executed by the one or more computers, cause the one or more computers to perform operations comprising: receiving, from a robotic device that includes a sensor and an end effector tool, a location that is estimated for the end effector tool using sensor data that is generated by the sensor included on the robotic device; receiving, from a global sensor that is not included on the robotic device, a location that is estimated for the end effector tool using sensor data that is generated by the global sensor; determining an adjustment to apply to the end effector tool using (i) the location that is estimated for the end effector tool using the sensor data that is generated by the sensor included on the robotic device, and (ii) the location that is estimated for the end effector tool using the sensor data that is generated by the global sensor; and providing, to the robotic device, an instruction to apply the adjustment to the end effector tool. ### Claim 11 (depends on 10) The system of claim 10, wherein the sensor included on the robotic device has a higher refresh rate and a lower spatial accuracy than the global sensor. ### Claim 12 (depends on 10) The system of claim 10, wherein the location that is estimated for the end effector tool using the sensor data that is generated by the sensor included on the robotic device is estimated using dead reckoning. ### Claim 13 (depends on 10) The system of claim 10, wherein the location that is estimated for the end effector tool using the sensor data that is generated by the sensor included on the robotic device is estimated using optical flow. ### Claim 14 (depends on 10) The system of claim 10, wherein the location that is estimated for the end effector tool using sensor data that is generated by the global sensor comprises a location of the end effector tool with respect to a worksite coordinate frame. ### Claim 15 (depends on 10) The system of claim 10, wherein the location that is estimated for the end effector tool using the sensor data that is generated by the sensor included on the robotic device comprises a location of the end effector tool with respect to a work surface. ### Claim 16 (depends on 10) The system of claim 10, wherein the global sensor comprises a camera. ### Claim 17 (depends on 16) The system of claim 16, wherein the global sensor comprises a laser tracker. ### Claim 18 (depends on 1) The system of claim 1, wherein the robotic device further includes markers for use by the global sensor in generating the sensor data. ### Claim 19 A non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that when executed by one or more processors cause the one or more processors to perform operations comprising: receiving, from a robotic device that includes a sensor and an end effector tool, a location that is estimated for the end effector tool using sensor data that is generated by the sensor included on the robotic device; receiving, from a global sensor that is not included on the robotic device, a location that is estimated for the end effector tool using sensor data that is generated by the global sensor; determining an adjustment to apply to the end effector tool using (i) the location that is estimated for the end effector tool using the sensor data that is generated by the sensor included on the robotic device, and (ii) the location that is estimated for the end effector tool using the sensor data that is generated by the global sensor; and providing, to the robotic device, an instruction to apply the adjustment to the end effector tool. ### Claim 20 (depends on 19) The medium of claim 19, wherein the sensor included on the robotic device has a higher refresh rate and a lower spatial accuracy than the global sensor.