# US 10,500,732 B1 · Multi-resolution localization system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,500,732 B1 - Document ID: US10500732B1 - Status: Granted - Kind: B1 - Application number: 16/050,471 - Filed: 2018-07-31 - Granted: 2019-12-10 - Earliest claimed: 2016-01-28 - Assignees: X Development LLC - Applicants: X Development LLC - Inventors: Jeff Linnell; Marek Michalowski; Anthony Sean Jules - Web page: https://patentgrep.com/patent/US10500732B1 ## CPC - B25J 9/1697: {Vision controlled systems} - B25J 9/1664: {characterised by motion, path, trajectory planning} - B25J 9/1692: {Calibration of manipulator} - G05B 2219/40601: Reference sensors - Y10S 901/09: Closed loop, sensor feedback controls arm movement - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - Y10S: TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS ## Abstract In one aspect, a method is described. The method may include providing an end effector tool of a robotic device configured to perform a task on a work surface within a worksite coordinate frame. The method may further include providing first location data indicating a first location of the end effector tool with respect to the work surface, providing second location data indicating a second location of the end effector tool within the worksite coordinate frame, and providing third location data indicating a third location of the end effector tool within the worksite coordinate frame. The method may further include tracking the location of the end effector tool based on the first, second, and third location data, and, based on the tracked location of the tool, instructing the robotic device to manipulate the end effector tool to perform a task on the work surface. ## Claims ### Claim 1 A computer-implemented method comprising: receiving, by a robot controller and from a worker robot, location data that reflects a location that the worker robot estimates for an end effector tool that is mounted on the worker robot, wherein the location data is provided to the robot controller from the worker robot at a first rate; receiving, by the robot controller and from an observer robot, location data that reflects a location that the observer robot estimates for the end effector tool that is mounted on the worker robot, wherein the location data is provided to the robot controller from the observer robot at a second rate that is slower than the first rate; and in response to receiving the location data from the worker robot and the location data from the observer robot, providing, by the robot controller and to the worker robot, location data indicating a location that the robot controller estimates for the end effector tool that is mounted on the worker robot. ### Claim 2 (depends on 1) The method of claim 1, wherein the worker robot includes a sensor that generates location data at a higher rate than a sensor on the observer robot that generates location data. ### Claim 3 (depends on 2) The method of claim 2, wherein the sensor on the observer robot has a higher spatial accuracy than the sensor on the worker robot. ### Claim 4 (depends on 1) The method of claim 1, wherein the observer robot generates the location data using dead reckoning. ### Claim 5 (depends on 1) The method of claim 1, wherein the observer robot generates the location data using optical flow. ### Claim 6 (depends on 1) The method of claim 1, wherein the location that the observer robot estimates for the end effector tool is specified in relation to a worksite coordinate frame. ### Claim 7 (depends on 1) The method of claim 1, wherein the location that the observer robot estimates for the end effector tool is specified in relation to a work surface. ### Claim 8 (depends on 1) The method of claim 1, wherein the worker robot includes visual markers for use by the observer in generating the location data that reflects the location that the observer robot estimates for the end effector tool that is mounted on the worker robot. ### Claim 9 A system comprising: one or more computers; and a non-transitory computer-readable medium coupled to the one or more computers having instructions stored thereon which, when executed by the one or more computers, cause the one or more computers to perform operations comprising: receiving, by a robot controller and from a worker robot, location data that reflects a location that the worker robot estimates for an end effector tool that is mounted on the worker robot, wherein the location data is provided to the robot controller from the worker robot at a first rate; receiving, by the robot controller and from an observer robot, location data that reflects a location that the observer robot estimates for the end effector tool that is mounted on the worker robot, wherein the location data is provided to the robot controller from the observer robot at a second rate that is slower than the first rate; and in response to receiving the location data from the worker robot and the location data from the observer robot, providing, by the robot controller and to the worker robot, location data indicating a location that the robot controller estimates for the end effector tool that is mounted on the worker robot. ### Claim 10 (depends on 9) The system of claim 9, wherein the worker robot includes a sensor that generates location data at a higher rate than a sensor on the observer robot that generates location data. ### Claim 11 (depends on 10) The system of claim 10, wherein the sensor on the observer robot has a higher spatial accuracy than the sensor on the worker robot. ### Claim 12 (depends on 9) The system of claim 9, wherein the observer robot generates the location data using dead reckoning. ### Claim 13 (depends on 9) The system of claim 9, wherein the observer robot generates the location data using optical flow. ### Claim 14 (depends on 9) The system of claim 9, wherein the location that the observer robot estimates for the end effector tool is specified in relation to a worksite coordinate frame. ### Claim 15 (depends on 9) The system of claim 9, wherein the location that the observer robot estimates for the end effector tool is specified in relation to a work surface. ### Claim 16 (depends on 9) The system of claim 9, wherein the worker robot includes visual markers for use by the observer in generating the location data that reflects the location that the observer robot estimates for the end effector tool that is mounted on the worker robot. ### Claim 17 A non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that when executed by one or more processors cause the one or more processors to perform operations comprising: receiving, by a robot controller and from a worker robot, location data that reflects a location that the worker robot estimates for an end effector tool that is mounted on the worker robot, wherein the location data is provided to the robot controller from the worker robot at a first rate; receiving, by the robot controller and from an observer robot, location data that reflects a location that the observer robot estimates for the end effector tool that is mounted on the worker robot, wherein the location data is provided to the robot controller from the observer robot at a second rate that is slower than the first rate; and in response to receiving the location data from the worker robot and the location data from the observer robot, providing, by the robot controller and to the worker robot, location data indicating a location that the robot controller estimates for the end effector tool that is mounted on the worker robot. ### Claim 18 (depends on 17) The medium of claim 17, wherein the worker robot includes a sensor that generates location data at a higher rate than a sensor on the observer robot that generates location data. ### Claim 19 (depends on 18) The medium of claim 18, wherein the sensor on the observer robot has a higher spatial accuracy than the sensor on the worker robot. ### Claim 20 (depends on 17) The medium of claim 17, wherein the observer robot generates the location data using dead reckoning.