# US 11,999,066 B2 · Robotics calibration in a lab environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,999,066 B2 - Document ID: US11999066B2 - Status: Granted - Kind: B2 - Application number: 17/392,125 - Filed: 2021-08-02 - Published: 2022-02-10 - Granted: 2024-06-04 - Earliest claimed: 2020-08-04 - Assignees: Artificial, Inc. - Applicants: Artificial, Inc. - Inventors: Jeff Washington; Geoffrey J. Budd; Nikhita Singh; Jake Sganga; Alexander Li Honda - Web page: https://patentgrep.com/patent/US11999066B2 ## CPC - B25J 9/1671: {characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems} - B25J 9/1605: {Simulation of manipulator lay-out, design, modelling of manipulator} - B25J 9/163: {learning, adaptive, model based, rule based expert control} - B25J 9/1653: {parameters identification, estimation, stiffness, accuracy, error analysis} - B25J 9/1656: {characterised by programming, planning systems for manipulators} - B25J 9/1658: {characterised by programming language} - B25J 9/1661: {characterised by task planning, object-oriented languages} - B25J 9/1679: {characterised by the tasks executed} - B25J 9/1692: {Calibration of manipulator} - B25J 9/1697: {Vision controlled systems} - B25J 13/089: {Determining the position of the robot with reference to its environment} - B25J 19/023: {including video camera means} - G01N 35/0099: {comprising robots or similar manipulators (robots per se B25J)} - G06F 3/04812: Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 40/10: Text processing (natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40) - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06N 20/00: Machine learning - G06T 7/50: Depth or shape recovery - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/73: using feature-based methods - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G01N 2035/0091: {GUI [graphical user interfaces]} - G01N 2035/0491: {Position sensing, encoding; closed-loop control} - G05B 2219/40392: Programming, visual robot programming language - G05B 2219/40393: Learn natural high level command, associate its template with a plan, sequence - G06T 2207/30204: Marker - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract A lab system calibrates robots and cameras within a lab. The lab system accesses, via a camera within a lab, an image of a robot arm, which comprises a visible tag located on an exterior. The lab system determines a position of the robot arm using position sensors located within the robot arm and determines a location of the camera relative to the robot arm based on the determined position and the location of the tag. The lab system calibrates the camera using the determined location of the camera relative to the robot arm. After calibrating the camera, the lab system accesses, via the camera, a second image of equipment in the lab that comprises a second visible tag on an exterior. The lab system determines, based on a location of the second visible tag within the accessed second image, a location of the equipment relative to the robot arm. ## Claims ### Claim 1 A method comprising: accessing, via a camera within a lab, a first image of a robot arm, the robot arm comprising a visible tag located on an exterior of the robot arm; determining a position of the robot arm using one or more position sensors within the robot arm; determining a location of the camera relative to the robot arm based on the determined position of the robot arm and a location of the tag within the first image; calibrating the camera using the determined location of the camera relative to the robot arm; after calibrating the camera, accessing, via the camera, a second image of lab equipment, the lab equipment comprising a second visible tag located on an exterior of the lab equipment; determining, based on a location of the second visible tag within the accessed second image, a location of the lab equipment relative to the robot arm; determining whether the location of lab equipment relative to the robot arm matches an expected location of the lab equipment; and in response to determining that the location of lab equipment relative to the robot arm does not match the expected location of the lab equipment, recalibrating the camera using the determined location of the lab equipment and the determined location of the camera relative to the robot arm. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the location of the camera relative to the robot arm is based on distortion of a lens of the camera. ### Claim 3 (depends on 1) The method of claim 1, further comprising: accessing, via the camera, a third image of the robot arm, the robot arm in the third image in a different position than in the first image; determining, based on the third image, that recalibration of the camera is necessary; determining a second location of the camera relative to the robot arm based on a second determined position of the robot arm in the third image and a second location of the tag within the third image; and calibrating the camera using the second location of the camera relative to the robot arm. ### Claim 4 (depends on 1) The method of claim 1, further comprising: periodically recalibrating the camera by: accessing, via the camera, a new image of the robot arm; determining a new position of the robot arm; determining a new location of the camera based on the new position of the robot arm; and calibrating the camera based on the new location of the camera. ### Claim 5 (depends on 1) The method of claim 1, wherein each visible tag comprises encoded information used to localize the robot arm and/or lab equipment. ### Claim 6 (depends on 1) The method of claim 1, wherein a width and shape of each visible tag is used to determine depth information from the images. ### Claim 7 (depends on 1) The method of claim 1, wherein the position of the robot arm is relative to a base of the robot and determining the location of the camera relative to the robot arm comprises: accessing a location of the base of the robot from a lab database; determining, based on the position, a location of the robot arm in the lab; and determining the location of the camera based on the location of the robot arm. ### Claim 8 A non-transitory computer-readable storage medium storing instructions executable by a processor, the instructions comprising: instructions for accessing, via a camera within a lab, a first image of a robot arm, the robot arm comprising a visible tag located on an exterior of the robot arm; instructions for determining a position of the robot arm using one or more position sensors within the robot arm; instructions for determining a location of the camera relative to the robot arm based on the determined position of the robot arm and a location of the tag within the first image; instructions for calibrating the camera using the determined location of the camera relative to the robot arm; instructions for, after calibrating the camera, accessing, via the camera, a second image of lab equipment, the lab equipment comprising a second visible tag located on an exterior of the lab equipment; instructions for determining, based on a location of the second visible tag within the accessed second image, a location of the lab equipment relative to the robot arm; instructions for determining whether the location of lab equipment relative to the robot arm matches an expected location of the lab equipment; and instructions for, in response to determining that the location of lab equipment relative to the robot arm does not match the expected location of the lab equipment, recalibrating the camera using the determined location of the lab equipment and the determined location of the camera relative to the robot arm. ### Claim 9 (depends on 8) The non-transitory computer-readable storage medium of claim 8, wherein determining the location of the camera relative to the robot arm is based on distortion of a lens of the camera. ### Claim 10 (depends on 8) The non-transitory computer-readable storage medium of claim 8, the instructions further comprising: instructions for accessing, via the camera, a third image of the robot arm, the robot arm in the third image in a different position than in the first image; instructions for determining, based on the third image, that recalibration of the camera is necessary; instructions for determining a second location of the camera relative to the robot arm based on a second determined position of the robot arm in the third image and a second location of the tag within the third image; and instructions for calibrating the camera using the second location of the camera relative to the robot arm. ### Claim 11 (depends on 8) The non-transitory computer-readable storage medium of claim 8, the instructions further comprising: instructions for periodically recalibrating the camera by: accessing, via the camera, a new image of the robot arm; determining a new position of the robot arm; determining a new location of the camera based on the new position of the robot arm; and calibrating the camera based on the new location of the camera. ### Claim 12 (depends on 8) The non-transitory computer-readable storage medium of claim 8, wherein each visible tag comprises encoded information used to localize the robot arm and/or lab equipment. ### Claim 13 (depends on 8) The non-transitory computer-readable storage medium of claim 8, wherein a width and shape of each visible tag is used to determine depth information from the images. ### Claim 14 (depends on 8) The non-transitory computer-readable storage medium of claim 8, wherein the position of the robot arm is relative to a base of the robot and the instructions for determining the location of the camera relative to the robot arm comprise: instructions for accessing a location of the base of the robot from a lab database; instructions for determining, based on the position, a location of the robot arm in the lab; and instructions for determining the location of the camera based on the location of the robot arm. ### Claim 15 A computer system comprising: a processor; and a non-transitory computer-readable storage medium storing instructions that, when executed, cause the processor to: access, via a camera within a lab, a first image of a robot arm, the robot arm comprising a visible tag located on an exterior of the robot arm; determine a position of the robot arm using one or more position sensors within the robot arm; determine a location of the camera relative to the robot arm based on the determined position of the robot arm and a location of the tag within the first image; calibrate the camera using the determined location of the camera relative to the robot arm; after calibrating the camera, access, via the camera, a second image of lab equipment, the lab equipment comprising a second visible tag located on an exterior of the lab equipment; determine, based on a location of the second visible tag within the accessed second image, a location of the lab equipment relative to the robot arm; determine whether the location of lab equipment relative to the robot arm matches an expected location of the lab equipment; and in response to determining that the location of lab equipment relative to the robot arm does not match the expected location of the lab equipment, recalibrate the camera using the determined location of the lab equipment and the determined location of the camera relative to the robot arm. ### Claim 16 (depends on 15) The computer system of claim 15, wherein determining the location of the camera relative to the robot arm is based on distortion of a lens of the camera. ### Claim 17 (depends on 15) The computer system of claim 15, the instructions further causing the processor to: access, via the camera, a third image of the robot arm, the robot arm in the third image in a different position than in the first image; determine, based on the third image, that recalibration of the camera is necessary; determine a second location of the camera relative to the robot arm based on a second determined position of the robot arm in the third image and a second location of the tag within the third image; and calibrate the camera using the second location of the camera relative to the robot arm. ### Claim 18 (depends on 15) The computer system of claim 15, the instructions further causing the processor to: periodically recalibrate the camera by: accessing, via the camera, a new image of the robot arm; determining a new position of the robot arm; determining a new location of the camera based on the new position of the robot arm; and calibrating the camera based on the new location of the camera. ### Claim 19 (depends on 15) The computer system of claim 15, wherein each visible tag comprises encoded information used to localize the robot arm and/or lab equipment. ### Claim 20 (depends on 15) The computer system of claim 15, wherein a width and shape of each visible tag is used to determine depth information from the images.