# US 11,826,918 B2 · Systems and methods for camera calibration with a fiducial of unknown position on an articulated arm of a programmable motion device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,826,918 B2 - Document ID: US11826918B2 - Status: Granted - Kind: B2 - Application number: 17/167,648 - Filed: 2021-02-04 - Published: 2021-08-12 - Granted: 2023-11-28 - Earliest claimed: 2020-02-06 - Assignees: Berkshire Grey Operating Company, Inc. - Applicants: Berkshire Grey Operating Company, Inc. - Inventors: Yu Ding; Christopher Geyer; Luis Galup; Jeffrey Kittredge; Thomas Koletschka; Lakshman Kumar; Anjana Nellithimaru; Ryan O'Hern; Jeremy Saslaw - Web page: https://patentgrep.com/patent/US11826918B2 ## CPC - B25J 9/1697: {Vision controlled systems} - B25J 9/1692: {Calibration of manipulator} - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06V 10/147: Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19) - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G05B 2219/37009: Calibration of vision system, camera, adapt light level - G05B 2219/39008: Fixed camera detects reference pattern held by end effector - G05B 2219/39046: Compare image of plate on robot with reference, move till coincidence, camera - G05B 2219/40607: Fixed camera to observe workspace, object, workpiece, global - G06T 2207/10016: Video; Image sequence - G06T 2207/30204: Marker - G06T 2207/30208: Marker matrix - G06V 2201/06: Recognition of objects for industrial automation - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract A system is disclosed for providing extrinsic calibration of a camera to a relative working environment of a programmable motion device that includes an end-effector. The system includes a fiducial located at or near the end-effector, at least one camera system for viewing the fiducial as the programmable motion device moves in at least three degrees of freedom, and for capturing a plurality of images containing the fiducial, and a calibration system for analyzing the plurality of images to determine a fiducial location with respect to the camera to permit calibration of the camera with the programmable motion device. ## Claims ### Claim 1 A system for providing extrinsic calibration of a camera to a relative working environment of a programmable motion device that includes an end-effector, said system comprising: a calibration unit attached to the end effector, the calibration unit including a plurality of fiducials; at least one camera for viewing a fiducial among the plurality of fiducials of the calibration unit as the programmable motion device moves in at least three degrees of freedom, and for capturing a plurality of images containing the fiducial; and a calibration processing system for analyzing the plurality of images to determine a fiducial location of the fiducial with respect to the at least one camera to permit calibration of the at least one camera with the programmable motion device, the calibration unit including a coupling portion for coupling to the end-effector and a transverse portion that is transverse to the coupling portion, wherein the plurality of fiducials are positioned on mutually opposing sides of the transverse portion of the calibration unit, the plurality of fiducials including a light emitting diode (LED) on a top side or a bottom side of the transverse portion, wherein the transverse portion of the calibration unit further includes control circuitry for controlling the LED. ### Claim 2 (depends on 1) The system as claimed in claim 1, wherein the LED provides intermittent visible illumination. ### Claim 3 (depends on 1) The system as claimed in claim 1, wherein the LED provides intermittent infra-red illumination. ### Claim 4 (depends on 1) The system as claimed in claim 1, wherein the plurality of fiducials further includes a retroreflective fiducial. ### Claim 5 (depends on 4) The system as claimed in claim 4, wherein the retroreflective fiducial is at least partially spherically shaped. ### Claim 6 (depends on 1) The system as claimed in claim 1, wherein the calibration system constructs a matrix for each of the plurality of images of the fiducial. ### Claim 7 (depends on 6) The system as claimed in claim 6, wherein the calibration system concatenates extrinsic parameters of the camera to the programmable motion device. ### Claim 8 (depends on 7) The system as claimed in claim 7, wherein the calibration system employs non-linear least square optimization. ### Claim 9 (depends on 1) The system as claimed in claim 1, wherein the at least one camera is positioned on an automated carrier. ### Claim 10 (depends on 1) The system as claimed in claim 1, wherein the at least one camera is positioned on a non-automated carrier. ### Claim 11 (depends on 1) The system as claimed in claim 1, wherein the at least one camera is positioned on a bin. ### Claim 12 (depends on 1) The system as claimed in claim 1, wherein the plurality of fiducials includes a first pair of light emitting diodes (LEDs) that are positioned on mutually opposing sides of the transverse portion and a second pair of LEDs that are positioned on mutually opposing sides of the transverse portion, wherein the first pair of LEDs and the second pair of LEDs are further positioned on mutually opposing sides of the end-effector, the first pair of LEDs being visible LEDs and the second pair of LEDs being infra-red LEDs, wherein the control circuitry included in the transverse portion of the calibration unit is configured to control the first pair of LEDs and the second pair of LEDs. ### Claim 13 A method for providing extrinsic calibration of a camera to a relative working environment of a programmable motion device that includes an end-effector, said method comprising: viewing with the camera a fiducial among a plurality of fiducials located on a calibration unit attached to the end-effector; capturing a plurality of images containing the fiducial using the camera as the programmable motion device moves in at least three degrees of freedom; and analyzing the plurality of images to determine a fiducial location with respect to the camera to permit calibration of the camera with the programmable motion device, wherein the calibration unit includes a coupling portion for coupling to the end-effector and a transverse portion that is transverse to the coupling portion, wherein the plurality of fiducials are positioned on mutually opposing sides of the transverse portion of the calibration unit, the plurality of fiducials including a light emitting diode (LED) on a top side or a bottom side of the transverse portion, wherein the transverse portion further includes control circuitry for controlling the LED. ### Claim 14 (depends on 13) The method as claimed in claim 13, wherein method includes providing intermittent visible illumination with the LED. ### Claim 15 (depends on 13) The method as claimed in claim 13, wherein method includes providing intermittent infra-red illumination with the LED. ### Claim 16 (depends on 13) The method as claimed in claim 13, wherein the plurality of fiducials further includes a retroreflective fiducial. ### Claim 17 (depends on 16) The method as claimed in claim 16, wherein the retroreflective fiducial is at least partially spherically shaped. ### Claim 18 (depends on 13) The method as claimed in claim 13, wherein analyzing the plurality of images includes constructing a matrix for each of the plurality of images of the fiducial. ### Claim 19 (depends on 18) The method as claimed in claim 18, wherein analyzing the plurality of images includes concatenating extrinsic parameters of the camera to the programmable motion device. ### Claim 20 (depends on 19) The method as claimed in claim 19, wherein analyzing the plurality of images includes employing non-linear least square optimization. ### Claim 21 (depends on 13) The method as claimed in claim 13, wherein the camera is positioned on an automated carrier. ### Claim 22 (depends on 13) The method as claimed in claim 13, wherein the camera is positioned on a non-automated carrier. ### Claim 23 (depends on 13) The method as claimed in claim 13, wherein the camera is positioned on a bin. ### Claim 24 (depends on 13) The method as claimed in claim 13, wherein viewing with the camera the fiducial as the programmable motion device moves in at least three degrees of freedom, capturing the plurality of images containing the fiducial, and analyzing the plurality of images to determine the fiducial location all occur during the processing of objects with the programmable motion device. ### Claim 25 A method for providing extrinsic calibration of a camera to a relative working environment of a programmable motion device that includes an end-effector, said method comprising: capturing a plurality of images containing a fiducial among a plurality of fiducials located on a calibration unit attached to the end-effector while the programmable motion device moves in at least three degrees of freedom; and analyzing the plurality of images to determine a fiducial location with respect to the camera to permit calibration of the camera with the programmable motion device, wherein the calibration unit including a coupling portion for coupling to the end-effector and a transverse portion that is transverse to the coupling portion, and wherein the plurality of fiducials includes a first pair of light emitting diodes (LEDs) that are positioned on mutually opposing sides of the transverse portion of the calibration unit and a second pair of LEDs that are positioned on mutually opposing sides of the transverse portion, wherein the first pair of LEDs and the second pair of LEDs are further positioned on mutually opposing sides of the end-effector, the first pair of LEDs being visible LEDs and the second pair of LEDs being infra-red LEDs, wherein the transverse portion of the calibration unit further includes control circuitry for controlling the first pair of LEDs and the second pair of LEDs. ### Claim 26 (depends on 25) The method as claimed in claim 25, wherein analyzing the plurality of images includes constructing a matrix for each of the plurality of images of the fiducial. ### Claim 27 (depends on 26) The method as claimed in claim 26, wherein analyzing the plurality of images includes concatenating extrinsic parameters of the camera to the programmable motion device. ### Claim 28 (depends on 27) The method as claimed in claim 27, wherein analyzing the plurality of images includes employing non-linear least square optimization.