# US 10,556,345 B2 · Robot arm apparatus and calibration method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,556,345 B2 - Document ID: US10556345B2 - Status: Granted - Kind: B2 - Application number: 15/119,631 - Filed: 2015-02-12 - Published: 2017-03-09 - Granted: 2020-02-11 - Earliest claimed: 2014-02-28 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Yasuhisa Kamikawa; Takara Kasai; Yohei Kuroda; Wataru Kokubo; Toshimitsu Tsuboi; Tetsuharu Fukushima; Yasuhiro Matsuda; Atsushi Miyamoto - Web page: https://patentgrep.com/patent/US10556345B2 ## CPC - B25J 9/1692: {Calibration of manipulator} - A61B 34/30: Surgical robots - B25J 9/1697: {Vision controlled systems} - A61B 90/361: {Image-producing devices, e.g. surgical cameras} - A61B 2017/00725: {Calibration or performance testing} - A61B 2034/102: {Modelling of surgical devices, implants or prosthesis} - G05B 2219/39016: Simultaneous calibration of manipulator and camera - G05B 2219/39029: Verify if calibration position is a correct, by comparing with range in rom - G05B 2219/39045: Camera on end effector detects reference pattern - G05B 2219/39057: Hand eye calibration, eye, camera on hand, end effector - G05B 2219/40613: Camera, laser scanner on end effector, hand eye manipulator, local - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract A robot arm apparatus including: an arm unit made up of a plurality of links joined by one or a plurality of joint units, the arm unit is connectable to an imaging unit. An internal model including at least geometric information of the arm unit and focus position information of the imaging unit is updated based on internal model information acquired in a state in which the imaging unit is pointed at a reference point in a real space. ## Claims ### Claim 1 A medical arm system, comprising: an arm unit that comprises a plurality of links, wherein the plurality of links is joined by a plurality of joint units; an imaging unit configured to capture an image of a surgical field, wherein the imaging unit is attachable to the arm unit; and a processor circuitry configured to: acquire internal model information; update an internal model based on the internal model information; and drive the medical arm system based on the internal model, wherein the internal model information comprises first geometric information of the arm unit, second geometric information of a jig attachable to the arm unit, and focus position information of the imaging unit, and the internal model information is acquired in a state in which the imaging unit is pointed at a specific point on the jig as a reference point in a real space. ### Claim 2 (depends on 1) The medical arm system according to claim 1, wherein the internal model information includes misalignment information that indicates an amount of misalignment between a field of view of the imaging unit and the reference point in the real space. ### Claim 3 (depends on 1) The medical arm system according to claim 1, wherein the processor circuitry is further configured to: acquire a plurality of internal model information in a plurality of states in which the arm unit and the imaging unit are in a plurality of positions and a plurality of orientations; and update the internal model based on the plurality of internal model information. ### Claim 4 (depends on 3) The medical arm system according to claim 3, wherein the processor circuitry is further configured to: acquire the plurality of internal model information with respect to a plurality of reference points; and update the internal model based on the plurality of internal model information corresponding to the plurality of reference points. ### Claim 5 (depends on 3) The medical arm system according to claim 3, wherein the processor circuitry is further configured to: create a plurality of internal model formulas based on the plurality of internal model information, wherein each internal model formula of the plurality of internal model formulas represents the internal model, and each internal model formula of the plurality of internal model formulas includes a discrepancy between a first coordinate system in the internal model and a second coordinate system in the real space; and update the internal model based on the plurality of internal model formulas. ### Claim 6 (depends on 1) The medical arm system according to claim 1, wherein the reference point is set as a point in the real space with known coordinates. ### Claim 7 (depends on 1) The medical arm system according to claim 1, wherein the processor circuitry is further configured to control a drive operation of a joint unit of the plurality of joint units based on: a state of the joint unit, and a first control value for cooperative control of the arm unit, wherein the first control value is based on a state of the arm unit acquired based on a plurality of states of the joint unit. ### Claim 8 (depends on 7) The medical arm system according to claim 7, wherein the processor circuitry is configured to control the drive operation of the joint unit based on a second control value for whole body cooperative control of the arm unit based on generalized inverse dynamics that utilizes the state of the arm unit acquired based on: a plurality of detected states of the joint unit, and a purpose of motion and a constraint condition of the arm unit. ### Claim 9 (depends on 8) The medical system according to claim 8, wherein the second control value is computed based on a virtual force that achieves the purpose of motion in an operation space, and the operation space describes a relationship between a force on the arm unit and an acceleration in the arm unit, and an actual force by which the virtual force drives the joint unit based on the constraint condition. ### Claim 10 (depends on 7) The medical arm system according to claim 7, wherein the processor circuitry is further configured to control the drive operation of the joint unit based on a command value, and a computation of the command value is based on a correction operation on an influence of a disturbance on the first control value. ### Claim 11 (depends on 10) The medical arm system according to claim 10, wherein the command value is based on a disturbance estimation value that indicates the influence of a disturbance on driving of the joint unit estimated based on a detected state of the joint unit. ### Claim 12 (depends on 8) The medical arm system according to claim 8, wherein the processor circuitry is configured to: control the drive operation of the joint unit based on the constraint condition that indicates a certain point in the real space is positioned on an optical axis of the imaging unit, and control the drive operation of the arm unit based on a pivot operation in which the certain point serves as an apex of the pivot operation in the state in which the imaging unit is pointed at the certain point. ### Claim 13 (depends on 8) The medical arm system according to claim 8, wherein the processor circuitry is further configured to: control the drive operation of the joint unit based on the constraint condition that indicates a first point that is locked to a second point, wherein the first point is on an optical axis of the imaging unit, and the second point is a certain point in the real space; and control the drive operation of the arm unit based on a pivot operation centered on the second point in the state in which the imaging unit is pointed at the certain point. ### Claim 14 (depends on 1) The medical arm system according to claim 1, wherein each joint unit of the plurality of joint units includes a joint state detecting unit that is configured to detect a state of a joint unit of the plurality of joint units, and the joint state detecting unit includes at least one of: a torque detecting unit configured to detect a generated torque in the joint unit and an external torque applied from an outside to the joint unit, and a rotational angle detecting unit configured to detect a rotational angle of the joint unit. ### Claim 15 (depends on 7) The medical arm system according to claim 7, wherein the first control value is a torque in the joint unit. ### Claim 16 (depends on 1) The medical arm system according to claim 1, wherein the imaging unit is a microscope. ### Claim 17 A calibration method, comprising: in a medical arm system comprising an arm unit and an imaging unit attachable to the arm unit, the arm unit comprising a plurality of links joined by at least one joint unit, and the imaging unit configured to capture an image of a surgical field: acquiring internal model information; and updating an internal model based on the internal model information, wherein the internal model is used to drive the medical arm system, the internal model information is acquired in a state in which the imaging unit pointed at a specific point on a jig attachable to the arm unit as a reference point in a real space, and the internal model information comprises first geometric information of the arm unit, second geometric information of the jig attachable to the arm unit, and focus position information of the imaging unit. ### Claim 18 A non-transitory computer-readable medium having stored thereon, computer-executable instructions, which when executed by a computer, cause the computer to execute operations, the operations comprising: in a medical arm system comprising an arm unit and an imaging unit attachable to the arm unit, the arm unit comprising a plurality of links joined by at least one joint unit, and the imaging unit configured to capture an image of a surgical field: acquiring internal model information; and updating an internal model based on the internal model information, wherein the internal model is used to drive the medical arm system, the internal model information is acquired in a state in which the imaging unit is pointed at a specific point on a jig attachable to the arm unit as a reference point in a real space, and the internal model information comprises first geometric information of the arm unit, second geometric information of the jig attachable to the arm unit, and focus position information of the imaging unit.