# US 12,214,489 B2 · Control device, control system, control method, and recording medium with control program recorded thereon > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,214,489 B2 - Document ID: US12214489B2 - Status: Granted - Kind: B2 - Application number: 17/909,065 - Filed: 2020-03-31 - Published: 2023-03-23 - Granted: 2025-02-04 - Earliest claimed: 2020-03-31 - Assignees: NEC Corporation - Applicants: NEC Corporation - Inventors: Mineto Satoh - Web page: https://patentgrep.com/patent/US12214489B2 ## CPC - B25J 9/1692: {Calibration of manipulator} - B25J 13/089: {Determining the position of the robot with reference to its environment} - G05B 2219/39024: Calibration of manipulator - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract This control device calculates a parameter, which represents transformation between a first coordinate system for image information obtained by capturing an image of a target to be controlled and a second coordinate system for defining the movement of the target to be controlled, by using position information indicating the position of the target to be controlled in the first coordinate system and position information indicating the position of the target to be controlled in the second coordinate system. Further, the control device uses image information containing the target to be controlled and the calculated parameter to calculate the position information of the target to be controlled for the second coordinate system, and controls the target to be controlled on the basis of the calculated position information of the target to be controlled. ## Claims ### Claim 1 A control device comprising: a memory storing instructions; and one or more processors configured to execute the instructions to: calculate a parameter representing transformation between a first coordinate system for image information obtained by capturing an image of a target to be controlled and a second coordinate system that defines an operation of the target to be controlled by using position information representing a position of the target to be controlled in the first coordinate system and position information representing a position of the target to be controlled in the second coordinate system; perform a predetermined calculation on the parameter with respect to the position of the target to be controlled in the image information and the position of the target to be controlled based on control information; calculate a distribution of the position of the target to be controlled in the first coordinate system and a distribution of the position of the target to be controlled in the second coordinate system; calculate the position information of the target to be controlled in the second coordinate system by using image information including the target to be controlled and the calculated parameter; and control the target to be controlled based on the calculated position information of the target to be controlled. ### Claim 2 (depends on 1) The control device according to claim 1, wherein the one or more processors are configured to execute the instructions to: calculate the distribution of the position of the target to be controlled in the first coordinate system and the distribution of the position of the target to be controlled in the second coordinate system by adding random numbers to the position of the target to be controlled in the image information and the position of the target to be controlled based on the control information. ### Claim 3 (depends on 2) The control device according to claim 2, wherein the random number is a random number according to a normal distribution. ### Claim 4 (depends on 1) The control device according to claim 1, wherein the one or more processors are further configured to execute the instructions to: transform a position in an image coordinate system or a camera coordinate system into a position in a world coordinate system; and extract the position of the target to be controlled in the image information by using the image information represented in the world coordinate system. ### Claim 5 (depends on 1) The control device according to claim 1, wherein the one or more processors are further configured to execute the instructions to: transform a position in a robot coordinate system into a position in a world coordinate system; and calculate the position of the target to be controlled using the control information represented in the world coordinate system. ### Claim 6 (depends on 1) The control device according to claim 1, wherein the one or more processors are further configured to execute the instructions to: transform a position in an image coordinate system or a position in a camera coordinate system into a position in a world coordinate system; transform a position in a robot coordinate system into a position in the world coordinate system; extract the position of the target to be controlled in the image information by using the image information represented in the world coordinate system; and calculate the position of the target to be controlled using the control information represented in the world coordinate system. ### Claim 7 (depends on 1) The control device according to claim 1, wherein the one or more processors are further configured to execute the instructions to: update an action policy with a decrease in a difference between the position information indicating the position transformed by using the parameter from the position information indicating the position of the target to be controlled in the first coordinate system and the position information indicating the position of the target to be controlled in the second coordinate system as a reward. ### Claim 8 A control method performed by a computer and comprising: calculating a parameter representing transformation between a first coordinate system for image information obtained by capturing an image of a target to be controlled and a second coordinate system that defines an operation of the target to be controlled by using position information representing a position of the target to be controlled in the first coordinate system and position information representing a position of the target to be controlled in the second coordinate system; performing a predetermined calculation on the parameter with respect to the position of the target to be controlled in the image information and the position of the target to be controlled based on control information; calculating a distribution of the position of the target to be controlled in the first coordinate system and a distribution of the position of the target to be controlled in the second coordinate system; calculating the position information of the target to be controlled in the second coordinate system by using image information including the target to be controlled and the calculated parameter; and controlling the target to be controlled based on the calculated position information of the target to be controlled. ### Claim 9 A non-transitory computer-readable recording medium storing a control program executable by a computer to perform processing comprising: calculating a parameter representing transformation between a first coordinate system for image information obtained by capturing an image of a target to be controlled and a second coordinate system that defines an operation of the target to be controlled by using position information representing a position of the target to be controlled in the first coordinate system and position information representing a position of the target to be controlled in the second coordinate system; performing a predetermined calculation on the parameter with respect to the position of the target to be controlled in the image information and the position of the target to be controlled based on control information; calculating a distribution of the position of the target to be controlled in the first coordinate system and a distribution of the position of the target to be controlled in the second coordinate system; calculating the position information of the target to be controlled in the second coordinate system by using image information including the target to be controlled and the calculated parameter; and controlling the target to be controlled based on the calculated position information of the target to be controlled.