# US 11,745,350 B2 · Control method and control system of manipulator > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,745,350 B2 - Document ID: US11745350B2 - Status: Granted - Kind: B2 - Application number: 16/691,331 - Filed: 2019-11-21 - Published: 2020-10-08 - Granted: 2023-09-05 - Earliest claimed: 2019-04-02 - Assignees: Hyundai Motor Company; Kia Motors Corporation - Applicants: Hyundai Motor Company; Kia Motors Corporation - Inventors: Jae Ho Lee; Seok Won Lee - Web page: https://patentgrep.com/patent/US11745350B2 ## CPC - B25J 9/1697: {Vision controlled systems} - G06T 7/74: {involving reference images or patches} - B25J 9/1612: {characterised by the hand, wrist, grip control} - B25J 19/023: {including video camera means} - G06T 7/0002: {Inspection of images, e.g. flaw detection} - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 20/80: Recognising image objects characterised by unique random patterns - G05B 2219/39484: Locate, reach and grasp, visual guided grasping - G05B 2219/39505: Control of gripping, grasping, contacting force, force distribution - G05B 2219/39523: Set holding force as function of dimension, weight, shape, hardness, surface - G05B 2219/39527: Workpiece detector, sensor mounted in, near hand, gripper - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06V 2201/12: Acquisition of 3D measurements of objects - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract A control method of a manipulator is provided. The method includes photographing a target using a camera and detected the target using the photographed data. A holding motion for the target is set based on the detected target and a robot is operated to hold the target based on the set holding motion. ## Claims ### Claim 1 A control method of a manipulator, comprising: capturing, by a controller, an image of a target by operating a camera and detecting the target using data of the captured image; setting, by the controller, a holding motion for the target based on the detected target; and operating, by the controller, a robot to hold the target based on the set holding motion, wherein, in detecting the target, a plurality of properties of the target are derived by comparing data of the captured image of the target with a database stored in advance, wherein the plurality of properties of the target are strength, a modulus of elasticity, and a coefficient of friction according to material of the target, and wherein, in setting the holding motion for the target, an initial holding force is set based on the strength of the detected target to inhibit prevent breaking or deforming the target. ### Claim 2 (depends on 1) The method of claim 1, wherein the camera is a Time of Flight (ToF) camera configured to photograph a target by transmitting and receiving pulses. ### Claim 3 (depends on 1) The method of claim 1, wherein, in detecting the target, a three-dimensional (3D) image of the captured target is detected. ### Claim 4 (depends on 1) The method of claim 1, wherein, in detecting the target, the material of the target is detected. ### Claim 5 (depends on 4) The method of claim 4, wherein, in detecting the target, when the target is made of a plurality of materials, the materials and shapes formed by the materials are separately detected. ### Claim 6 (depends on 1) The method of claim 1, wherein, in detecting the target, mass of the target is further derived by comparing data of the captured image of the target with a database stored in advance. ### Claim 7 (depends on 6) The method of claim 6, wherein the plurality of properties of the target further includes density according to the material of the target. ### Claim 8 (depends on 1) The method of claim 1, wherein, in setting the holding motion for the target, an initial holding posture is further set based on the detected target. ### Claim 9 (depends on 1) The method of claim 1, wherein, in the setting of a holding motion for the target, whether it is possible to hold the target from a side with an initial holding force is determined based on mass and a modulus of elasticity of the target, and when it is impossible to hold the target with the initial holding force, an initial holding posture that supports weight of the target from the bottom is set. ### Claim 10 (depends on 1) The method of claim 1, wherein, in operating the robot to hold the target, the motion of the robot is operated in real time based on the set holding motion and a holding force that is applied by the robot is adjusted to be or less than a maximum holding force for the detected target. ### Claim 11 (depends on 10) The method of claim 10, wherein the maximum holding force for the target is determined based on the strength according to the material of the detected target. ### Claim 12 A control system of a manipulator, comprising: a camera configured to photograph a target; a target recognizer configured to detect the target using data photographed by the camera; a holding motion setter configured to set a holding motion for the target based on the target detected by the target recognizer; and a robot controller configured to operate a robot to hold the target based on the holding motion set by the holding motion setter, and a memory in which three-dimensional (3D) images according to data or a database of materials is stored in advance, and the target recognizer is configured to derive properties of the target by comparing the data of the photographed target with the database stored in advance in the memory, wherein the properties of the target are strength, a modulus of elasticity, and a coefficient of friction according to material of the target, and wherein, in setting the holding motion for the target, an initial holding force is set based on the strength of the detected target to prevent breaking or deforming the target. ### Claim 13 (depends on 12) The system of claim 12, wherein the camera is a Time of Flight (ToF) camera including a transmitter that transmits pulses and a receiver that receives pulses reflected by the target, and using the pulses received by the receiver as data. ### Claim 14 (depends on 12) The system of claim 12, wherein: the target recognizer is configured to further derive mass of the target by comparing the data of the photographed target with the database stored in advance in the memory. ### Claim 15 (depends on 12) The system of claim 12, wherein the robot that is operated by the robot controller is a manipulator including a plurality of joints and including a gripper at an end of the manipulator.