# US 11,027,433 B2 · Robot system and control method of robot system for taking out workpieces loaded in bulk > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,027,433 B2 - Document ID: US11027433B2 - Status: Granted - Kind: B2 - Application number: 16/384,923 - Filed: 2019-04-16 - Published: 2019-11-21 - Granted: 2021-06-08 - Earliest claimed: 2018-05-18 - Assignees: Fanuc Corporation - Applicants: Fanuc Corporation - Inventors: Tadanori Suzuki - Web page: https://patentgrep.com/patent/US11027433B2 ## CPC - B25J 9/1697: {Vision controlled systems} - B25J 9/042: {comprising an articulated arm} - B25J 9/1005: {comprising adjusting means} - B25J 9/1612: {characterised by the hand, wrist, grip control} - B25J 15/0033: {with gripping surfaces having special shapes} - B25J 15/0608: {with magnetic holding means} - B25J 19/023: {including video camera means} - G06T 1/0014: {Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023)} - G06T 7/75: {involving models} - G05B 2219/40537: Detect if robot has picked up more than one piece from bin; interlocked parts - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract A controller includes a generation unit that generates three-dimensional information, a deletion unit that deletes information about at least part of the measurement points in the three-dimensional information, and a determination unit that determines a state where the robot grips a workpiece. The generation unit generates first three-dimensional information before the robot implements an operation of gripping a target workpiece. The generation unit generates second three-dimensional information after the robot has implemented an operation of lifting the target workpiece. The deletion unit generates third three-dimensional information in which information about measurement points in the second three-dimensional information is deleted from the first three-dimensional information. The determination unit determines whether or not the workpiece in the third three-dimensional information matches the target workpiece. ## Claims ### Claim 1 A robot system that takes out each of workpieces loaded in bulk inside a case, the robot system comprising: a hand that grips a workpiece, a robot that moves the hand, a three-dimensional sensor that acquires information about distances to the workpieces, and a controller that controls the hand and the robot, wherein the controller includes: an operation control unit that sends operation commands to the hand and the robot; a generation unit that generates three-dimensional information including information about positions of measurement points set to the workpieces based on an output from the three-dimensional sensor; a storage unit that stores three-dimensional shape data of the workpieces; a detection unit that detects positions and orientations of the workpieces by implementing model matching in which the three-dimensional information about the workpieces is compared with the three-dimensional shape data; a selection unit that selects a target workpiece to be taken out by the robot based on the positions and the orientations of the workpieces; a deletion unit that deletes information about at least part of the measurement points included in the three-dimensional information; and a determination unit that determines a state after the robot implements an operation of lifting the workpiece, the generation unit generates first three-dimensional information based on an output from the three-dimensional sensor that has captured an image of the workpieces before the robot implements an operation of gripping the target workpiece, the operation control unit controls the robot so as to stop after implementing an operation of gripping and lifting the target workpiece selected based on the first three-dimensional information, the generation unit generates second three-dimensional information based on an output from the three-dimensional sensor that has captured an image of the workpieces after the robot implements the operation of lifting the target workpiece, the deletion unit compares a position of the measurement point in the first three-dimensional information with a position of the measurement point in the second three-dimensional information, detects, as a specified measurement point, the measurement point in the first three-dimensional information from which the measurement point in the second three-dimensional information is present within a predetermined distance range, and generates third three-dimensional information in which information about the specified measurement point is deleted from the first three-dimensional information, the detection unit detects a position and an orientation of a workpiece included in the third three-dimensional information, the determination unit determines whether or not the workpiece included in the third three-dimensional information matches the target workpiece, and when the workpiece included in the third three-dimensional information matches the target workpiece, the operation control unit controls the robot so as to convey the workpiece gripped by the hand to a predetermined conveyance destination position. ### Claim 2 (depends on 1) The robot system according to claim 1, wherein the operation control unit controls the robot so as to return the workpiece gripped by the hand to a vicinity of an original position of the workpiece before being gripped by the hand, when the determination unit determines that the workpiece included in the third three-dimensional information is different from the target workpiece or that two or more workpieces are included in the third three-dimensional information, and after the robot returns the workpiece to the vicinity of the original position, the three-dimensional sensor captures an image of the workpieces arranged in the case, the detection unit detects positions and orientations of the workpieces, and the selection unit selects a next target workpiece to be taken out by the robot. ### Claim 3 (depends on 1) The robot system according to claim 1, wherein the determination unit determines whether or not number of measurement points in the first three-dimensional information other than specified measurement points is larger than a predetermined determination value related to load collapse, when the determination unit determines that the number of measurement points other than specified measurement points is larger than the determination value related to the load collapse, the operation control unit controls the robot so as to return the workpiece gripped by the hand to the vicinity of the original position of the workpiece before being gripped by the hand, and after the robot returns the workpiece to the vicinity of the original position, the three-dimensional sensor captures an image of the workpieces arranged in the case, the detection unit detects positions and orientations of the workpieces, and the selection unit selects a next target workpiece to be taken out by the robot. ### Claim 4 (depends on 1) The robot system according to claim 1, wherein the determination unit determines whether or not number of measurement points in the first three-dimensional information other than specified measurement points is smaller than a predetermined determination value related to failure in gripping, and when the determination unit determines that the number of measurement points other than specified measurement points is smaller than the determination value related to the failure in the gripping, the three-dimensional sensor captures an image of the workpieces arranged in the case, the detection unit detects positions and orientations of the workpieces, and the selection unit selects a next target workpiece to be taken out by the robot. ### Claim 5 A control method of a robot system that includes a robot and a hand and takes out each of workpieces loaded in bulk in a case, the method comprising: an operation control step of changing a position and an orientation of the robot; an image capturing step of capturing an image of the workpieces with a three-dimensional sensor that acquires information about distances to the workpieces; a generation step of generating three-dimensional information including information about positions of measurement points set to the workpieces based on an output from the three-dimensional sensor; a detection step of detecting the positions and the orientations of the workpieces by implementing model matching in which the three-dimensional information about the workpieces is compared with three-dimensional shape data of the workpieces; a selection step of selecting a target workpiece to be taken out by the robot based on the positions and the orientations of the workpieces; a deletion step of deleting information about at least part of the measurement points included in the three-dimensional information; and a determination step of determining a state after the robot implements an operation of lifting the workpiece, wherein the generation step includes generating first three-dimensional information based on an output from the three-dimensional sensor that has captured an image of the workpiece before the robot implements an operation of gripping the target workpiece, the operation control step includes a step of controlling the robot so as to stop after implementing an operation of gripping and lifting the target workpiece selected based on the first three-dimensional information, the generation step includes generating second three-dimensional information based on an output from the three-dimensional sensor that has captured an image of the workpieces after the robot has implemented the operation of lifting the target workpiece, the deletion step includes a step of comparing a position of the measurement point in the first three-dimensional information with a position of the measurement point in the second three-dimensional information, a step of detecting, as a specified measurement point, the measurement point in the first three-dimensional information from which the measurement point in the second three-dimensional information is present within a predetermined distance range, and a step of generating third three-dimensional information in which information about the specified measurement point is deleted from the first three-dimensional information, the detection step includes a step of detecting a position and an orientation of a workpiece included in the third three-dimensional information, the determination step includes a step of determining whether or not the workpiece included in the third three-dimensional information matches the target workpiece, and the operation control step includes a step of controlling, when the workpiece included in the third three-dimensional information matches the target workpiece, the robot so as to convey the workpiece gripped by the hand to a predetermined conveyance destination position. ### Claim 6 (depends on 5) The control method of a robot system according to claim 5, wherein the robot is controlled to return the workpiece gripped by the hand to a vicinity of an original position of the workpiece before being gripped by the hand, in response to determining in the determination step that the workpiece included in the third three-dimensional information is different from the target workpiece or that two or more workpieces are included in the third three-dimensional information, and after the robot returns the workpiece to the vicinity of the original position, an image capturing step of capturing an image of the workpieces arranged in the case with the three-dimensional sensor is performed, a detection step of detecting positions and orientations of the workpieces is performed, and a selection step of selecting a next target workpiece to be taken out by the robot is performed. ### Claim 7 (depends on 5) The control method of a robot system according to claim 5, wherein the determination step includes determining whether or not number of measurement points in the first three-dimensional information other than specified measurement points is larger than a predetermined determination value related to load collapse, in response to determining that the number of measurement points other than specified measurement points is larger than the determination value related to the load collapse, the robot is controlled so as to return the workpiece gripped by the hand to the vicinity of the original position of the workpiece before being gripped by the hand, and after the robot returns the workpiece to the vicinity of the original position, an image capturing step of capturing an image of the workpieces arranged in the case with the three-dimensional sensor is performed, a detection step of detecting positions and orientations of the workpieces is performed, and a selection step of selecting a next target workpiece to be taken out by the robot is performed. ### Claim 8 (depends on 5) The control method of a robot system according to claim 5, wherein the determination step includes determining whether or not number of measurement points in the first three-dimensional information other than specified measurement points is smaller than a predetermined determination value related to failure in gripping, and in response to determining that the number of measurement points other than specified measurement points is smaller than the determination value related to the failure in the gripping, an image capturing step of capturing an image of the workpieces arranged in the case with the three-dimensional sensor is performed, a detection step of detecting positions and orientations of the workpieces is performed, and a selection step of selecting a next target workpiece to be taken out by the robot is performed.