# US 11,027,427 B2 · Control device, picking system, distribution system, program, and control method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,027,427 B2 - Document ID: US11027427B2 - Status: Granted - Kind: B2 - Application number: 16/592,760 - Filed: 2019-10-03 - Published: 2020-01-30 - Granted: 2021-06-08 - Earliest claimed: 2017-04-04 - Assignees: MUJIN, Inc. - Applicants: MUJIN, Inc. - Inventors: Rosen Diankov - Web page: https://patentgrep.com/patent/US11027427B2 ## CPC - B25J 9/1666: {Avoiding collision or forbidden zones} - B25J 9/1674: {characterised by safety, monitoring, diagnostic} - B25J 9/1697: {Vision controlled systems} - B65G 47/905: {Control arrangements} - G06T 1/0014: {Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023)} - G05B 2219/40577: Multisensor object recognition - G05B 2219/45063: Pick and place manipulator - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - B65G: TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS (packaging B65B; handling thin or filamentary materials, e.g. paper sheets or thread, B65H; cranes B66C; portable mobile or lifting or hauling appliances, e.g. hoists, B66D; devices for lifting or lowering goods for loading or unloading purposes, e.g. fork-lift trucks, B66F9/00; emptying bottles, jars, casks, barrels or similar containers, not otherwise provided for, B67C9/00; delivering or transferring liquids B67D; filling or discharging vessels for liquefied, solidified or compressed gases F17C; pipe-line systems for fluids F17D) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract A point cloud information acquiring unit for acquiring point cloud information of a region including at least one of the picking hand and the work at a time which is at least one of (i) when the picking hand picks the work at the first position, (ii) when the picking hand moves the work out of the first container, (iii) when the picking hand moves the work in the second container and (iv) when the picking hand arranges the work at the second position; and an abnormality detection unit for detecting an abnormality based on the point cloud information acquired by the point cloud information acquiring unit are included. ## Claims ### Claim 1 A control device for controlling a picking apparatus that arranges a work picked at a first position of a first container by using a picking hand at a second position of a second container being different from the first container, the control device comprising: a point cloud information acquiring unit for acquiring point cloud information of a region including at least one of the picking hand and the work at a time which is at least one of (i) when the picking hand moves the work in the second container and (ii) when the picking hand arranges the work at the second position; an abnormality detection unit for detecting an abnormality based on the point cloud information acquired by the point cloud information acquiring unit; and an instruction sending unit for sending, to the picking apparatus, an instruction for suspending an operation of the picking apparatus in response to the abnormality detection unit detecting the abnormality, wherein: the abnormality detection unit is for detecting a load collapse inside the second container based on the point cloud information acquired by the point cloud information unit, and the instruction sending unit is for sending, to the picking apparatus, the instruction for suspending the operation of the picking apparatus in response to the abnormality detection unit detecting the load collapse. ### Claim 2 (depends on 1) The control device according to claim 1, further comprising an image acquiring unit for acquiring image data of an image including at least one of the picking hand and the work in a field angle at the at least one time, wherein: the point cloud information acquiring unit acquires the point cloud information based on the image acquired by the image acquiring unit. ### Claim 3 (depends on 1) The control device according to claim 1, wherein the point cloud information is three-dimensional point cloud information. ### Claim 4 (depends on 1) The control device according to claim 1, wherein: the abnormality detection unit is for detecting the abnormality based on (i) the point cloud information and (ii) at least one selected from a group including: a measurement value for a sensor arranged on the picking apparatus; a control value for at least one of the picking hand, and a robot arm coupled with the picking hand; and at least one of torque, a voltage value and a current value for a driving unit that drives at least one of the picking hand and the robot arm. ### Claim 5 (depends on 1) The control device according to claim 1, wherein: the abnormality detection unit is for detecting an interference based on the point cloud information acquired by the point cloud information unit, and the instruction sending unit is for sending, to the picking apparatus, an instruction for suspending an operation of the picking apparatus in response to the abnormality detection unit detecting the interference. ### Claim 6 A tangible, non-transitory computer-readable medium having processor instructions stored thereon that, when executed cause the processor to perform operations for controlling a picking apparatus that arranges a work picked at a first position of a first container by using a picking hand at a second position of a second container being different from the first container, the operations comprising: acquiring point cloud information representing a region including at least one of the picking hand and the work at a time that is at least one of (i) when the picking hand moves the work in the second container, and (ii) when the picking hand arranges the work at the second position; detecting an abnormality based on the point cloud information acquired in the acquiring point cloud information and sending, to the picking apparatus, an instruction for suspending an operation of the picking apparatus in response to the detecting abnormality detecting the abnormality, wherein: the detecting the abnormality includes detecting a load collapse inside the second container based on the point cloud information acquired in the acquiring point cloud information, the sending the instruction includes sending, to the picking apparatus, the instruction for suspending the operation of the picking apparatus in response to detecting the load collapse. ### Claim 7 A control method for controlling a picking apparatus that arranges a work picked at a first position of a first container by using a picking hand at a second position of a second container being different from the first container, the control method comprising: acquiring point cloud information of a region including at least one of the picking hand and the work at a time that is at least one of (i) when the picking hand moves the work in the second container, and (ii) when the picking hand arranges the work at the second position; and detecting an abnormality based on the point cloud information acquired in the acquiring point cloud information and sending, to the picking apparatus, an instruction for suspending an operation of the picking apparatus in response to the detecting abnormality detecting the abnormality, wherein: the detecting the abnormality includes detecting a load collapse inside the second container based on the point cloud information acquired in the acquiring point cloud information, the sending the instruction includes sending, to the picking apparatus, the instruction for suspending the operation of the picking apparatus in response to detecting the load collapse.