# US 12,214,512 B2 · Coordinating multiple robots to meet workflow and avoid conflict > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,214,512 B2 - Document ID: US12214512B2 - Status: Granted - Kind: B2 - Application number: 18/238,048 - Filed: 2023-08-25 - Published: 2024-03-21 - Granted: 2025-02-04 - Earliest claimed: 2019-10-25 - Assignees: Dexterity, Inc. - Applicants: Dexterity, Inc. - Inventors: Zhouwen Sun; Kevin Jose Chavez; Samir Menon; Harry Zhe Su; Talbot Morris-Downing; Rohit Arka Pidaparthi; Ayush Sharma; Raphael Georg Wirth - Web page: https://patentgrep.com/patent/US12214512B2 ## CPC - B25J 9/1697: {Vision controlled systems} - G05B 19/4182: {manipulators and conveyor only} - B07C 1/04: Forming a stream from a bulk; Controlling the stream, e.g. spacing the articles - B07C 5/36: Sorting apparatus characterised by the means used for distribution {(sorting according to destination B07C3/003, B07C3/02)} - B25J 9/0093: {co-operating with conveyor means} - B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor} - B25J 9/1612: {characterised by the hand, wrist, grip control} - B25J 9/1653: {parameters identification, estimation, stiffness, accuracy, error analysis} - B25J 9/1664: {characterised by motion, path, trajectory planning} - B25J 9/1671: {characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems} - B25J 9/1689: {Teleoperation} - B25J 13/08: by means of sensing devices, e.g. viewing or touching devices - B25J 15/0616: {with vacuum} - B25J 19/023: {including video camera means} - B65G 47/905: {Control arrangements} - B65G 47/917: {control arrangements} - B07C 2501/0063: Using robots - B25J 9/1679: {characterised by the tasks executed} - B65G 2201/0285: Postal items, e.g. letters, parcels - B65G 2203/0216: Codes or marks on the article - B65G 2203/041: Camera - G05B 2219/39102: Manipulator cooperating with conveyor - G05B 2219/40161: Visual display of machining, operation, remote viewing - G05B 2219/40195: Tele-operation, computer assisted manual operation - G06Q 10/08741: {by picking of items from inventory for fulfillment} - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - B07C: POSTAL SORTING; SORTING INDIVIDUAL ARTICLES, OR BULK MATERIAL FIT TO BE SORTED PIECE-MEAL, e.g. BY PICKING (specially adapted for a specific purpose covered by another class, see the relevant place, e.g. A43D33/06, B23Q7/12) - B65G: TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS (packaging B65B; handling thin or filamentary materials, e.g. paper sheets or thread, B65H; cranes B66C; portable mobile or lifting or hauling appliances, e.g. hoists, B66D; devices for lifting or lowering goods for loading or unloading purposes, e.g. fork-lift trucks, B66F9/00; emptying bottles, jars, casks, barrels or similar containers, not otherwise provided for, B67C9/00; delivering or transferring liquids B67D; filling or discharging vessels for liquefied, solidified or compressed gases F17C; pipe-line systems for fluids F17D) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract A robotic singulation system is disclosed. In various embodiments, sensor data including image data associated with a workspace is received. The sensor data is used to generate a three dimensional view of at least a portion of the workspace, the three dimensional view including boundaries of a plurality of items present in the workspace. A grasp strategy is determined for each of at least a subset of items, and for each grasp strategy a corresponding probability of grasp success is computed. The grasp strategies and corresponding probabilities of grasp success are used to determine and implement a plan to autonomously operate a robotic structure to pick one or more items from the workplace and place each item singly in a corresponding location in a singulation conveyance structure. ## Claims ### Claim 1 A system, comprising: a communication interface; and a processor coupled to the communication interface and configured to: receive sensor data via the communication interface, the sensor data including image data associated with a workspace; use the sensor data to generate a three dimensional view of at least a portion of the workspace, the three dimensional view including boundaries of a plurality of items present in the workspace; determine that no available grasp strategies exist for grasping an item among the plurality of items in the workspace; in response to a determination that no available grasp strategies exist for grasping an item in the workspace, determine a plan to operate robotic structure to change a state of the plurality of items or a flow of items into the workspace; cause the robotic structure to implement the plan to change a state of the plurality of items or a flow of items into the workspace; and in response to changing the state of the plurality of items or the flow of items, for each item of at least a subset of items, determine a grasp strategy for a current state of the item; for each grasp strategy, determine a probability of grasp success, wherein the probability of grasp success is computed based at least in part on learned data corresponding to historical attempts to grasp a similar item; and use the grasp strategies and corresponding probabilities of grasp success to determine and implement a plan to autonomously operate a robotic structure to pick one or more items from the workspace and place each item singly in a corresponding location in a singulation conveyance structure. ### Claim 2 (depends on 1) The system of claim 1, wherein determining that no available grasp strategies exist for grasping the item among the plurality of items in the workspace comprises: determining that none of the plurality of items present in the workspace can be successfully grasped based on a determination that none of the plurality of items have a corresponding grasp strategy for which the probability of grasp success exceeds a predefined minimum success probability threshold. ### Claim 3 (depends on 1) The system of claim 1, wherein the learned data includes information pertaining to one or more of (i) a position of the subset of items in relation to one or more other items of the plurality of items, (ii) an extent of item overlap, and (iii) item orientation. ### Claim 4 (depends on 1) The system of claim 1, wherein the using the grasp strategies and corresponding probability of grasp success comprises resume autonomous operation to implement the plan to pick the one or more items from the workspace and place each item singly in a corresponding location in the singulation conveyance structure. ### Claim 5 (depends on 1) The system of claim 1, wherein the processor is configured to: determine whether the workspace comprises an item for which the corresponding probability of grasp success is satisfies a preset minimum threshold; and in response to a determination that the workspace does not comprise an item for which the corresponding probability of grasp success is satisfies the preset minimum threshold, cause at least a subset of the plurality of items present in the workspace to be disrupted. ### Claim 6 (depends on 1) The system of claim 1, wherein causing the robotic structure to implement the plan to change a state of the plurality of items or a flow of items into the workspace comprises using the robotic structure to nudge, pull, or push one or more items into a different position in the workspace. ### Claim 7 (depends on 1) The system of claim 1, wherein the processor is configured to update the three dimensional view, a grasp strategy, and a corresponding probability of grasp success in response to an item being picked from the workspace and placed in the corresponding location in the singulation conveyance structure. ### Claim 8 (depends on 1) The system of claim 1, wherein the robotic structure comprises a robotic arm; the robotic arm comprises a suction-based end effector configured to be used to grasp the one or more items, the suction-based end effector including two or more separately-actuated sets of suction cups. ### Claim 9 (depends on 8) The system of claim 8, wherein each set of suction cups comprises one or more suction cups, and the processor is configured to use two or more of the separately-actuated sets of suction cups to each contemporaneously grasp a corresponding one of the items from the workspace. ### Claim 10 (depends on 9) The system of claim 9, wherein the two or more of the separately-actuated sets of suction cups move the items together to a destination location of a first one of the items. ### Claim 11 (depends on 9) The system of claim 9, wherein one of the separately-actuated sets of suction cups releases a first item at a destination for the first item. ### Claim 12 (depends on 1) The system of claim 1, wherein the processor is configured to iteratively receive sensors data, use the sensor data to generate a three dimensional view of at least a portion of the workspace, determine for each of at least a subset of items then present in the workspace a corresponding grasp strategy and for each grasp strategy a corresponding probability of grasp success, and use the grasp strategies and corresponding probabilities of grasp success to determine and implement a plan to autonomously operate a robotic structure to pick a next one or more items from the workspace and place each item singly in a corresponding location in a singulation conveyance structure. ### Claim 13 (depends on 12) The system of claim 12, wherein the processor is configured to interrupt operation based at least in part on a determination that no further item is able to be grasped autonomously despite one or more items remaining present in the workspace. ### Claim 14 (depends on 1) The system of claim 1, wherein the processor is configured to determine the probabilities of grasp success based at least in part on one or more of an item attribute for an item and a degree of overlap of the item by one or more other items. ### Claim 15 (depends on 1) The system of claim 1, wherein the processor is configured to determine one or more candidate grasp strategies for an item, compute a probability of grasp strategy success for each candidate grasp strategy, and use the respective computed probabilities of grasp strategy success to select a selected best grasp strategy for the item. ### Claim 16 (depends on 1) The system of claim 1, wherein the workspace comprises a chute or other receptacle, the robotic structure comprises one of a plurality of robotic structures associated with the workspace, and the processor is configured to coordinate operation of the robotic structures to pick and place items from the workspace. ### Claim 17 (depends on 1) The system of claim 1, wherein the workspace comprises a first workspace included in a plurality of workspaces associated with the singulation conveyance structure, each workspace having one or more robotic structures associated therewith, and wherein the processor is configured to operate the respective robotic structures associated with each workspace to pick items from the workspace and place them singly on the singulation conveyance structure. ### Claim 18 (depends on 17) The system of claim 17, wherein the processor is configured to coordinate operation of the respective robotic structures associated with each workspace to maximize collective throughput of placement of items singly on the singulation conveyance structure. ### Claim 19 A method, comprising: receiving sensor data via a communication interface, the sensor data including image data associated with a workspace; using the sensor data to generate a three dimensional view of at least a portion of the workspace, the three dimensional view including boundaries of a plurality of items present in the workspace; determining that no available grasp strategies exist for grasping an item among the plurality of items in the workspace; in response to a determination that no available grasp strategies exist for grasping an item in the workspace, determining a plan to operate robotic structure to change a state of the plurality of items or a flow of items into the workspace; causing the robotic structure to implement the plan to change a state of the plurality of items or a flow of items into the workspace; and in response to changing the state of the plurality of items or the flow of items, for each item of at least a subset of items, determining a grasp strategy for a current state of the item; for each grasp strategy, determining probability of grasp success, wherein the corresponding probability of grasp success is computed based at least in part on learned data corresponding to historical attempts to grasp a similar item; and using the grasp strategies and corresponding probabilities of grasp success to determine and implement a plan to autonomously operate a robotic structure to pick one or more items from the workspace and place each item singly in a corresponding location in a singulation conveyance structure. ### Claim 20 A computer program product embodied in a non-transitory computer readable medium, comprising computer instructions for: receiving sensor data via a communication interface, the sensor data including image data associated with a workspace; using the sensor data to generate a three dimensional view of at least a portion of the workspace, the three dimensional view including boundaries of a plurality of items present in the workspace; determining that no available grasp strategies exist for grasping an item among the plurality of items in the workspace; in response to a determination that no available grasp strategies exist for grasping an item in the workspace, determining a plan to operate robotic structure to change a state of the plurality of items or a flow of items into the workspace; causing the robotic structure to implement the plan to change a state of the plurality of items or a flow of items into the workspace; and in response to changing the state of the plurality of items or the flow of items, for each item of at least a subset of items, determining a grasp strategy for a current state of the item; for each grasp strategy, determining a probability of grasp success, wherein the corresponding probability of grasp success is computed based at least in part on learned data corresponding to historical attempts to grasp a similar item; and using the grasp strategies and corresponding probabilities of grasp success to determine and implement a plan to autonomously operate a robotic structure to pick one or more items from the workspace and place each item singly in a corresponding location in a singulation conveyance structure.