# US 11,806,880 B2 · Robotic handling of soft products in non-rigid packaging > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,806,880 B2 - Document ID: US11806880B2 - Status: Granted - Kind: B2 - Application number: 16/797,359 - Filed: 2020-02-21 - Published: 2020-08-27 - Granted: 2023-11-07 - Earliest claimed: 2019-02-22 - Assignees: Dexterity, Inc. - Applicants: Dexterity, Inc. - Inventors: Kevin Jose Chavez; Robert Hegdahl; Yuan Gao; Andrew Lovett; Roger Lam Hau; Samir Menon - Web page: https://patentgrep.com/patent/US11806880B2 ## CPC - B25J 9/1687: {Assembly, peg and hole, palletising, straight line, weaving pattern movement} - B25J 15/0052: {multiple gripper units or multiple end effectors} - B25J 9/1612: {characterised by the hand, wrist, grip control} - B25J 9/1689: {Teleoperation} - B25J 9/1694: {characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion} - B25J 9/1697: {Vision controlled systems} - B25J 11/0045: {Manipulators used in the food industry} - B25J 15/0616: {with vacuum} - B25J 15/0683: {Details of suction cup structure, e.g. grooves or ridges} - B25J 19/023: {including video camera means} - G05B 2219/39469: Grip flexible, deformable plate, object and manipulate it - G05B 2219/39558: Vacuum hand has selective gripper area - G06V 20/64: Three-dimensional [3D] objects - G06V 2201/06: Recognition of objects for industrial automation - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Techniques are disclosed to perform robotic handling of soft products in non-rigid packaging. In various embodiments, sensor data associated with a workspace is received. An action to be performed in the workspace using one or more robotic elements is determined, the action including moving an end effector of one of the robotic elements relatively quickly to a location in proximity to an item to be grasped; actuating a grasping mechanism of the end effector to grasp the item using an amount of force and structures associated with minimized risk of damage to one or both of the item and its packaging; and using sensor data generated subsequent to the item being grasped to ensure the item has been grasped securely. Control communications are sent to the robotic element via the communication interface to cause robotic element to perform the action. ## Claims ### Claim 1 A robotic system, comprising: a communication interface; and a processor coupled to the communication interface and configured to: receive via the communication interface sensor data associated with a workspace in which one or more robotic elements controlled at least in part by the robotic system are present, wherein the sensor data is received from a plurality of different types of sensors; determine based at least in part on the sensor data an action to be performed in the workspace using one or more of the robotic elements, the action including moving an end effector of a robotic element of the one or more robotic elements to a location within a prescribed distance of a surface of an item to be grasped; monitoring the sensor data to determine that the end effector has reached the location within the prescribed distance of the surface of the item; stopping gross movement of the end effector based at least in part on the determination that the end effector has reached the location within the prescribed distance of the surface of the item; actuating a grasping mechanism of the end effector, subsequent to stopping gross movement of the end effector, to grasp the item using an amount of force and structures associated with minimized risk of damage to one or both of the item and its packaging; and using sensor data generated by the plurality of different types of sensors subsequent to the item being grasped to ensure the item has been grasped securely at least in part by determining whether sensor data generated by a threshold proportion of the different types of sensors indicates the grasp of the item remains secure and moving the item to a buffer location in response to determining that the sensor data generated by the threshold proportion of the different types of sensors does not indicate that the grasp of the item remains secure; and send control communications to the robotic element via the communication interface to cause the robotic element to perform the action. ### Claim 2 (depends on 1) The robotic system of claim 1, wherein the processor is further configured to use the robotic element to move the grasped item to a destination location. ### Claim 3 (depends on 1) The robotic system of claim 1, wherein the end effector comprises a suction-based grasping mechanism. ### Claim 4 (depends on 3) The robotic system of claim 3, wherein the end effector comprises one or more suction cups. ### Claim 5 (depends on 4) The robotic system of claim 4, wherein the one or more suction cups comprises a compliant material that gives way upon coming in contact with the item. ### Claim 6 (depends on 1) The robotic system of claim 1, wherein the sensor data comprise image data generated by one or more cameras. ### Claim 7 (depends on 1) The robotic system of claim 1, wherein the sensor data comprise data generated by one or more of the following: pressure sensors, force sensors, torque sensors, and contact sensors. ### Claim 8 (depends on 1) The robotic system of claim 1, wherein the processor is further configured to use the robotic element to move the grasped item to a destination location and place the item at the destination location. ### Claim 9 (depends on 8) The robotic system of claim 8, wherein the processor is configured to place the item at least in part by moving the item to the destination location, in a first phase of operation, and using the sensor data to adjust the position to the item to be more closely adjacent to a structure or second item adjacent to the destination location. ### Claim 10 (depends on 1) The robotic system of claim 1, wherein the processor is configured to determine the action based at least in part on a plan. ### Claim 11 (depends on 10) The robotic system of claim 10, wherein the processor is configured to generate the plan based on a high level objective. ### Claim 12 (depends on 11) The robotic system of claim 11, wherein the high level objective includes an identification of a plurality of items and for each item a corresponding quantity to be included in a set of items associated with an end destination. ### Claim 13 (depends on 12) The robotic system of claim 12, wherein the items comprise different bread products and the high level objective comprises a manifest or other listing or items and quantities of each to be delivered to a retail store or other destination. ### Claim 14 (depends on 1) The robotic system of claim 1, wherein the processor is further configured to use the robotic element to move the grasped item to a destination location and place the item at the destination location, including by using subsequently-received sensor information to verify placement of the item prior to releasing the item from the grasping mechanism of the end effector. ### Claim 15 (depends on 1) The robotic system of claim 1, wherein the processor uses sensor information to verify placement of the item prior to releasing the item from the grasping mechanism of the end effector at least in part by using the sensor data to determine a height of the product above a height reference and comparing the determined value to an expected value associated with successful placement. ### Claim 16 (depends on 1) The robotic system of claim 1, wherein the end effector includes a plurality of independently-actuated sets of one or more suction cups and the processor is further configured to control the robotic element to grasp simultaneously two or more items, each using one or more of the independently-actuated sets of one or more suction cups. ### Claim 17 (depends on 16) The robotic system of claim 16, wherein the processor is further configured to determine an optimal plan to pick and place a plurality of items, including by determining possible combinations of two or more items capable of being grasped simultaneously and determine an optimal plan that includes a mix of single-grasp and multiple-grasp pick and place actions. ### Claim 18 (depends on 1) The robotic system of claim 1, wherein the processor is further configured to determine a strategy to grasp the item using the robotic element. ### Claim 19 A method, comprising: receiving via a communication interface sensor data associated with a workspace in which one or more robotic elements controlled at least in part by the robotic system are present, wherein the sensor data is received from a plurality of different types of sensors; determining based at least in part on the sensor data an action to be performed in the workspace using one or more of the robotic elements, the action including moving an end effector of a robotic element of the one or more robotic elements to a location within a prescribed distance of a surface of an item to be grasped; monitoring the sensor data to determine that the end effector has reached the location within the prescribed distance of the surface of the item; stopping gross movement of the end effector based at least in part on the determination that the end effector has reached the location within the prescribed distance of the surface of the item; actuating a grasping mechanism of the end effector, subsequent to stopping gross movement of the end effector, to grasp the item using an amount of force and structures associated with minimized risk of damage to one or both of the item and its packaging; and using sensor data generated by the plurality of different types of sensors subsequent to the item being grasped to ensure the item has been grasped securely at least in part by determining whether sensor data generated by a threshold proportion of the different types of sensors indicates the grasp of the item remains secure and moving the item to a buffer location in response to determining that the sensor data generated by the threshold proportion of the different types of sensors does not indicate that the grasp of the item remains secure; and sending control communications to the robotic element via the communication interface to cause the robotic element to perform the action. ### Claim 20 A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for: receiving via a communication interface sensor data associated with a workspace in which one or more robotic elements controlled at least in part by the robotic system are present, wherein the sensor data is received from a plurality of different types of sensors; determining based at least in part on the sensor data an action to be performed in the workspace using one or more of the robotic elements, the action including moving an end effector of a robotic element of the one or more robotic elements to a location within a prescribed distance of a surface of an item to be grasped; monitoring the sensor data to determine that the end effector has reached the location within the prescribed distance of the surface of the item; stopping gross movement of the end effector based at least in part on the determination that the end effector has reached the location within the prescribed distance of the surface of the item; actuating a grasping mechanism of the end effector, subsequent to stopping gross movement of the end effector, to grasp the item using an amount of force and structures associated with minimized risk of damage to one or both of the item and its packaging; and using sensor data generated by the plurality of different types of sensors subsequent to the item being grasped to ensure the item has been grasped securely at least in part by determining whether sensor data generated by a threshold proportion of the different types of sensors indicates the grasp of the item remains secure and moving the item to a buffer location in response to determining that the sensor data generated by the threshold proportion of the different types of sensors does not indicate that the grasp of the item remains secure; and sending control communications to the robotic element via the communication interface to cause the robotic element to perform the action.