# US 10,759,054 B1 · Method and system for handling deformable objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,759,054 B1
- Document ID: US10759054B1
- Status: Granted
- Kind: B1
- Application number: 16/802,375
- Filed: 2020-02-26
- Granted: 2020-09-01
- Earliest claimed: 2020-02-26
- Assignees: Grey Orange Pte. Ltd.
- Applicants: Grey Orange Pte. Ltd.
- Inventors: Akash Patil; Avilash Kumar; Sumit Tiwary; Manish Soni; Nikhil Sorout; Sameer Narkar; Anirudh Shekhawat; Vaibhav Tolia; Daniel Echeverria; Andreas Hofmann; Mathew Livianu; Robert Pitha; Shawn Schaffert; Andrew Kiruluta; Andrew Lewis
- Web page: https://patentgrep.com/patent/US10759054B1

## CPC

- B25J 9/1682: {Dual arm manipulator; Coordination of several manipulators}
- B25J 9/0087: {Dual arms (double SCARA arms B25J9/043)}
- B25J 9/1697: {Vision controlled systems}
- B25J 15/0014: {having fork, comb or plate shaped means for engaging the lower surface on a object to be transported}
- G05B 13/028: {using expert systems only}
- G06T 1/0014: {Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023)}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G05B 2219/39001: Robot, manipulator control
- G05B 2219/39135: For multiple manipulators operating at same time, avoid collision
- G05B 2219/39469: Grip flexible, deformable plate, object and manipulate it
- G05B 2219/39558: Vacuum hand has selective gripper area
- G05B 2219/40053: Pick 3-D object from pile of objects
- G05B 2219/40564: Recognize shape, contour of object, extract position and orientation
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A control server controls a dual-arm robotic manipulator (DARM) for handling deformable objects in a stack. The control server receives a set of images of the stack captured by a set of image sensors, and determines a contour of the stack based the set of images. Based on the contour and historical data associated with the deformable objects in the stack, the control server determines a sequence of actions to be performed by the DARM for handling a first deformable object in the stack, and controls the DARM to handle the first deformable object by communicating a set of commands corresponding to each action in sequence of actions. The first deformable object is handled such that original form factors of the first deformable object and the remaining stack are maintained.

## Claims

### Claim 1

A system for handling one or more deformable objects that are arranged in a stack, the system comprising:
a dual-arm robotic manipulator comprising:
first and second robotic arms; and
first and second end effectors connected to the first and second robotic arms, respectively;
a set of image sensors configured to capture a first set of images of the stack; and
a control server in communication with the dual-arm robotic manipulator and the set of image sensors, the control server configured to:
detect the one or more deformable objects arranged in the stack based on the first set of images captured by the set of image sensors;
determine a contour of the detected one or more deformable objects based on the first set of images;
determine, based on the contour, a sequence of a plurality of actions to be performed by the dual-arm robotic manipulator for handling a first deformable object in the stack, wherein the handling includes a pick operation to be executed on the first deformable object; and
control, in the pick operation based on the determined sequence of the plurality of actions:
the first robotic arm to grip the first deformable object from a gripping end by way of the first end effector, and lift the gripped end to a predetermined height to partially lift the first deformable object;
the second robotic arm to slide the second end effector beneath the partially lifted first deformable object, and lift the first deformable object in entirety; and
the first robotic arm to release the grip of the first end effector on the first deformable object for successfully executing the pick operation.

### Claim 2 (depends on 1)

The system of claim 1, wherein the handling further includes a put-down operation to be executed on the successfully picked first deformable object, and wherein, in the put-down operation, the control server is further configured to control, based on the determined sequence of the plurality of actions, the second robotic arm to put-down the successfully picked first deformable object at a destination location.

### Claim 3 (depends on 2)

The system of claim 2, further comprising a database associated with the control server, wherein the control server is further configured to store, upon successful handling of the first deformable object, plan information associated with the sequence of the plurality of actions in the database.

### Claim 4 (depends on 1)

The system of claim 1, wherein the sequence of the plurality of actions to be performed by the dual-arm robotic manipulator is further determined based on historical data associated with the one or more deformable objects, wherein the historical data includes at least one of a set of physical attributes of each of the one or more deformable objects or information associated with previous handling of the one or more deformable objects, and wherein the set of physical attributes of each deformable object includes at least one of a shape, a size, a weight, a set of dimensions, a number of folds, or actual depth information, of the corresponding deformable object.

### Claim 5 (depends on 4)

The system of claim 4, wherein the control server is further configured to:
identify one or more layers in the contour, wherein a first layer in the contour corresponds to a fold of the first deformable object or the first deformable object in entirety;
identify, based on the contour, perceived depth information of each of the one or more deformable objects; and
determine an orientation of the first deformable object with respect to the stack based on the perceived depth information of each of the one or more deformable objects and the historical data, wherein the sequence of the plurality of actions to be performed by the dual-arm robotic manipulator is further determined based on the one or more layers in the contour and the orientation of the first deformable object.

### Claim 6 (depends on 5)

The system of claim 5, wherein the control server is further configured to identify the gripping end based on the contour, the one or more layers, the orientation of the first deformable object, and the historical data.

### Claim 7 (depends on 1)

The system of claim 1, further comprising:
a storage unit having a plurality of shelves, wherein the one or more deformable objects are arranged in the stack on a first shelf of the plurality of shelves; and
a mobile robot that is configured to transport to the storage unit from a first location to a second location that is within an operational range of the dual-arm robotic manipulator.

### Claim 8 (depends on 1)

The system of claim 1, wherein the set of image sensors is further configured to capture a second set of images of the stack, while the first robotic arm lifts the gripping end of the first deformable object, and wherein the control server is further configured to determine, based on the second set of images, whether the gripping end of the first deformable object is lifted to the predetermined height.

### Claim 9 (depends on 1)

The system of claim 1, wherein the second end effector comprises a set of pressure sensors that is configured to record, when the second end effector lifts the first deformable object, pressure data corresponding to pressure exerted by the lifted first deformable object on the second end effector, wherein the control server is further configured to determine whether the second end effector has accurately lifted the first deformable object based on the recorded pressure data, and wherein the accurate lifting of the first deformable object corresponds to a uniform weight distribution of the first deformable object on the second end effector.

### Claim 10 (depends on 1)

The system of claim 1, wherein the first end effector is a vacuum gripper, and wherein the second end effector includes a spatula-shaped base, an adjustable arm connected to the spatula-shaped base, and a roller connected to the adjustable arm such that the roller is oriented parallel to and at an adjustable height above the spatula-shaped base.

### Claim 11

A method for handling one or more deformable objects that are arranged in a stack, the method comprising:
detecting, by a control server, the one or more deformable objects arranged in the stack based on a first set of images of the stack captured by a set of image sensors;
determining, by the control server, a contour of the detected one or more deformable objects based on the first set of images;
determining, by the control server, based on the contour, a sequence of a plurality of actions to be performed by a dual-arm robotic manipulator for handling a first deformable object in the stack, wherein the handling includes a pick operation to be executed on the first deformable object; and
controlling, by the control server in the pick operation based on the determined sequence of the plurality of actions:
a first robotic arm of the dual-arm robotic manipulator to grip the first deformable object from a gripping end by way of a first end effector connected to the first robotic arm, and lift the gripped end to a predetermined height to partially lift the first deformable object;
a second robotic arm of the dual-arm robotic manipulator to slide a second end effector connected to the second robotic arm beneath the partially lifted first deformable object, and lift the first deformable object in entirety; and
the first robotic arm to release the grip of the first end effector on the first deformable object for successfully executing the pick operation.

### Claim 12 (depends on 11)

The method of claim 11, wherein the handling further includes a put-down operation to be executed on the successfully picked first deformable object, and wherein, in the put-down operation based on the determined sequence of the plurality of actions, the second robotic arm is further controlled by the control server to place the successfully picked first deformable object at a destination location.

### Claim 13 (depends on 12)

The method of claim 12, further comprising storing, by the control server upon successful handling of the first deformable object, plan information associated with the sequence of the plurality of actions in a database associated with the control server.

### Claim 14 (depends on 11)

The method of claim 11, wherein the sequence of the plurality of actions to be performed by the dual-arm robotic manipulator is further determined based on historical data associated with the one or more deformable objects, wherein the historical data includes at least one of a set of physical attributes of each of the one or more deformable objects or information associated with previous handling of the one or more deformable objects, and wherein the set of physical attributes of each deformable object includes at least one of a shape, a size, a weight, a set of dimensions, a number of folds, or actual depth information, of the corresponding deformable object.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
identifying, by the control server, one or more layers in the contour, wherein a first layer in the contour corresponds to a fold of the first deformable object or the first deformable object in entirety;
identifying, by the control server based on the contour, perceived depth information of each of the one or more deformable objects; and
determining, by the control server, an orientation of the first deformable object with respect to the stack based on the perceived depth information of each of the one or more deformable objects and the historical data, wherein the sequence of the plurality of actions to be performed by the dual-arm robotic manipulator is further determined based on the one or more layers in the contour and the orientation of the first deformable object.

### Claim 16 (depends on 15)

The method of claim 15, further comprising identifying, by the control server, the gripping end based on the contour, the one or more layers, the orientation of the first deformable object, and the historical data.

### Claim 17 (depends on 11)

The method of claim 11, further comprising determining, by the control server, whether the gripping end of the first deformable object is lifted to the predetermined height based on a second set of images of the stack captured by the set of image sensors, wherein the second set of images of the stack is captured by the set of image sensors while the first robotic arm lifts the gripping end of the first deformable object.

### Claim 18 (depends on 11)

The method of claim 11, further comprising determining, by the control server, whether the second end effector has accurately lifted the first deformable object based on pressure data recorded by a set of pressure sensors installed on the second end effector, wherein the pressure data corresponds to pressure exerted by the first deformable object on the second end effector when the first deformable object is lifted by the second end effector, and wherein the accurate lifting of the first deformable object corresponds to a uniform weight distribution of the first deformable object on the second end effector.

### Claim 19

A system for handling one or more deformable objects that are arranged in a stack, the system comprising:
a control server configured to:
detect the one or more deformable objects arranged in the stack based on a first set of images of the stack captured by a set of image sensors;
determine a contour of the detected one or more deformable objects based on the first set of images;
determine, based on the contour, a sequence of a plurality of actions to be performed by a dual-arm robotic manipulator for handling a first deformable object in the stack, wherein the handling includes a pick operation to be executed on the first deformable object; and
control, in the pick operation based on the determined sequence of the plurality of actions:
a first robotic arm of the dual-arm robotic manipulator to grip the first deformable object from a gripping end by way of a first end effector connected to the first robotic arm, and lift the gripped end to a predetermined height to partially lift the first deformable object;
a second robotic arm of the dual-arm robotic manipulator to slide a second end effector connected to the second robotic arm beneath the partially lifted first deformable object, and lift the first deformable object in entirety; and
the first robotic arm to release the grip of the first end effector on the first deformable object for successfully executing the pick operation.

### Claim 20 (depends on 19)

The system of claim 19, wherein the handling further includes a put-down operation to be executed on the successfully picked first deformable object, and wherein, in the put-down operation based on the determined sequence of the plurality of actions, the control server is further configured to control the second robotic arm to place the successfully picked first deformable object at a destination location.