# US 10,766,149 B2 · Optimization-based spring lattice deformation model for soft materials > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,766,149 B2 - Document ID: US10766149B2 - Status: Granted - Kind: B2 - Application number: 15/934,498 - Filed: 2018-03-23 - Published: 2019-09-26 - Granted: 2020-09-08 - Earliest claimed: 2018-03-23 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: Andrew D. Marchese; William Clay Flannigan; Parris S. Wellman - Web page: https://patentgrep.com/patent/US10766149B2 ## CPC - B25J 15/0616: {with vacuum} - B25J 9/0009: {Constructional details, e.g. manipulator supports, bases} - B25J 9/02: characterised by movement of the arms, e.g. cartesian coordinate type (B25J9/06 takes precedence) - B25J 9/1605: {Simulation of manipulator lay-out, design, modelling of manipulator} - B25J 9/1612: {characterised by the hand, wrist, grip control} - B25J 9/1661: {characterised by task planning, object-oriented languages} - B25J 9/1669: {characterised by special application, e.g. multi-arm co-operation, assembly, grasping} - B25J 9/1697: {Vision controlled systems} - B25J 15/0019: {End effectors other than grippers} - G05B 2219/39484: Locate, reach and grasp, visual guided grasping - G05B 2219/39514: Stability of grasped objects - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract Techniques for controlling a robotic picking arm using estimated seal quality metrics. A plurality of candidate contact points for holding an item using a suction device of the robotic picking arm, based on captured images of the item and an n-dimensional surface model of the item. An expected seal quality metric for a first one of the candidate contact points, by processing the n-dimensional surface model of the item and physical properties of the suction device of the robotic picking arm. Based on the expected seal quality metric, embodiments can determine whether to retrieve the item from the storage container by holding the item at the first candidate contact point using the suction device of the robotic picking arm. ## Claims ### Claim 1 A method, comprising: capturing one or more images of an item within a storage container to be retrieved using a robotic picking arm having a suction device; retrieving a n-dimensional surface model of the item, based on a unique identifier corresponding to the item; determining a plurality of candidate contact points for holding the item using the suction device of the robotic picking arm, based on the captured one or more images of the item and the n-dimensional surface model of the item; determining an expected seal quality metric for a first one of the candidate contact points, by processing the n-dimensional surface model of the item and physical properties of the suction device of the robotic picking arm as inputs, comprising: modelling deformation and contact between the suction device and the n-dimensional surface model at the first candidate contact point, using at least one n-dimensional spring-lattice to model deformation and contact between the suction device and the n-dimensional surface model at the first candidate contact point, such that potential energy across the n-dimensional spring-lattice is minimized; and calculating the expected seal quality metric, using the n-dimensional spring-lattice and a fluids model relating fluid force and leak pathways; and determining, based on the expected seal quality metric, whether to retrieve the item from the storage container by holding the item at the first candidate contact point using the suction device of the robotic picking arm. ### Claim 2 (depends on 1) The method of claim 1, further comprising: upon determining to retrieve the item from the storage container by holding the item at the first candidate contact point using the suction device of the robotic picking arm, transmitting an instruction to control logic for the robotic picking arm, the instruction specifying at least location information identifying the first candidate contact point. ### Claim 3 (depends on 1) The method of claim 1, further comprising: upon determining not to retrieve the item from the storage container by holding the item at the first candidate contact point using the suction device of the robotic picking arm: selecting a second one of the plurality of candidate contact points; determining a second expected seal quality metric for the second candidate contact point; and determining, based on the second expected seal quality metric, whether to retrieve the item from the storage container by holding the item at the second candidate contact point using the suction device of the robotic picking arm. ### Claim 4 (depends on 3) The method of claim 3, wherein determining a second expected seal quality metric for the second candidate contact point further comprises: modelling deformation and contact between the suction device and the second candidate contact point, using at least one n-dimensional spring-lattice to model deformation and contact between the suction device and the second candidate contact point for the item, such that potential energy across the n-dimensional spring-lattice is minimized; and calculating the expected seal quality metric, using the n-dimensional spring-lattice for the second candidate contact point and the fluids model. ### Claim 5 (depends on 1) The method of claim 1, further comprising: determining a pose of the item within the storage container, based on the captured one or more images of the item and the retrieved n-dimensional surface model for the item, wherein determining the plurality of candidate contact points for holding the item using the suction device of the robotic picking arm, is further based on the determined pose of the item within the storage container. ### Claim 6 (depends on 5) The method of claim 5, wherein determining the plurality of candidate contact points for holding the item using the suction device of the robotic picking arm further comprises: retrieving an item profile, based on the unique identifier corresponding to the item, wherein the item profile specifies a set of candidate contact points for the item; and determining that the plurality of candidate contact points are available, based on the determined pose of the item within the storage container, wherein the plurality of candidate contact points is a subset of the set of candidate contact points for the item within the item profile. ### Claim 7 A system, comprising: a robotic picking arm having a suction device adapted for retrieving items and reporting the items to a determined location; one or more computer processors; and control logic for the robotic picking arm that, when executed by operation of the one or more computer processors, performs an operation, comprising: determining a first candidate contact point for holding a first item using the suction device of the robotic picking arm, based on a surface geometry of the first item; estimating a seal quality metric for the first candidate contact point, based on a predicted deformed n-dimensional shape of the suction device and a n-dimensional shape associated with the first item, comprising generating the predicted deformed n-dimensional shape of the suction device by modelling deformation and contact between the suction device and the n-dimensional shape associated with the first item at the first candidate contact point, using at least a first n-dimensional spring-lattice to model deformation and contact between the suction device and the n-dimensional shape associated with the first item at the first candidate contact point, such that potential energy across the first n-dimensional spring-lattice is minimized; and upon determining that the estimated seal quality metric for the first candidate contact point satisfies one or more predefined criteria, controlling movement of the robotic picking arm to retrieve the first item by holding the first item at the first candidate contact point. ### Claim 8 (depends on 7) The system of claim 7, wherein a first predefined criteria of the one or more predefined criteria further comprises the estimated seal quality metric exceeding a predefined threshold level of seal quality. ### Claim 9 (depends on 7) The system of claim 7, wherein estimating the seal quality metric for the first candidate contact point, based on the predicted deformed n-dimensional shape of the suction device and the n-dimensional shape associated with the first item, further comprises calculating the seal quality metric, using the first n-dimensional spring-lattice and a fluids model relating fluid force and leak pathways. ### Claim 10 (depends on 9) The system of claim 9, wherein estimating the seal quality metric for the first candidate contact point, based on the predicted deformed n-dimensional shape of the suction device and the n-dimensional shape associated with the first item, further comprises: retrieving an item profile corresponding the first item and specifying data describing physical attributes of the first item; and generating a second predicted deformed n-dimensional shape for the first item, using at least a second n-dimensional spring-lattice to model deformation and contact between the suction device and the first candidate contact point of the first item, such that potential energy across the first n-dimensional spring-lattice is minimized, wherein calculating the seal quality metric is further based on the second predicted deformed n-dimensional shape for the first item. ### Claim 11 (depends on 9) The system of claim 9, wherein estimating the seal quality metric for the first candidate contact point, based on the predicted deformed n-dimensional shape of the suction device and the n-dimensional shape associated with the first item, further comprises: determining a second n-dimensional shape associated with the first item, wherein calculating the seal quality metric is further based on the second n-dimensional shape. ### Claim 12 (depends on 7) The system of claim 7, the operation further comprising: determining a second candidate contact point for holding a second item using the suction device of the robotic picking arm, based on a second surface geometry of the second item; estimating a second seal quality metric for the second candidate contact point, based on a predicted deformed n-dimensional shape of the suction device and a n-dimensional shape associated with the second item; and upon determining that the second seal quality metric for the second candidate contact point does not satisfy the one or more predefined criteria: determining a third candidate contact point for holding the second item using the suction device of the robotic picking arm, based on the second surface geometry of the second item estimating a third seal quality metric for the third candidate contact point, based on a third deformed n-dimensional shape of the suction device and a third n-dimensional shape associated with the second item; and upon determining that the third seal quality metric for the second candidate contact point does satisfies the one or more predefined criteria, controlling movement of the robotic picking arm to retrieve the second item by holding the second item at the third candidate contact point. ### Claim 13 (depends on 7) The system of claim 7, the operation further comprising: capturing one or more images of first item; determining a unique identifier corresponding to the first item within a product catalog for a fulfillment center; retrieving a n-dimensional model of the first item, using the unique identifier; and determining the surface geometry of the first item, based on at least one of the captured one or more images of the first item and the retrieved n-dimensional model of the first item. ### Claim 14 A non-transitory computer-readable medium containing computer program code that, when executed by operation of one or more computer processors, performs an operation comprising: determining a plurality of candidate contact points for holding a first item using a suction device of a robotic picking arm, based on an estimated surface geometry of the first item; calculating, for each of the plurality of candidate contact points, a respective seal quality metric, based on a predicted deformed n-dimensional shape of the suction device and a n-dimensional shape associated with the candidate contact point on the first item, comprising: modelling deformation and contact between the suction device and the respective candidate contact point, using at least one n-dimensional spring-lattice to model deformation and contact between the suction device and the respective candidate contact point on the first item; and calculating the respective seal quality metric, using the n-dimensional spring-lattice and a fluids model; selecting one of the plurality of candidate contact points, based on the calculated seal quality metrics; and controlling movement of the robotic picking arm to retrieve the first item by holding the first item at the selected candidate contact point. ### Claim 15 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the operation further comprises: capturing one or more images of the first item, using one or more camera devices; retrieving a n-dimensional model of the first item, using a unique identifier corresponding to the first item; and estimating a pose of the first item, based on the captured one or more images of the first item, the retrieved n-dimensional model of the first item, and positional information associated with the one or more camera devices. ### Claim 16 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the estimated surface geometry of the first item is based on the estimated pose of the first item. ### Claim 17 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein selecting one of the plurality of candidate contact points, based on the calculated seal quality metrics, further comprises: determining at least one of the plurality of candidate contact points that satisfies one or more predefined criteria; and selecting an optimal candidate contact point, from the determined at least one candidate contact point. ### Claim 18 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein calculating, for each of the plurality of candidate contact points, the respective seal quality metric, further comprises: iteratively refining a n-dimensional shape of a first one of the at least one n-dimensional spring-lattice to minimize potential energy across the first n-dimensional spring-lattice, wherein a resulting n-dimensional shape of the first n-dimensional spring-lattice represents the n-dimensional shape of the suction device when suction is applied to the respective candidate contact point of the first item. ### Claim 19 (depends on 18) The non-transitory computer-readable medium of claim 18, wherein calculating, for each of the plurality of candidate contact points, the respective seal quality metric, further comprises: iteratively refining a second n-dimensional shape of a second one of the at least one n-dimensional spring-lattice to minimize potential energy across the second n-dimensional spring-lattice, wherein a resulting n-dimensional shape of the second n-dimensional spring-lattice represents the n-dimensional shape for the respective candidate contact point on the first item when suction is applied using the suction device, wherein calculating the respective seal quality metric is further based on the n-dimensional shape for the respective candidate contact point on the first item when suction is applied using the suction device. ### Claim 20 (depends on 18) The non-transitory computer-readable medium of claim 18, wherein calculating the respective seal quality metric is further based on an n-dimensional shape for the respective candidate contact point on the first item when no suction is applied. ### Claim 21 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein: the operation further comprises retrieving an item profile using a unique identifier corresponding to the first item, the item profile specifying a set of candidate contact points for the first item; and the plurality of candidate contact points are a subset of the set of candidate contact points for the first item within the item profile.