# US 12,214,975 B2 · State estimation using geometric data and vision system for palletizing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,214,975 B2 - Document ID: US12214975B2 - Status: Granted - Kind: B2 - Application number: 17/837,775 - Filed: 2022-06-10 - Published: 2022-12-22 - Granted: 2025-02-04 - Earliest claimed: 2021-06-16 - Assignees: Dexterity, Inc. - Applicants: Dexterity, Inc. - Inventors: Rohit Arka Pidaparthi; William Arthur Clary; Neeraja Abhyankar; Jonathan Kuck; Ben Varkey Benjamin Pottayil; Kevin Jose Chavez; Shitij Kumar - Web page: https://patentgrep.com/patent/US12214975B2 ## CPC - B65G 61/00: Use of pick-up or transfer devices or of manipulators for stacking or de-stacking articles not otherwise provided for - B25J 9/1661: {characterised by task planning, object-oriented languages} - B25J 9/1671: {characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems} - B25J 9/1687: {Assembly, peg and hole, palletising, straight line, weaving pattern movement} - B25J 9/1697: {Vision controlled systems} - B65G 57/03: from above - B65G 2203/0233: Position of the article - B65G 2203/0258: Weight of the article - B65G 2203/041: Camera - B65G 2203/042: Sensors - G05B 2219/40006: Placing, palletize, un palletize, paper roll placing, box stacking - G05B 2219/40607: Fixed camera to observe workspace, object, workpiece, global - G05B 2219/45063: Pick and place manipulator - B65G: TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS (packaging B65B; handling thin or filamentary materials, e.g. paper sheets or thread, B65H; cranes B66C; portable mobile or lifting or hauling appliances, e.g. hoists, B66D; devices for lifting or lowering goods for loading or unloading purposes, e.g. fork-lift trucks, B66F9/00; emptying bottles, jars, casks, barrels or similar containers, not otherwise provided for, B67C9/00; delivering or transferring liquids B67D; filling or discharging vessels for liquefied, solidified or compressed gases F17C; pipe-line systems for fluids F17D) - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract A robotic system is disclosed. The system includes a communication interface that receives, from a sensor(s) deployed in a workspace, sensor data indicative of a current state of the workspace, the workspace comprising a pallet or other receptacle and a plurality of items stacked on or in the receptacle. The system includes one or more processors that control a robotic arm to place a first set of items on or in, or remove the first set of items from, the pallet or other receptacle, update a geometric model based on the first set of items placed on or in a receptacle, use the geometric model in combination with the sensor data to estimate a stack of one or more items on or in the receptacle, and use the estimated state to generate or update a plan to control the robotic arm to place a second set of items. ## Claims ### Claim 1 A robotic system, comprising: a communication interface configured to receive, from one or more sensors deployed in a workspace, sensor data indicative of a current state of the workspace, the workspace comprising a pallet or other receptacle and a plurality of items stacked on the pallet or in the other receptacle; and one or more processors coupled to the communication interface and configured to: control a robotic arm to place a first set of items on or in, or remove the first set of items from, the pallet or the other receptacle; update a geometric model based on the first set of items placed on or in, or removed from, the pallet or ether the other receptacle; use the geometric model in combination with the sensor data to estimate a state of the pallet or the other receptacle and one or more items stacked on the pallet or in the other receptacle; and use the estimated state to generate or update a plan to control the robotic arm to place a second set of items on or in, or remove the second set of items from, the pallet or the other receptacle, wherein the estimated state of the pallet or other receptacle and the one or more items stacked on the pallet or in the other receptacle is determined based at least in part on an interpolation between the sensor data and the geometric model. ### Claim 2 (depends on 1) The robotic system of claim 1, wherein the geometric model reflects respective attributes of one or more of the first set of items and the second set of items. ### Claim 3 (depends on 2) The robotic system of claim 2, wherein the geometric model reflects respective weights of items comprised in the one or more of the first set of items and the second set of items. ### Claim 4 (depends on 1) The robotic system of claim 1, wherein the geometric model reflects a respective rigidity compressibility of items comprised in one or more of the first set of items and the second set of items. ### Claim 5 (depends on 1) The robotic system of claim 1, wherein the geometric model reflects a respective strength of packaging of items comprised in one or more of the first set of items and the second set of items. ### Claim 6 (depends on 1) The robotic system of claim 1, wherein the geometric model includes a distribution of weight across a plurality of items comprised in one or more of the first set of items and the second set of items. ### Claim 7 (depends on 1) The robotic system of claim 1, wherein the geometric model comprises an expected stability of the one or more items stacked on the pallet or in the other receptacle. ### Claim 8 (depends on 7) The robotic system of claim 7, wherein the expected stability is based at least in part on a set of locations at which the robotic system believes each item was placed, and one or more attributes of each item stacked on the pallet or in the other receptacle. ### Claim 9 (depends on 1) The robotic system of claim 1, wherein the geometric model comprises an expected stability of the one or more items stacked on the pallet or in the other receptacle and one or more simulated items for which simulation of a stacking of the one or more simulated items is performed. ### Claim 10 (depends on 1) The robotic system of claim 1, wherein the one or more sensors include image sensors. ### Claim 11 (depends on 10) The robotic system of claim 10, wherein the image sensors comprise a 3D camera. ### Claim 12 (depends on 1) The robotic system of claim 1, wherein a point cloud is determined based at least in part on the sensor data. ### Claim 13 (depends on 1) The robotic system of claim 1, wherein a location of each item is determined based on an interpolation between (i) a location of the respective item according to the sensor data, and (ii) a location of the respective item according to the geometric model. ### Claim 14 (depends on 1) The robotic system of claim 1, wherein the estimated state of the pallet or the other receptacle and the one or more items stacked on the pallet or in the other receptacle is determined based at least in part on a midpoint between an item location of a first item according to the sensor data and the item location of the first item according to the geometric model. ### Claim 15 (depends on 1) The robotic system of claim 1, wherein the one or more processors are further configured to: use the geometric model to update the current state of the workspace according to the sensor data in response to a determination that the current state of the workspace according to the sensor data comprises an anomaly. ### Claim 16 (depends on 15) The robotic system of claim 15, wherein the anomaly is caused by one or more of (i) a sensor of the one or more sensors being blocked or obscured from view with respect to a particular point in the workspace, (ii) a noise or gap in the sensor data that causes a gap in the current state of the workspace. ### Claim 17 (depends on 1) The robotic system of claim 1, wherein the one or more processors are further configured to: determine that a difference between a state expected based on the geometric model and a state observed via the one or more sensors exceed a threshold. ### Claim 18 (depends on 17) The robotic system of claim 17, wherein in response to a determination that the difference exceeds the threshold, communicate an alert to a user to confirm that a stack of the one or more items on the pallet or in the other receptacle is not problematic. ### Claim 19 (depends on 1) The robotic system of claim 1, wherein the one or more processors comprise: a first subset of the one or more processors that is configured to: generate or update the plan to control the robotic arm to place the second set of items on or in, or remove the second set of items from, the pallet or other receptacle, wherein: the plan to control the robotic arm is generated or updated based at least in part on the estimated state; and a second subset of the one or more processors that is configured to: update the geometric model and to determine the estimate of the state of the pallet or the other receptacle and the one or more items stacked on the pallet or in the other receptacle based at least in part on the geometric model in combination with the sensor data. ### Claim 20 (depends on 19) The robotic system of claim 19, wherein the second subset of the one or more processors is remote from the workspace. ### Claim 21 (depends on 1) The robotic system of claim 1, wherein: the first set of items comprises N items; and N is a positive integer that is dynamically determined. ### Claim 22 (depends on 21) The robotic system of claim 21, wherein the state of the one or more items stacked on or in the pallet or the other receptacle is estimated after the N items are stacked. ### Claim 23 (depends on 22) The robotic system of claim 22, wherein the state is estimated before the N items are stacked in response to a determination that an irregularly shaped item was placed. ### Claim 24 (depends on 1) The robotic system of claim 1, wherein: the second set of items corresponds to a next M items on a source to be placed on the pallet or receptacle; and M is a positive integer. ### Claim 25 (depends on 24) The robotic system of claim 24, wherein the source is a conveyor. ### Claim 26 A method to control a robot, comprising: receiving, from one or more sensors deployed in a workspace, sensor data indicative of a current state of the workspace, the workspace comprising a pallet or other receptacle and a plurality of items stacked on the pallet or in the other receptacle; controlling, by one or more processors, a robotic arm to place a first set of items on or in, or remove the first set of items from, the pallet or the other receptacle; updating a geometric model based on the first set of items placed on or in, or removed from, the pallet or the other receptacle; using the geometric model in combination with the sensor data to estimate a state of the pallet or the other receptacle and one or more items stacked on the pallet or in the other receptacle; and using the estimated state to generate or update a plan to control the robotic arm to place a second set of items on or in, or remove the second set of items from, the pallet or the other receptacle, wherein the estimated state of the pallet or other receptacle and the one or more items stacked on the pallet or in the other receptacle is determined based at least in part on an interpolation between the sensor data and the geometric model. ### Claim 27 A non-transitory computer readable mediums storing thereon a computer program product, comprising computer instructions executed by a processor for: receiving, from one or more sensors deployed in a workspace, sensor data indicative of a current state of the workspace, the workspace comprising a pallet or other receptacle and a plurality of items stacked on the pallet or in the other receptacle; controlling, by one or more processors, a robotic arm to place a first set of items on or in, or remove the first set of items from, the pallet or the other receptacle; updating a geometric model based on the first set of items placed on or in, or removed from, the pallet or the other receptacle; using the geometric model in combination with the sensor data to estimate a state of the pallet or the other receptacle and one or more items stacked on the pallet or in the other receptacle; and using the estimated state to generate or update a plan to control the robotic arm to place a second set of items on or in, or remove the second set of items from, the pallet or the other receptacle, wherein the estimated state of the pallet or other receptacle and the one or more items stacked on the pallet or in the other receptacle is determined based at least in part on an interpolation between the sensor data and the geometric model. ### Claim 28 A robotic system, comprising: a communication interface configured to receive, from one or more sensors deployed in a workspace, sensor data indicative of a current state of the workspace, the workspace comprising a pallet or other receptacle and a plurality of items stacked on the pallet or in the other receptacle; and one or more processors coupled to the communication interface and configured to: control a robotic arm to place a first set of items on or in, or remove the first set of items from, the pallet or the other receptacle; update a geometric model based on the first set of items placed on or in, or removed from, the pallet or the other receptacle; use the geometric model in combination with the sensor data to estimate a state of the pallet or the other receptacle and one or more items stacked on the pallet or in the other receptacle; and use the estimated state to generate or update a plan to control the robotic arm to place a second set of items on or in, or remove the second set of items from, the pallet or the other receptacle, wherein the estimated state of the pallet or the other receptacle and the one or more items stacked on the pallet or in the other receptacle is determined based at least in part on a midpoint between an item location of a first item according to the sensor data and the item location of the first item according to the geometric model. ### Claim 29 A method to control a robot, comprising: receiving, from one or more sensors deployed in a workspace, sensor data indicative of a current state of the workspace, the workspace comprising a pallet or other receptacle and a plurality of items stacked on the pallet or in the other receptacle; controlling, by one or more processors, a robotic arm to place a first set of items on or in, or remove the first set of items from, the pallet or the other receptacle; updating a geometric model based on the first set of items placed on or in, or removed from, the pallet or the other receptacle; using the geometric model in combination with the sensor data to estimate a state of the pallet or the other receptacle and one or more items stacked on the pallet or in the other receptacle; and using the estimated state to generate or update a plan to control the robotic arm to place a second set of items on or in, or remove the second set of items from, the pallet or the other receptacle, wherein the estimated state of the pallet or the other receptacle and the one or more items stacked on the pallet or in the other receptacle is determined based at least in part on a midpoint between an item location of a first item according to the sensor data and the item location of the first item according to the geometric model. ### Claim 30 A robotic system, comprising: a communication interface configured to receive, from one or more sensors deployed in a workspace, sensor data indicative of a current state of the workspace, the workspace comprising a pallet or other receptacle and a plurality of items stacked on the pallet or in the other receptacle; and one or more processors coupled to the communication interface and configured to: control a robotic arm to place a first set of items on or in, or remove the first set of items from, the pallet or the other receptacle; update a geometric model based on the first set of items placed on or in, or removed from, the pallet or the other receptacle; use the geometric model in combination with the sensor data to estimate a state of the pallet or the other receptacle and one or more items stacked on the pallet or in the other receptacle; use the estimated state to generate or update a plan to control the robotic arm to place a second set of items on or in, or remove the second set of items from, the pallet or the other receptacle; and determine that a difference between a state expected based on the geometric model and a state observed via the one or more sensors exceed a threshold, and performing an active measure in response to determining that the difference exceeds the threshold. ### Claim 31 A method to control a robot, comprising: receiving, from one or more sensors deployed in a workspace, sensor data indicative of a current state of the workspace, the workspace comprising a pallet or other receptacle and a plurality of items stacked on the pallet or in the other receptacle; controlling, by one or more processors, a robotic arm to place a first set of items on or in, or remove the first set of items from, the pallet or the other receptacle; updating a geometric model based on the first set of items placed on or in, or removed from, the pallet or the other receptacle; using the geometric model in combination with the sensor data to estimate a state of the pallet or the other receptacle and one or more items stacked on the pallet or in the other receptacle; using the estimated state to generate or update a plan to control the robotic arm to place a second set of items on or in, or remove the second set of items from, the pallet or the other receptacle; and determining that a difference between a state expected based on the geometric model and a state observed via the one or more sensors exceed a threshold, and performing an active measure in response to determining that the difference exceeds the threshold.