# US 11,999,064 B2 · Excavation learning for rigid objects in clutter > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,999,064 B2 - Document ID: US11999064B2 - Status: Granted - Kind: B2 - Application number: 17/380,589 - Filed: 2021-07-20 - Published: 2023-02-02 - Granted: 2024-06-04 - Earliest claimed: 2021-07-20 - Assignees: Baidu USA LLC - Applicants: Baidu USA, LLC - Inventors: Qingkai Lu; Liangjun Zhang - Web page: https://patentgrep.com/patent/US11999064B2 ## CPC - B25J 9/1664: {characterised by motion, path, trajectory planning} - E02F 9/265: {with follow-up actions (e.g. control signals sent to actuate the work tool)} - B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor} - B25J 9/1697: {Vision controlled systems} - B25J 13/08: by means of sensing devices, e.g. viewing or touching devices - E02F 3/437: {providing automatic sequences of movements, e.g. linear excavation, keeping dipper angle constant} - E02F 9/262: {with follow-up actions to control the work tool, e.g. controller} - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06T 17/05: Geographic models - G06T 17/10: Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes - G05B 2219/40442: Voxel map, 3-D grid map - G05B 2219/45012: Excavator - G06N 3/048: Activation functions - G06T 2210/21: Collision detection, intersection - G06T 2210/56: Particle system, point based geometry or rendering - G06T 2210/61: Scene description - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - E02F: DREDGING; SOIL-SHIFTING (winning peat E21C49/00) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract Embodiments of a learning-based excavation planning method are disclosed for excavating rigid objects in clutter, which is challenging due to high variance of geometric and physical properties of objects, and large resistive force during the excavation. A convolutional neural network is utilized to predict a probability of excavation success. Embodiments of a sampling-based optimization method are disclosed for planning high-quality excavation trajectories by leveraging the learned prediction model. To reduce simulation-to-real gap for excavation learning, voxel-based representations of an excavation scene are used. Excavation experiments were performed in both simulation and real world to evaluate the learning-based excavation planners. Experimental results show that embodiments of the disclosed method may plan high-quality excavations for rigid objects in clutter and outperform baseline methods by large margins. ## Claims ### Claim 1 A computer-implemented method for autonomous excavation learning comprising: tiling each parameter of an excavation trajectory, which is represented by multiple trajectory parameters, across a voxel-grid to obtain a tiled trajectory parameter voxel-grid for each parameter of the excavation trajectory; concatenating the tiled trajectory parameter voxel-grids with a voxel-grid representation of an excavation scene to generate an input voxel-grid of an excavation instance that defines a pair of the excavation trajectory and the voxel-grid representation of the excavation scene; feeding the input voxel-grid into a neural network to generate a multi-dimensional feature vector; and processing, using at least one fully-connected layer, the multi-dimensional feature vector to predict an excavation success probability. ### Claim 2 (depends on 1) The computer-implemented method of claim 1 wherein the multiple trajectory parameters comprise coordinates of a point of attack (PoA), an attacking excavation angle, a penetration depth, a closing angle, and a dragging length. ### Claim 3 (depends on 1) The computer-implemented method of claim 1 wherein the voxel-grid representation of the excavation scene is generated by steps comprising: transforming a pointcloud of an excavation scene, obtained from an RGB-Depth camera, into an excavation region frame; filtering the transformed pointcloud into a filtered pointcloud to correspond to a size of a specific excavation cuboid space; and voxelizing the filtered pointcloud to generate the voxel-grid representation of the excavation scene. ### Claim 4 (depends on 1) The computer-implemented method of claim 1 wherein the neural network is a three dimensional (3D) convolutional neural network (CNN) comprising multiple 3D convolution filters. ### Claim 5 (depends on 1) The computer-implemented method of claim 1 wherein the excavation scene comprises one or more rigid objects that are not deformable during excavation. ### Claim 6 (depends on 1) The computer-implemented method of claim 1 wherein the at least one fully-connected layer is followed by a sigmoid output layer. ### Claim 7 (depends on 1) The computer-implemented method of claim 1 wherein the at least one fully-connected layer is followed by an additional fully-connected layer for a regression model for excavation prediction. ### Claim 8 (depends on 1) The computer-implemented method of claim 1 wherein the excavation trajectory is one of multiple heuristic excavation trajectories that are generated using an excavation planner to initialize a distribution of trajectory parameters. ### Claim 9 (depends on 8) The computer-implemented method of claim 8 further comprising: updating the distribution through multiple iterations of excavation trajectory sampling and excavation trajectory selecting in terms of a predicted excavation success probability for each sampled excavation trajectory; sampling, from the distribution at the last iteration, a subset of excavation trajectories for evaluation; and choosing, from the subset of excavation trajectories, one excavation trajectory with the highest predicted excavation success probability as a planned excavation trajectory. ### Claim 10 A system for autonomous excavation learning and planning comprising: one or more processors; and a non-transitory computer-readable medium or media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising: tiling each parameter of an excavation trajectory, which is represented by multiple trajectory parameters, across a voxel-grid to obtain a tiled trajectory parameter voxel-grids for each parameter of the excavation trajectory; concatenating the tiled trajectory parameter voxel-grids with a voxel-grid representation of an excavation scene to generate an input voxel-grid of an excavation instance that comprises the excavation trajectory and the voxel-grid representation of the excavation scene; feeding the input voxel-grid into a neural network to generate a multi-dimensional feature vector; and processing, using one or more fully-connected layers, the multi-dimensional feature vector to predict an excavation success probability. ### Claim 11 (depends on 10) The system of claim 10 wherein the voxel-grid representation of the excavation scene is generated by steps comprising: transforming a pointcloud of an excavation scene, obtained from an RGB-Depth camera, into an excavation region frame; filtering the transformed pointcloud into a filtered pointcloud to correspond to a size of a specific excavation cuboid space; and voxelizing the filtered pointcloud to generate the voxel-grid representation of the excavation scene. ### Claim 12 (depends on 10) The system of claim 10 wherein the multiple trajectory parameters comprise coordinates of a point of attack (PoA), an attacking excavation angle, a penetration depth, a closing angle, and a dragging length. ### Claim 13 (depends on 10) The system of claim 10 wherein the neural network is a three dimensional (3D) convolutional neural network (CNN) comprising multiple 3D convolution filters. ### Claim 14 A computer-implemented method for excavation planning comprising: generating, using an excavation planner, multiple excavation trajectories given a voxel-grid representation of an excavation scene to initialize a distribution of trajectory parameters, each excavation trajectory is presented by multiple trajectory parameters; updating the distribution through multiple iterations of sampling excavation trajectories and selecting a set of excavation trajectories from the sampled excavation trajectory using predicted excavation success probabilities for the sampled excavation trajectories that were predicted by an excavation prediction model by performing, for each sampled excavation trajectory, steps comprising: tiling each parameter of the sampled excavation trajectory across a voxel-grid to obtain a tiled trajectory parameter voxel-grid corresponding to the parameter; concatenating the tiled trajectory parameter voxel-grids with the voxel-grid representation of the excavation scene to generate an input voxel-grid of an excavation instance that comprises the sampled excavation trajectory and the voxel-grid representation of the excavation scene; feeding the input voxel-grid into a neural network to generate a multi-dimensional feature vector; and processing, using one or more fully-connected layers, the multi-dimensional feature vector to output the predicted excavation success probability; sampling, from the distribution at the last iteration, a subset of excavation trajectories; and choosing, from a subset of excavation trajectories, one excavation trajectory with the highest predicted success probability as a planned excavation trajectory. ### Claim 15 (depends on 14) The computer-implemented method of claim 14 wherein the multiple trajectory parameters comprise coordinates of a point of attack (PoA), an attacking excavation angle, a penetration depth, a closing angle, and a dragging length. ### Claim 16 (depends on 14) The computer-implemented method of claim 14 wherein the neural network is a three dimensional (3D) convolutional neural network (CNN) comprising multiple 3D convolution filters. ### Claim 17 (depends on 14) The computer-implemented method of claim 14 wherein the one or more fully-connected layers is followed by a sigmoid output layer for processing the multi-dimensional feature vector. ### Claim 18 (depends on 15) The computer-implemented method of claim 15 wherein the multiple excavation trajectories are heuristic excavation trajectories. ### Claim 19 (depends on 14) The computer-implemented method of claim 14 wherein the excavation trajectory represents a three-dimensional trajectory.