# US 12,112,494 B2 · Robotic manipulation using domain-invariant 3D representations predicted from 2.5D vision data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,494 B2 - Document ID: US12112494B2 - Status: Granted - Kind: B2 - Application number: 17/053,335 - Filed: 2020-02-28 - Published: 2021-04-08 - Granted: 2024-10-08 - Earliest claimed: 2019-03-01 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Honglak Lee; Xinchen Yan; Soeren Pirk; Yunfei Bai; Seyed Mohammad Khansari Zadeh; Yuanzheng Gong; Jasmine Hsu - Web page: https://patentgrep.com/patent/US12112494B2 ## CPC - G06T 7/55: from multiple images - B25J 9/1605: {Simulation of manipulator lay-out, design, modelling of manipulator} - B25J 9/163: {learning, adaptive, model based, rule based expert control} - B25J 9/1669: {characterised by special application, e.g. multi-arm co-operation, assembly, grasping} - B25J 9/1671: {characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems} - B25J 9/1697: {Vision controlled systems} - B25J 13/08: by means of sensing devices, e.g. viewing or touching devices - G06F 18/2163: {Partitioning the feature space} - G06T 7/50: Depth or shape recovery - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 20/64: Three-dimensional [3D] objects - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20132: Image cropping - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Implementations relate to training a point cloud prediction model that can be utilized to process a single-view two-and-a-half-dimensional (2.5D) observation of an object, to generate a domain-invariant three-dimensional (3D) representation of the object. Implementations additionally or alternatively relate to utilizing the domain-invariant 3D representation to train a robotic manipulation policy model using, as at least part of the input to the robotic manipulation policy model during training, the domain-invariant 3D representations of simulated objects to be manipulated. Implementations additionally or alternatively relate to utilizing the trained robotic manipulation policy model in control of a robot based on output generated by processing generated domain-invariant 3D representations utilizing the robotic manipulation policy model. ## Claims ### Claim 1 A method implemented by one or more processors of a robot, the method comprising: identifying an image captured by a camera of the robot, the image capturing an object to be manipulated by the robot, and the image comprising multiple channels, including one or more color channels and a depth channel; generating an object mask of the object to be manipulated by the robot, wherein generating the object mask comprises: processing one or more of the channels of the image using an object detection network; generating a three-dimensional (3D) point cloud of the object, wherein generating the 3D point cloud of the object comprises: processing, using a point cloud prediction network: all of the channels of at least a portion of the image, and the generated object mask of the object; generating a prediction of successful manipulation of the object, wherein generating the prediction of successful manipulation of the object comprises: generating the prediction of successful manipulation by processing a transformation of the 3D point cloud, using a robotic manipulation policy model, wherein generating the prediction of successful manipulation comprises: sampling, from multiple candidate end effector poses, a candidate end effector pose, of an end effector of the robot, generating the transformation of the 3D point cloud by transforming the 3D point cloud to an end effector frame that is relative to the candidate end effector pose, and generating the prediction of successful manipulation by processing the transformation of the 3D point cloud using the robotic manipulation policy model; and controlling one or more actuators of the robot based on the prediction of successful manipulation. ### Claim 2 (depends on 1) The method of claim 1, wherein the object detection network is trained for processing images to generate bounding boxes and masks of objects in the images, and further comprising: generating a bounding box for the object to be manipulated by the robot, wherein generating the bounding box comprises: processing one or more of the channels of the image using the object detection network. ### Claim 3 (depends on 2) The method of claim 2, wherein the at least the portion of the image that is processed using the point cloud prediction network is a crop of the image selected based on the bounding box. ### Claim 4 (depends on 3) The method of claim 3, further comprising selecting the crop of the image based on pixels of the image, included in the crop, being encompassed by the bounding box. ### Claim 5 (depends on 3) The method of claim 3, wherein generating the 3D point cloud of the object further comprises: processing, using the point cloud prediction network: one or more camera intrinsics, the one or more camera intrinsics defining one or more intrinsic parameters of the camera that take into account the crop of the image. ### Claim 6 (depends on 5) The method of claim 5, wherein in processing the one or more camera intrinsics using the point cloud prediction network, the one or more camera intrinsics are applied, as side input to the point cloud prediction network, downstream from an initial input at which all of the channels of at least the portion of the image, and the generated object mask of the object are applied as initial input. ### Claim 7 (depends on 6) The method of claim 6, wherein all of the channels of at least the portion of the image and the generated object mask of the object are initially processed using an initial encoder of the point cloud prediction network, and wherein the camera intrinsics are applied as side input after the initial encoder and prior to processing using an initial decoder of the point cloud prediction network. ### Claim 8 (depends on 1) The method of claim 1, wherein controlling the one or more actuators of the robot based on the prediction of successful manipulation comprises: selecting the candidate end effector pose based on the prediction of successful manipulation satisfying at least one criterion; and in response to selecting the candidate end effector pose: controlling the one or more actuators of the robot to cause the end effector to traverse to the candidate end effector pose. ### Claim 9 (depends on 8) The method of claim 8, wherein the manipulation is grasping, and wherein controlling the one or more actuators of the robot based on the prediction of successful manipulation further comprises: causing the end effector to attempt a grasp of the object after the end effector is in the candidate end effector pose. ### Claim 10 (depends on 8) The method of claim 8, further comprising: sampling, from the multiple candidate end effector poses, an additional candidate end effector pose of the end effector; generating an additional transformation of the 3D point cloud by transforming the 3D point cloud to an additional end effector frame that is relative to the additional candidate end effector pose; and generating an additional prediction of successful manipulation by processing the additional transformation of the 3D point cloud using the critic network; wherein the at least one criterion utilized in selecting the candidate end effector pose based on the prediction of successful manipulation comprises the prediction of successful manipulation being more indicative of success than the additional prediction of successful manipulation. ### Claim 11 (depends on 1) The method of claim 1, wherein the point cloud prediction network comprises a plurality of encoder-decoder modules, and at least one fully-connected layer. ### Claim 12 A system comprising: one or more actuators operably coupled to a robot; one or more processors; and a memory, the memory comprising computer readable instructions that, when executed by the one or more processor, cause the system to perform a method comprising: identifying an image captured by a camera of the robot, the image capturing an object to be manipulated by the robot, and the image comprising multiple channels, including one or more color channels and a depth channel; generating an object mask of the object to be manipulated by the robot, wherein generating the object mask comprises: processing one or more of the channels of the image using an object detection network; generating a three-dimensional (3D) point cloud of the object, wherein generating the 3D point cloud of the object comprises: processing, using a point cloud prediction network: all of the channels of at least a portion of the image, and the generated object mask of the object; generating a prediction of successful manipulation of the object, wherein generating the prediction of successful manipulation of the object comprises: generating the prediction of successful manipulation by processing a transformation of the 3D point cloud, using a robotic manipulation policy model, wherein generating the prediction of successful manipulation comprises: sampling, from multiple candidate end effector poses, a candidate end effector pose, of an end effector of the robot, generating the transformation of the 3D point cloud by transforming the 3D point cloud to an end effector frame that is relative to the candidate end effector pose, and generating the prediction of successful manipulation by processing the transformation of the 3D point cloud using the robotic manipulation policy model; and controlling the one or more actuators of the robot based on the prediction of successful manipulation. ### Claim 13 (depends on 12) The system of claim 12, wherein the object detection network is trained for processing images to generate bounding boxes and masks of objects in the images, and the method further comprises: generating a bounding box for the object to be manipulated by the robot, wherein generating the bounding box comprises: processing one or more of the channels of the image using the object detection network. ### Claim 14 (depends on 13) The system of claim 13, wherein the at least the portion of the image that is processed using the point cloud prediction network is a crop of the image selected based on the bounding box; and wherein generating the 3D point cloud of the object further comprises: processing, using the point cloud prediction network: one or more camera intrinsics, the one or more camera intrinsics defining one or more intrinsic parameters of the camera that take into account the crop of the image. ### Claim 15 (depends on 14) The system of claim 14, wherein in processing the one or more camera intrinsics using the point cloud prediction network, the one or more camera intrinsics are applied, as side input to the point cloud prediction network, downstream from an initial input at which all of the channels of at least the portion of the image, and the generated object mask of the object are applied as initial input. ### Claim 16 (depends on 15) The system of claim 15, wherein all of the channels of at least the portion of the image and the generated object mask of the object are initially processed using an initial encoder of the point cloud prediction network, and wherein the camera intrinsics are applied as side input after the initial encoder and prior to processing using an initial decoder of the point cloud prediction network. ### Claim 17 (depends on 1) The method of claim 1, wherein controlling the one or more actuators of the robot is based on the prediction of successful manipulation and is based on an additional prediction of successful manipulation of the object, and further comprising: generating the additional prediction of successful manipulation of the object, wherein generating the additional prediction of successful manipulation of the object comprises: generating the additional prediction of successful manipulation by processing an additional transformation of the 3D point cloud, using the robotic manipulation policy model, wherein generating the additional prediction of successful manipulation comprises: sampling an additional candidate end effector pose, of the multiple candidate end effector poses of the end effector of the robot, generating the additional transformation of the 3D point cloud by transforming the 3D point cloud to an additional end effector frame that is relative to the additional candidate end effector pose, and generating the additional prediction of successful manipulation by processing the additional transformation of the 3D point cloud using the robotic manipulation policy model. ### Claim 18 (depends on 12) The system of claim 12, wherein controlling the one or more actuators of the robot is based on the prediction of successful manipulation and is based on an additional prediction of successful manipulation of the object, and wherein the method, performed by the system when the computer readable instructions are executed by the one or more processors, further comprises: generating the additional prediction of successful manipulation of the object, wherein generating the additional prediction of successful manipulation of the object comprises: generating the additional prediction of successful manipulation by processing an additional transformation of the 3D point cloud, using the robotic manipulation policy model, wherein generating the additional prediction of successful manipulation comprises: sampling an additional candidate end effector pose, of the multiple candidate end effector poses of the end effector of the robot, generating the additional transformation of the 3D point cloud by transforming the 3D point cloud to an additional end effector frame that is relative to the additional candidate end effector pose, and generating the additional prediction of successful manipulation by processing the additional transformation of the 3D point cloud using the robotic manipulation policy model.