# US 12,002,227 B1 · Deep partial point cloud registration of objects > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,227 B1 - Document ID: US12002227B1 - Status: Granted - Kind: B1 - Application number: 17/376,313 - Filed: 2021-07-15 - Granted: 2024-06-04 - Earliest claimed: 2020-08-26 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Donghoon Lee; Thorsten Gernoth; Onur C. Hamsici; Shuo Feng - Web page: https://patentgrep.com/patent/US12002227B1 ## CPC - G06T 7/344: {involving models} - G06N 3/045: Combinations of networks - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06T 7/33: using feature-based methods - G06T 15/205: {Image-based rendering} - G06T 2207/10016: Video; Image sequence - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Devices, systems, and methods are disclosed for partial point cloud registration. In some implementations, a method includes obtaining a first set of three-dimensional (3D) points corresponding to an object in a physical environment, the first set of 3D points having locations in a first coordinate system, obtaining a second set of 3D points corresponding to the object in the physical environment, the second set of 3D points having locations in a second coordinate system, predicting, via a machine learning model, locations of the first set of 3D points in the second coordinate system, and determining transform parameters relating the first set of 3D points and the second set of 3D points based on the predicted location of the first set of 3D points in the second coordinate system. ## Claims ### Claim 1 A method comprising: at an electronic device having a processor: obtaining a first set of three-dimensional (3D) points corresponding to an object in a physical environment, the first set of 3D points having locations in a first coordinate system; obtaining a second set of 3D points corresponding to the object in the physical environment, the second set of 3D points having locations in a second coordinate system; predicting, via a machine learning model, locations of the first set of 3D points in the second coordinate system; and determining transform parameters relating the first set of 3D points and the second set of 3D points based on the predicted locations of the first set of 3D points in the second coordinate system. ### Claim 2 (depends on 1) The method of claim 1, wherein the first set of 3D points and the second set of 3D points are obtained via a depth sensor. ### Claim 3 (depends on 1) The method of claim 1, wherein the second coordinate system is different than the first coordinate system. ### Claim 4 (depends on 1) The method of claim 1, wherein the first set of 3D points are obtained at a first viewpoint and the second set of 3D points are obtained at a second viewpoint different than the first viewpoint. ### Claim 5 (depends on 1) The method of claim 1, wherein the first set of 3D points are obtained at a first viewpoint and the second set of 3D points are obtained at a second viewpoint that is the same as the first viewpoint. ### Claim 6 (depends on 1) The method of claim 1, wherein the machine learning model is a neural network. ### Claim 7 (depends on 6) The method of claim 6, wherein the first set of 3D points comprises a particular order and the neural network outputs the locations of the first set of 3D points in the second coordinate system such that the order of the first set of 3D points is preserved. ### Claim 8 (depends on 6) The method of claim 6, wherein the neural network is trained to encode features corresponding to the object and generate the predicted locations of the first set of 3D points based on the encoded features. ### Claim 9 (depends on 1) The method of claim 1, further comprising and preceding predicting the locations of the first set of 3D points in the second coordinate system via the machine learning model: determining local and global features of the first and second set of 3D points. ### Claim 10 (depends on 9) The method of claim 9, wherein the predicted locations of the first set of 3D points are based on the local and the global features of the first set of 3D points and the global features of the second set of 3D points. ### Claim 11 (depends on 1) The method of claim 1, further comprising: predicting, via the machine learning model, locations of the second set of 3D points in the first coordinate system. ### Claim 12 (depends on 1) The method of claim 1, further comprising: obtaining two or more sets of 3D points corresponding to the object in the physical environment, where each of the two or more sets of 3D points have locations in a respective coordinate system; predicting, via the machine learning model, locations of the first set of 3D points in each of the two or more coordinate systems associated with the two or more sets of 3D points; and determining transform parameters relating the first set of 3D points and each of the two or more sets of 3D points based on the predicted location of the first set of 3D points in each of the two or more coordinate systems associated with the two or more sets of 3D points. ### Claim 13 (depends on 1) The method of claim 1, further comprising and preceding predicting the locations of the first set of 3D points in the second coordinate system via the machine learning model: randomly rotating a viewpoint of at least one of the first set of 3D points and the second set of 3D points; and predicting the locations of the first set of 3D points in the second coordinate system based on the random rotation of the viewpoint of the at least one of the first set of 3D points and the second set of 3D points. ### Claim 14 (depends on 13) The method of claim 13, further comprising and preceding determining the transform parameters relating the first set of 3D points and the second set of 3D points: reversing the random rotation of the viewpoint of the at least one of the first set of 3D points and the second set of 3D points. ### Claim 15 (depends on 1) The method of claim 1, wherein the transform parameters comprise at least one of rotation parameters and translation parameters. ### Claim 16 (depends on 1) The method of claim 1, wherein predicting the locations of the first set of 3D points and/or determining the transform parameters are performed via one or more neural networks trained using ground truth data that includes: (i) known transform parameters, and (ii) one or more distance losses based on a measured distance between a ground set of 3D points transformed based on the determined transform parameters and the known transform parameters. ### Claim 17 A system comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising: obtaining a first set of three-dimensional (3D) points corresponding to an object in a physical environment, the first set of 3D points having locations in a first coordinate system; obtaining a second set of 3D points corresponding to the object in the physical environment, the second set of 3D points having locations in a second coordinate system; predicting, via a machine learning model, locations of the first set of 3D points in the second coordinate system; and determining transform parameters relating the first set of 3D points and the second set of 3D points based on the predicted locations of the first set of 3D points in the second coordinate system. ### Claim 18 (depends on 17) The system of claim 17, wherein the first set of 3D points and the second set of 3D points are obtained via a depth sensor. ### Claim 19 (depends on 17) The system of claim 17, wherein the second coordinate system is different than the first coordinate system. ### Claim 20 A non-transitory computer-readable storage medium, storing program instructions executable on a device to perform operations comprising: obtaining a first set of three-dimensional (3D) points corresponding to an object in a physical environment, the first set of 3D points having locations in a first coordinate system; obtaining a second set of 3D points corresponding to the object in the physical environment, the second set of 3D points having locations in a second coordinate system; predicting, via a machine learning model, locations of the first set of 3D points in the second coordinate system; and determining transform parameters relating the first set of 3D points and the second set of 3D points based on the predicted locations of the first set of 3D points in the second coordinate system.