# US 10,313,639 B2 · Methods and systems for large-scale determination of RGBD camera poses > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,313,639 B2 - Document ID: US10313639B2 - Status: Granted - Kind: B2 - Application number: 15/356,372 - Filed: 2016-11-18 - Published: 2017-05-25 - Granted: 2019-06-04 - Earliest claimed: 2015-11-20 - Assignees: Magic Leap, Inc. - Applicants: Magic Leap, Inc. - Inventors: Xiaolin Wei - Web page: https://patentgrep.com/patent/US10313639B2 ## CPC - G06T 7/73: using feature-based methods - G06T 7/33: using feature-based methods - H04N 23/10: for generating image signals from different wavelengths - G06T 2200/04: involving 3D image data - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20224: Image subtraction - G06T 2207/30244: Camera pose - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method of determining camera poses includes capturing a plurality of image frames using a camera, computing relative poses between each set of image frame pairs to provide a relative pose set and an uncategorized relative pose set, and detecting and removing miscategorized relative poses to provide a remaining relative pose set. The method also includes determining global poses using the remaining relative pose set and computing extended relative poses for at least a portion of the miscategorized relative poses and at least a portion of the uncategorized relative pose set to provide an extended relative pose set and an extended uncategorized relative pose set. The method further includes detecting and removing extended miscategorized relative poses to provide a remaining extended relative pose set and determining updated global poses for the plurality of image frames using the remaining relative pose set and the remaining extended relative pose set. ## Claims ### Claim 1 A method of determining camera poses for a plurality of image frames, the method comprising: capturing the plurality of image frames using a camera; computing relative poses between each set of image frame pairs to provide a relative pose set, wherein computing the relative poses comprises performing: a first process for a first subset of the image frame pairs having a temporal separation between image frames of the image frame pairs less than a threshold; and a second process for a second subset of the image frame pairs having a temporal separation between image frames of the image frame pairs greater than the threshold; detecting and removing miscategorized relative poses from the relative pose set to provide a remaining relative pose set; determining global poses for the plurality of image frames using the remaining relative pose set; computing extended relative poses for spatially close image frame pairs to provide an extended relative pose set; detecting and removing extended miscategorized relative poses from the extended relative pose set to provide a remaining extended relative pose set; and determining updated global poses for the plurality of image frames using the remaining relative pose set and the remaining extended relative pose set. ### Claim 2 (depends on 1) The method of claim 1 further comprising refining the updated global poses through N iterations. ### Claim 3 (depends on 2) The method of claim 2 wherein refining the updated global poses comprises aligning scene geometry derived from depth maps associated with different image frames of the plurality of image frames. ### Claim 4 (depends on 2) The method of claim 2 wherein refining the updated global poses comprises adjusting the updated global poses toward a convergence for each of the updated global poses. ### Claim 5 (depends on 2) The method of claim 2 wherein refining the updated global poses through N iterations comprises performing a predetermined number of iterations. ### Claim 6 (depends on 2) The method of claim 2 wherein refining the updated global poses through N iterations comprises performing a predetermined number of iterations based on a threshold for differences in camera poses between iterations. ### Claim 7 (depends on 1) The method of claim 1 wherein the relative pose set includes valid relative poses and miscategorized relative poses. ### Claim 8 (depends on 1) The method of claim 1 wherein the extended relative pose set includes valid extended relative poses and extended miscategorized relative poses. ### Claim 9 (depends on 1) The method of claim 1 wherein the global poses for the plurality of image frames are referenced to a global reference frame. ### Claim 10 (depends on 1) The method of claim 1 wherein the camera comprises an RGBD camera. ### Claim 11 (depends on 1) The method of claim 1 wherein the plurality of image frames are captured at a predetermined frame rate. ### Claim 12 (depends on 1) The method of claim 1 wherein the plurality of image frames comprise color data and depth data for each pixel in each of the plurality of image frames. ### Claim 13 (depends on 1) The method of claim 1 wherein the second process comprises detecting and matching features between the image frame pairs in the second subset. ### Claim 14 (depends on 13) The method of claim 13 wherein the second process further comprises performing depth matching between the image frame pairs in the second subset. ### Claim 15 (depends on 1) The method of claim 1, wherein the first process comprises aligning depth data of the image frame pairs in the first subset. ### Claim 16 (depends on 1) The method of claim 1 further comprising determining that an image frame pair comprises a spatially close image frame pair. ### Claim 17 (depends on 16) The method of claim 16 wherein the spatially close image frame pair is determined by: performing frustum intersection analysis; and performing pointwise overlap analysis. ### Claim 18 (depends on 17) The method of claim 17, wherein performing frustum intersection analysis comprises comparing a frustum of a first image frame and a frustum of a second image frame to determine that the frustum of the first image frame and the frustum of the second image frame intersect. ### Claim 19 (depends on 17) The method of claim 17, wherein performing pointwise overlap analysis comprises: (a) for a point in a first image frame, determining a closest point in a second image frame; (b) for the closest point in the second image frame, determining a closest point in the first image frame; (c) determining that the closest point in the first image frame is the same as the point in the first image frame; and repeating (a)-(c) for each point in the first image frame. ### Claim 20 (depends on 19) The method of claim 19 wherein the intersecting frustums are characterized by a viewing range, a viewing direction, and an overlap between point pairs. ### Claim 21 A non-transitory computer-readable storage medium comprising a plurality of computer-readable instructions tangibly embodied on the computer-readable storage medium, which, when executed by a data processor, determining camera poses for a plurality of image frames, the plurality of instructions comprising: instructions that cause the data processor to capture the plurality of image frames using a camera; instructions that cause the data processor to compute relative poses between each set of image frame pairs to provide a relative pose set, wherein computing the relative poses comprises performing: a first process for a first subset of the image frame pairs having a temporal separation between image frames of the image frame pairs less than a threshold; and a second process for a second subset of the image frame pairs having a temporal separation between image frames of the image frame pairs greater than the threshold; instructions that cause the data processor to detect and remove miscategorized relative poses from the relative pose set to provide a remaining relative pose set; instructions that cause the data processor to determine global poses for the plurality of image frames using the remaining relative pose set; instructions that cause the data processor to compute extended relative poses for spatially close image frame pairs to provide an extended relative pose set; instructions that cause the data processor to detect and remove extended miscategorized relative poses from the extended relative pose set to provide a remaining extended relative pose set; and instructions that cause the data processor to determine updated global poses for the plurality of image frames using the remaining relative pose set and the remaining extended relative pose set. ### Claim 22 (depends on 1) The method of claim 1 wherein the spatially close image frame pairs are characterized by intersecting frustums. ### Claim 23 (depends on 21) The computer-readable storage medium of claim 21 wherein the plurality of instructions further comprise instructions that cause the data processor to refine the updated global poses through N iterations. ### Claim 24 (depends on 21) The computer-readable storage medium of claim 21 wherein the relative pose set includes valid relative poses and miscategorized relative poses. ### Claim 25 (depends on 21) The computer-readable storage medium of claim 21 wherein the global poses for the plurality of image frames are referenced to a global reference frame. ### Claim 26 (depends on 21) The computer-readable storage medium of claim 21 wherein the camera comprises an RGBD camera and wherein the plurality of image frames are captured at a predetermined frame rate.