# US 2021/0279953 A1 · Cross reality system with wifi/gps based map merge > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0279953 A1 - Document ID: US20210279953A1 - Status: Published application - Kind: A1 - Application number: 17/180,453 - Filed: 2021-02-19 - Published: 2021-09-09 - Earliest claimed: 2020-02-20 - Assignees: Magic Leap, Inc. - Applicants: Magic Leap, Inc. - Inventors: Moshe Bouhnik; Ben Weisbih; Miguel Andres Granados Velasquez; Ali Shahrokni; Ashwin Swaminathan - Web page: https://patentgrep.com/patent/US20210279953A1 ## CPC - G06T 17/05: Geographic models - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/337: {involving reference images or patches} - G06T 7/55: from multiple images - G06T 15/005: {General purpose rendering architectures} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A cross reality system receives tracking information in a tracking map and first location metadata associated with at least a portion of the tracking map. A sub-portion of a canonical map is determined based at least in part on a correspondence between the first location metadata associated with the at least the portion of the tracking map and second location metadata associated with the sub-portion of the canonical map. The sub-portion of the canonical map may be merged with the at least the portion of the tracking map into a merged map. The cross reality system may further generate the tracking map by using at least the pose information from one or more images and localize the tracking map to the canonical may at least by using a persistent coordinate frame in the canonical map and the location metadata associated with the location represented in the tracking map. ## Claims ### Claim 1 A portable computing device configured for rendering virtual content for extended-reality, comprising: at least one processor; a non-transitory computer-readable storage medium operatively coupled to the at least one processor and having stored therein a canonical map and a sequence of instructions which, when executed by the at least one processor, causes the at least one processor to perform a set of acts, the set of acts comprising: receiving, from the portable computing device, tracking information in a tracking map and first location metadata associated with at least a portion of the tracking map; determining a sub-portion of the canonical map based at least in part on a correspondence between the first location metadata associated with the at least the portion of the tracking map and second location metadata associated with the sub-portion of the canonical map; and merging the sub-portion of the canonical map with the at least the portion of the tracking map into a merged map. ### Claim 2 (depends on 1) The portable computer device of claim 1, wherein merging the sub-portion of the canonical map with the tracking map comprises: identifying a first plurality of features in the at least the portion of the tracking map; identifying a second plurality of features in the selected sub-portion of the canonical map; and identifying a transformation that respectively aligns the first plurality of features with the second plurality of features. ### Claim 3 (depends on 2) The portable computer device of claim 2, wherein merging the sub-portion of the canonical map with the at least the portion of the tracking map further comprises identifying an image frame pertaining to the tacking map and a persistent coordinate frame in the sub-portion of the canonical map. ### Claim 4 (depends on 1) The computing device of claim 1, further comprising integrating the merged map with the canonical map, wherein integrating the merged map with the canonical map comprises replacing the selected sub-portion of the canonical map with the merged map or extending the selected sub-portion of the canonical map based at least in part upon the merged map. ### Claim 5 (depends on 1) The computer device of claim 1, determining the sub-portion of the canonical map comprising: determining first pose information from an image frame that is associated with the tracking map; identifying a persistent coordinate frame from the canonical map based at least in part upon the first pose information; and determining the sub-portion of the canonical map based at least in part upon the persistent coordinate frame. ### Claim 6 (depends on 1) The computer device of claim 1, wherein receiving, from the portable device, the location metadata associated with at least a portion of the tracking map comprises: receiving, from the portable device, global positioning system (GPS) location information indicative of a prior location of the portable device within the 3D environment. ### Claim 7 (depends on 1) The computer device of claim 1, further comprising localizing the at least the portion of the tracking map to the portable computing device, localizing the at least the portion of the tracking map comprising: determining first pose information from an image frame that is associated with the tracking map; identifying a persistent coordinate frame from a plurality of persistent coordinate frames in the canonical map based at least in part upon the first pose information; and localizing the at least the portion of the tracking map to the portable computing device based at least in part upon the persistent coordinate frame in the canonical map and the first pose information in the tracking map. ### Claim 8 (depends on 1) The computer device of claim 1, further comprising receiving location metadata associated with a tile of multiple tiles in the tracking map, wherein the location metadata associated with the at least the portion of the tracking map is further associated with a persistent pose in the tracking map. ### Claim 9 (depends on 1) The computer device of claim 1, wherein the tracking information or the first location metadata associated with the at least the portion of the tracking map comprises wireless network information indicative of a prior location of the portable computing device or a different portable computing device with respect to one or more wireless network access points. ### Claim 10 (depends on 1) The computer device of claim 1, further comprising ranking, by the portable computing device or a remote server computer, a plurality of canonical maps with respective rankings based at least in part upon the tracking map, where a respective ranking of a corresponding canonical map indicates that a correspondence between the corresponding canonical map and the tracking map. ### Claim 11 (depends on 1) The computer device of claim 1, further comprising generating, at the portable computing device, the canonical map, generating the canonical map comprising: identifying an object or a feature thereof from an image frame as a persistent pose; transforming the pose into a persistent coordinate frame; and storing the persistent coordinate frame in the canonical map, wherein the persistent coordinate frame includes or is associated with the transformation. ### Claim 12 (depends on 11) The computer device of claim 11, the set of acts further comprising promoting the canonical map into a promoted map at least by localizing the tracking map created by the portable computing device with the canonical map, wherein the canonical map includes a persistent coordinate frame representing an object or a feature of the object, the tracking map comprises persistent pose data, and the canonical map inherits one or more attributes of the tracking map. ### Claim 13 A method for rendering virtual content for extended-reality, comprising: identifying a canonical map; receiving, from a portable computing device, tracking information in a tracking map and first location metadata associated with at least a portion of the tracking map; determining a sub-portion of the canonical map based at least in part on a correspondence between the first location metadata associated with the at least the portion of the tracking map and second location metadata associated with the sub-portion of the canonical map; and merging the sub-portion of the canonical map with the at least the portion of the tracking map into a merged map. ### Claim 14 (depends on 13) The method of claim 13, wherein merging the sub-portion of the canonical map with the tracking map comprises: identifying a first plurality of features in the at least the portion of the tracking map; identifying a second plurality of features in the selected sub-portion of the canonical map; and identifying a transformation that respectively aligns the first plurality of features with the second plurality of features. ### Claim 15 (depends on 13) The method of claim 13, wherein determining the sub-portion of the canonical map comprising: determining first pose information from an image frame that is associated with the tracking map; identifying a persistent coordinate frame from the canonical map based at least in part upon the first pose information; and determining the sub-portion of the canonical map based at least in part upon the persistent coordinate frame. ### Claim 16 (depends on 13) The method of claim 13, further comprising localizing the at least the portion of the tracking map to the portable computing device, localizing the at least the portion of the tracking map comprising: determining first pose information from an image frame that is associated with the tracking map; identifying a persistent coordinate frame from a plurality of persistent coordinate frames in the canonical map based at least in part upon the first pose information; and localizing the at least the portion of the tracking map to the portable computing device based at least in part upon the persistent coordinate frame in the canonical map and the first pose information in the tracking map. ### Claim 17 A portable computing device configured for rendering virtual content for extended-reality, comprising: an image sensor; a processor coupled to the image sensor; a non-transitory computer readable storage medium having stored thereupon a sequence of instructions which, when executed by the processor, causes the process to perform a set of acts, the set of acts comprising: generating a tracking map using pose information from one or more images captured by the image sensor; identifying location metadata associated with a location represented in the tracking map; and localizing the tracking map to a canonical map at least by using a persistent coordinate frame in the canonical map and the location metadata associated with the location represented in the tracking map. ### Claim 18 (depends on 17) The portable computing device of claim 17, wherein localizing the tracking map further comprises: anchoring the canonical map to one or more anchors in the tracking map; updating the tracking map into an updated tracking map at least by adding data, which is absent in the tracking map, in the canonical map to the tracking map. ### Claim 19 (depends on 18) The portable computing device of claim 18, the set of acts further comprising generating a new canonical map at least by merging at least a portion of the updated tracking map with at least a sub-portion of the canonical map, wherein the tracking map comprises a plurality of tiles, and a tile in the tracking map encompasses a persistent pose. ### Claim 20 (depends on 17) The portable computing device of claim 17, wherein the location metadata comprises first type location metadata associated with a first portion of a plurality of locations represented in the tracking map and second type metadata associated with a second portion of the plurality of locations represented in the tracking map, the first type location metadata comprises wireless location metadata indicative of a prior location of the portable computing device or a different portable computing device with respect to a plurality of wireless network access points, and the second type location metadata comprises geo-location metadata indicative of a prior GPS location of the different portable device. ### Claim 21 (depends on 17) The portable computing device of claim 17, the set of acts further comprising: determining, based at least in part on the location metadata associated with at least a portion of the tracking map and at least one persistent coordinate frame in the canonical map, whether there exists at least one candidate sub-portion the canonical map that represents a same region of the 3D environment as a portion of the tracking map; when it is determined that there exists the at least one candidate sub-portion, performing merge processing comprising a feature-based comparison of the at least the portion of the tracking map and the candidate sub-portion; and when it is determined that no candidate sub-portion exists, skipping the feature-based comparison and identifying a different canonical map. ### Claim 22 (depends on 21) The computer device of claim 21, the set of acts further comprising: identifying a first set of features in the at least the portion of the tracking map; identifying a second set of features in the candidate sub-portion of the canonical map; and searching for a transformation that aligns the first set of features with the second set of features with an error below a threshold. ### Claim 23 A computer program product comprising a non-transitory computer readable storage medium having stored thereupon a sequence of instructions which, when executed by a processor, causes the processor to perform a set of acts for rendering virtual content for extended-reality, the set of acts comprising: identifying, at a portable computing device, tracking information in a tracking map and first location metadata associated with at least a portion of the tracking map; determining a sub-portion of a canonical map based at least in part on a correspondence between the first location metadata associated with the at least the portion of the tracking map and second location metadata associated with the sub-portion of the canonical map; and merging the sub-portion of the canonical map with the at least the portion of the tracking map into a merged map. ### Claim 24 (depends on 23) The computer program product of claim 23, the set of acts further comprising: determining first pose information from an image frame that is associated with the tracking map; identifying a persistent coordinate frame from the canonical map based at least in part upon the first pose information; and determining the sub-portion of the canonical map based at least in part upon the persistent coordinate frame. ### Claim 25 (depends on 23) The computer program product of claim 23, the set of acts further comprising: determining first pose information from an image frame that is associated with the tracking map; identifying a persistent coordinate frame from a plurality of persistent coordinate frames in the canonical map based at least in part upon the first pose information; and localizing the at least the portion of the tracking map to the portable computing device based at least in part upon the persistent coordinate frame in the canonical map and the first pose information in the tracking map. ### Claim 26 (depends on 23) The computer program product of claim 23, the set of acts further comprising: identifying an object or a feature thereof from an image frame as a persistent pose; transforming the pose into a persistent coordinate frame; and storing the persistent coordinate frame in the canonical map, wherein the persistent coordinate frame includes or is associated with the transformation.