# US 12,762,804 B2 · Cross reality system for large scale environments > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,762,804 B2 - Document ID: US12762804B2 - Status: Granted - Kind: B2 - Application number: 18/396,682 - Filed: 2023-12-26 - Published: 2024-04-25 - Granted: 2026-10-13 - Earliest claimed: 2020-04-29 - Assignees: Magic Leap, Inc. - Applicants: Magic Leap, Inc. - Inventors: Javier Victorio Gomez Gonzalez; Miguel Andres Granados Velasquez; Mukta Prasad; Dominik Michael Kasper; Eran Guendelman; Keng-Sheng Lin - Web page: https://patentgrep.com/patent/US12762804B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A cross reality system enables portable devices to access stored maps and efficiently and accurately render virtual content specified in relation to those maps. The system may process images acquired with a portable device to quickly and accurately localize the portable device to the persisted maps by constraining the result of localization based on the estimated direction of gravity of a persisted map and the coordinate frame in which data in a localization request is posed. The system may actively align the data in the localization request with an estimated direction of gravity during the localization processing, and/or a portable device may establish a coordinate frame in which the data in the localization request is posed aligned with an estimated direction of gravity such that the subsequently acquired data for inclusion in a localization request, when posed in that coordinate frame, is passively aligned with the estimated direction of gravity. ## Claims ### Claim 1 A system that supports specification of a position of virtual content relative to one or more persisted maps in a database of persisted maps, the system comprising: a remote localization service configured to receive from a portable electronic device a localization request comprising a batch of snapshots, each snapshot of the batch of snapshots comprising a collection of features from a three-dimensional (3D) environment and posed with respect to a coordinate frame local to the portable electronic device and a vector indicating an estimated direction of gravity by the portable electronic device in the coordinate frame local to the portable electronic device, wherein: the remote localization service comprises at least one processor configured to execute computer-executable instructions, the computer-executable instructions implementing a remote localization component, the remote localization component configured to: compute a plurality of candidate localizations for the batch of snapshots by constraining degrees of freedom based on vectors indicating estimated directions of gravity of the batch of snapshots to determine transformations that each aligns the collection of features of a respective snapshot in the coordinate frame local to the portable electronic device with features of a persisted map in the database of persisted maps in a persisted coordinate frame of the persisted map, wherein computing a plurality of candidate localizations is computed for each collection of features of the batch of snapshots using a two-step process including a rough localization and a refined localization, and wherein the rough localization is performed on subsets of features in each collection of features; and generate a localization of the portable electronic device based on consensus among the plurality of candidate localizations. ### Claim 2 (depends on 1) The system of claim 1, wherein, for each snapshot of the batch of snapshots: the vector indicating the estimated direction of gravity is represented as an offset between the estimated direction of gravity and the coordinate frame local to the portable electronic device. ### Claim 3 (depends on 1) The system of claim 1, wherein: the rough localization comprises: computing a rough transformation of the coordinate frame local to the portable electronic device with respect to the persisted coordinate frame of the persisted map, and computing the transformations that each aligns the collection of features of a respective snapshot in the coordinate frame local to the portable electronic device with features of the persisted map in the persisted coordinate frame based on the rough transformation; and the rough transformation is computed by constraining degrees of freedom based on respective vectors indicating estimated directions of gravity. ### Claim 4 (depends on 3) The system of claim 3, wherein: the plurality of candidate localizations are computed without constraining the degrees of freedom based on respective vectors indicating estimated directions of gravity. ### Claim 5 (depends on 3) The system of claim 3, wherein: the rough transformation is computed by reducing six degrees of freedom to four degrees of freedom based on the respective vectors indicating estimated directions of gravity. ### Claim 6 (depends on 5) The system of claim 5, wherein the rough transformation is computed by: fixing two rotational degrees of freedom based, at least in part, on the respective vectors indicating estimated directions of gravity, and computing one rotational degree of freedom and three translational degrees of freedom based, at least in part, on the fixed two rotational degrees of freedom. ### Claim 7 (depends on 1) The system of claim 1, wherein: the plurality of candidate localizations are computed by reducing six degrees of freedom to four degrees of freedom based on the vectors indicating estimated directions of gravity. ### Claim 8 (depends on 7) The system of claim 7, wherein each candidate localization of the plurality of candidate localizations are computed by: fixing two rotational degrees of freedom based, at least in part, on the vectors indicating estimated directions of gravity, and computing one rotational degree of freedom and three translational degrees of freedom based, at least in part, on the fixed two rotational degrees of freedom. ### Claim 9 A system that supports specification of a position of virtual content relative to persisted maps in a database of persisted maps, the system comprising: a remote localization service configured to receive from a portable electronic device a localization request comprising a batch of snapshots, each snapshot of the batch of snapshots comprising a collection of features from a three-dimensional (3D) environment and posed in a coordinate frame local to the portable electronic device having a dimension aligned with respect to an estimated direction of gravity, wherein: the remote localization service comprises at least one processor configured to execute computer-executable instructions, the computer-executable instructions implementing a remote localization component, the remote localization component configured to: compute a plurality of candidate localizations for the batch of snapshots by, for each snapshot of the batch of snapshots, performing a process to determine as a candidate localization a transformation between the collection of features of the snapshot and a portion of a persisted map in the database of persisted maps, wherein computing a plurality of candidate localizations is computed for each collection of features of the batch of snapshots using a two-step process including a rough localization and a refined localization, wherein the rough localization is performed on subsets of features in each collection of features, and wherein: the portion of the persisted map has an associated estimated direction of gravity; and the process of determining is constrained to determine transformations that align the estimated direction of gravity for the collection of features of the snapshot with the associated estimated direction of gravity of the portion of the persisted map; and generate a localization of the portable electronic device based on consensus among the plurality of candidate localizations. ### Claim 10 (depends on 9) The system of claim 9, wherein: the rough localization comprises: computing a rough transformation of the coordinate frame local to the portable electronic device with respect to a persisted coordinate frame of the persisted map, and computing transformations that each align the collection of features of a respective snapshot in the coordinate frame local to the portable electronic device with features of the persisted map in the persisted coordinate frame based on the rough transformation; and the rough transformation is computed by constraining degrees of freedom based on respective vectors indicating estimated directions of gravity. ### Claim 11 (depends on 10) The system of claim 10, wherein: the plurality of candidate localizations are computed without constraining the degrees of freedom based on the respective vectors indicating estimated directions of gravity. ### Claim 12 (depends on 9) The system of claim 9, wherein: the plurality of candidate localizations are computed by reducing six degrees of freedom to four degrees of freedom based on respective vectors indicating estimated directions of gravity. ### Claim 13 (depends on 12) The system of claim 12, wherein each candidate localization of the plurality of candidate localizations are computed by: fixing two rotational degrees of freedom based, at least in part, on the respective vectors indicating estimated directions of gravity, and computing one rotational degree of freedom and three translational degrees of freedom based, at least in part, on the fixed two rotational degrees of freedom. ### Claim 14 (depends on 10) The system of claim 10, wherein: the rough transformation is computed by reducing six degrees of freedom to four degrees of freedom based on the respective vectors indicating estimated directions of gravity. ### Claim 15 (depends on 14) The system of claim 14, wherein the rough transformation is computed by: fixing two rotational degrees of freedom based, at least in part, on the respective vectors indicating estimated directions of gravity, and computing one rotational degree of freedom and three translational degrees of freedom based, at least in part, on the fixed two rotational degrees of freedom. ### Claim 16 (depends on 9) The system of claim 9, further comprising a map merge service comprising: a map alignment component configured to determine an alignment between a tracking map of the portable electronic device and a persisted map in the database of persisted maps, the tracking map comprising at least a portion of the batch of snapshots; and a gravity estimate component configured to compute an estimated direction of gravity for the tracking map of the portable electronic device based, at least in part, on an estimate of a direction of gravity with respect to the persisted map and the determined alignment between the tracking map and the persisted map, wherein the estimated direction of gravity for the tracking map is sent to the portable electronic device. ### Claim 17 An electronic device configured to operate within a system, the electronic device comprising: one or more sensors configured to capture information about a three-dimensional (3D) environment, the captured information comprising a plurality of images; and at least one processor configured to execute computer executable instructions, wherein the computer executable instructions comprise instructions for: extracting a plurality of collections of features from the plurality of images of the 3D environment; determining an estimated direction of gravity with respect to a coordinate frame local to the electronic device; aligning, as an aligned coordinate frame, a dimension of the coordinate frame local to the electronic device with the estimated direction of gravity; expressing positions of the features of the plurality of collections of features in the aligned coordinate frame local to the electronic device; sending to a remote localization service of the system a plurality of snapshots, each snapshot of the plurality of snapshots comprising a collection of features posed in the aligned coordinate frame local to the electronic device; computing a plurality of candidate localizations for the plurality of snapshots, wherein computing a plurality of candidate localizations is computed for each collection of features of the plurality of snapshots using a two-step process including a rough localization and a refined localization, wherein the rough localization is performed on subsets of features in each collection of features; and receiving from the remote localization service a pose of the electronic device with respect to a persisted map in a database of persisted maps. ### Claim 18 (depends on 17) The electronic device of claim 17, wherein determining the estimated direction of gravity comprises: receiving the estimated direction of gravity from a map merge service of the system. ### Claim 19 (depends on 17) The electronic device of claim 17, wherein determining the estimated direction of gravity comprises: sending at least a portion of the plurality of collections of features to a map merge service of the system such that the estimated direction of gravity is based on the at least a portion of the plurality of collections of features. ### Claim 20 (depends on 17) The electronic device of claim 17, wherein the plurality of collections of features comprises descriptors for individual features.