# US 12,002,170 B2 · Cross reality system with buffering for localization accuracy

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,170 B2
- Document ID: US12002170B2
- Status: Granted
- Kind: B2
- Application number: 17/894,336
- Filed: 2022-08-24
- Published: 2022-12-22
- Granted: 2024-06-04
- Earliest claimed: 2020-02-26
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Ali Shahrokni; Keng-Sheng Lin; Xuan Zhao; Christian Ivan Robert Moore; Ashwin Swaminathan
- Web page: https://patentgrep.com/patent/US12002170B2

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- H04L 67/52: specially adapted for the location of the user terminal
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30244: Camera pose
- G06T 2210/61: Scene description
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A system form localizing an electronic device with dynamic buffering identifies, from the buffer, a first set of features that is extracted from a first image captured by the electronic device and receives, at the system, a second set of features that is extracted from a second image captured by the electronic device. The system further determines a first characteristic for the first set of features and a second characteristic for the second set of features and determines whether a triggering condition for dynamically changing a size of the buffer is satisfied based at least in part upon the first characteristic for the first set of features and the second characteristic for the second set of features.

## Claims

### Claim 1

An electronic device for rendering virtual content with dynamic buffering in a cross reality environment, comprising:
a sensor that captures image information about a three-dimensional (3D) environment, the image information comprising a plurality of images;
a processor that executes computer executable instructions, wherein the computer executable instructions comprise instructions, when executed by the processor, cause the processor to perform a set of acts, the set of acts comprising:
generating a local coordinate frame for representing an object or a portion thereof in the 3D environment;
determining a first set of features from the image information of the 3D environment;
forwarding the first set of features expressed in the local coordinate frame to a localization service on a remote server; and
receiving, from the localization service, at least one transform that localizes the local coordinate frame to a separate coordinate frame of a canonical map of a shareable world model,
wherein the set of acts further comprises sending a request for localization to the localization service based at least in part upon first cross entropy for the first set of features and second cross entropy for a second set of features that has been transmitted to the localization service.

### Claim 2 (depends on 1)

The electronic device of claim 1, wherein the first set of features indicates a change in position information for the electronic device with respect to the second set of features that has been transmitted to the localization service, and the change in the position information comprises a change in translation distance, rotation angle, or combinations thereof exceeding a threshold.

### Claim 3 (depends on 1)

The electronic device of claim 1, wherein the second cross entropy comprises data pertaining to a change in translation distance, a rotation angle, a viewpoint between a first image and a second image, a wireless fingerprint of the electronic device capturing the second image, pose tracking data of the electronic device based at least in part on a tracking map, or visual similarity of the second image to the first image.

### Claim 4 (depends on 1)

The electronic device of claim 1, wherein the set of acts further comprises generating a tracking map with the first set of features, wherein the tracking map includes one or more point nodes respectively representing one or more headposes pertaining to the electronic device, and the canonical map comprises one or more persistent coordinate frames.

### Claim 5 (depends on 1)

The electronic device of claim 1, wherein the set of acts further comprises forwarding position information expressed in the local coordinate frame to the localization service on the remote server.

### Claim 6 (depends on 1)

The electronic device of claim 1, wherein the set of acts further comprises triggering the instructions for determining the first set of features to determine the first set of features in response to a detection of an event.

### Claim 7 (depends on 6)

The electronic device of claim 6, wherein the event is selected from the group consisting of:
a change in output of an inertial measurement unit (IMU) that indicates translation of the electronic device above a threshold distance;
a change in a computed pose of the electronic device with respect to a tracking map that indicates the translation of the electronic device above the threshold distance;
a change in the output of the IMU that indicates rotation of the electronic device above a threshold angle;
a change in the computed pose of the electronic device with respect to the tracking map that indicates the rotation of the electronic device above the threshold angle;
a change in wireless fingerprint that indicates the translation of the electronic device above the threshold distance;
a change in value of a descriptor, representative of content of an image frame, for image frames captured sequentially by image sensors on the electronic device; and
a change in another measurable parameter that indicates the computed pose of the electronic device has changed by more than a threshold amount.

### Claim 8 (depends on 1)

The electronic device of claim 1, wherein the transform aligns the first set of features with the second set of features in the canonical map of the shareable world model.

### Claim 9 (depends on 8)

The electronic device of claim 8, wherein the second set of features defines the separate coordinate frame.

### Claim 10 (depends on 1)

The electronic device of claim 1, wherein the separate coordinate frame is a persistent coordinate frame.

### Claim 11 (depends on 1)

The electronic device of claim 1, wherein the set of acts further comprises forwarding pose information relating to the sensor to the localization service on the remote server.

### Claim 12 (depends on 11)

The electronic device of claim 11, wherein the transform is computed based on the first set of features and the pose information.

### Claim 13 (depends on 1)

The electronic device of claim 1, wherein the set of acts further comprises:
determining a plurality of sets of features from the image information of the 3D environment, the plurality of sets of features including the first set of features; and
forwarding the plurality of sets of features expressed in the local coordinate frame to the localization service on the remote server, and
wherein the transform is computed based on the plurality of sets of features.

### Claim 14

An electronic device for rendering virtual content with dynamic buffering in a cross reality environment, comprising:
a sensor that captures image information about a three-dimensional (3D) environment, the image information comprising a plurality of images;
a processor that executes computer executable instructions, wherein the computer executable instructions comprise instructions, when executed by the processor, cause the processor to perform a set of acts, the set of acts comprising:
generating a local coordinate frame for representing an object or a portion thereof in the 3D environment;
determining a first set of features from the image information of the 3D environment;
forwarding the first set of features expressed in the local coordinate frame to a localization service on a remote server; and
receiving, from the localization service, at least one transform that localizes the local coordinate frame to a separate coordinate frame of a canonical map of a shareable world model wherein the set of acts further comprises:
determining a plurality of sets of features from the image information of the 3D environment, the plurality of sets of features including the first set of features; and
forwarding the plurality of sets of features expressed in the local coordinate frame to the localization service on the remote server, and
wherein the transform is computed based on the plurality of sets of features, and
wherein forwarding the plurality of sets of features to the localization service on the remote server increases a cross-entropy of the plurality of sets of features.