# US 12,118,685 B2 · Localization accuracy response

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,685 B2
- Document ID: US12118685B2
- Status: Granted
- Kind: B2
- Application number: 18/378,264
- Filed: 2023-10-10
- Published: 2024-02-01
- Granted: 2024-10-15
- Earliest claimed: 2021-04-06
- Applicants: Apple Inc.
- Inventors: Jean-Charles Bernard Marcel Bazin; Anselm Grundhoefer; Bart Trzynadlowski; Thomas J. Moore
- Web page: https://patentgrep.com/patent/US12118685B2

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 15/00: Three-dimensional [3D] image rendering
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30244: Camera pose
- G06T 2219/004: Annotating, labelling
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Various implementations disclosed herein include devices, systems, and methods that provide XR in which virtual objects are positioned based on the accuracy of localizing an electronic device in a physical environment. In some implementations, the technique assesses the accuracy of localization (e.g., centimeter-level accuracy, room-level accuracy, and building-level accuracy) and dynamically adjusts a display strategy. In some implementations, the technique determines a condition causing inaccuracy (e.g., a semantic condition such as “too fast”, “too far”, “too dark”), and provides a notification (e.g., “too fast-slow down”, “too far-move closer”, “too dark-turn on a light”) at the electronic device based on the condition causing the inaccuracy in the localization.

## Claims

### Claim 1

A method comprising:
at a processor:
performing a localization of an electronic device within a physical environment, the localization based on sensor data obtained by the electronic device;
determining a condition causing inaccuracy;
providing a view of an extended reality (XR) environment, wherein the inaccuracy in the localization affects the inclusion of virtual objects in the view of the XR environment; and
providing a notification at the electronic device based on the condition causing inaccuracy in the localization.

### Claim 2 (depends on 1)

The method of claim 1, further comprising correcting the condition causing the inaccuracy in the localization based on the notification.

### Claim 3 (depends on 2)

The method of claim 2, wherein correcting the condition causing the inaccuracy comprises a user of the electronic device modifying the physical environment or changing the sensor data used to perform the localization of the electronic device within the physical environment.

### Claim 4 (depends on 1)

The method of claim 1, wherein the condition causing the inaccuracy in the localization is determined using a machine learning model.

### Claim 5 (depends on 4)

The method of claim 4, wherein the machine learning model is trained using a classification error loss determined by comparing an estimated classification of the condition and ground truth classification of the condition causing the inaccuracy.

### Claim 6 (depends on 1)

The method of claim 1, wherein providing the view of the XR environment stops displaying the virtual objects in the view of the XR environment.

### Claim 7 (depends on 1)

The method of claim 1, wherein providing the view of the XR environment displays the virtual objects at fixed positions on a display of the electronic device in the view of the XR environment.

### Claim 8 (depends on 1)

The method of claim 1, wherein providing the view of the XR environment displays the virtual objects at fixed positions relative to the electronic device in the view of the XR environment.

### Claim 9 (depends on 1)

The method of claim 1, further comprising determining an accuracy of the localization.

### Claim 10 (depends on 9)

The method of claim 9, wherein the accuracy of the localization is determined using a machine learning model, wherein the machine learning model is trained using an error loss determined by comparing estimated poses of electronic devices and ground truth poses of electronic devices.

### Claim 11

A system comprising:
memory; and
one or more processors at a device coupled to the memory, wherein the memory comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
performing a localization of an electronic device within a physical environment, the localization based on sensor data obtained by the electronic device;
determining a condition causing inaccuracy;
providing a view of an extended reality (XR) environment, wherein the inaccuracy in the localization affects the inclusion of virtual objects in the view of the XR environment; and
providing a notification at the electronic device based on the condition causing inaccuracy in the localization.

### Claim 12 (depends on 11)

The system of claim 11, wherein the operations further comprise correcting the condition causing the inaccuracy in the localization based on the notification.

### Claim 13 (depends on 12)

The system of claim 12, wherein correcting the condition causing the inaccuracy comprises a user of the electronic device modifying the physical environment or changing the sensor data used to perform the localization of the electronic device within the physical environment.

### Claim 14 (depends on 11)

The system of claim 11, wherein the condition causing the inaccuracy in the localization is determined using a machine learning model.

### Claim 15 (depends on 14)

The system of claim 14, wherein the machine learning model is trained using a classification error loss determined by comparing an estimated classification of the condition and ground truth classification of the condition causing the inaccuracy.

### Claim 16 (depends on 11)

The system of claim 11, wherein providing the view of the XR environment stops displaying the virtual objects in the view of the XR environment.

### Claim 17 (depends on 11)

The system of claim 11, wherein providing the view of the XR environment displays the virtual objects at fixed positions on a display of the electronic device in the view of the XR environment.

### Claim 18 (depends on 11)

The system of claim 11, wherein providing the view of the XR environment displays the virtual objects at fixed positions relative to the electronic device in the view of the XR environment.

### Claim 19 (depends on 11)

The system of claim 11, wherein the operations further comprise determining an accuracy of the localization.

### Claim 20

A non-transitory computer-readable storage medium, storing program instructions executable via one or more processors to perform operations comprising:
performing a localization of an electronic device within a physical environment, the localization based on sensor data obtained by the electronic device;
determining a condition causing inaccuracy;
providing a view of an extended reality (XR) environment, wherein the inaccuracy in the localization affects the inclusion of virtual objects in the view of the XR environment; and
providing a notification at the electronic device based on the condition causing inaccuracy in the localization.