# US 2022/0036078 A1 · Methods and apparatuses for determining and/or evaluating localizing maps of image display devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036078 A1
- Document ID: US20220036078A1
- Status: Published application
- Kind: A1
- Application number: 17/500,798
- Filed: 2021-10-13
- Published: 2022-02-03
- Earliest claimed: 2018-07-24
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Divya SHARMA; Ali SHAHROKNI; Anush MOHAN; Prateek SINGHAL; Xuan ZHAO; Sergiu SIMA; Benjamin LANGMANN
- Web page: https://patentgrep.com/patent/US20220036078A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- G06F 1/1688: {the I/O peripheral being integrated loudspeakers}
- G06F 1/1694: {the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 15/005: {General purpose rendering architectures}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An apparatus configured to be worn on a head of a user, includes: a screen configured to present graphics to the user; a camera system configured to view an environment in which the user is located; and a processing unit configured to determine a map based at least in part on output(s) from the camera system, wherein the map is configured for use by the processing unit to localize the user with respect to the environment; wherein the processing unit of the apparatus is also configured to obtain a metric indicating a likelihood of success to localize the user using the map, and wherein the processing unit is configured to obtain the metric by computing the metric or by receiving the metric.

## Claims

### Claim 1

An augmented reality (AR) apparatus configured to be worn on a head of a user, comprising:
a processing unit configured to:
determine a map based at least in part on an image, wherein the map is configured for use by the processing unit to localize the user with respect to the environment, and
obtain a metric indicating a likelihood of success to localize the user using the map by computing the metric or by receiving the metric.

### Claim 2 (depends on 1)

The AR apparatus of claim 1, wherein the processing unit is configured to determine the metric based on a co-visibility of a point of interest that is associated with different camera positions.

### Claim 3 (depends on 1)

The AR apparatus of claim 1, wherein the metric indicates a number of reference points that are useable to localize the user with respect to the environment.

### Claim 4 (depends on 1)

The AR apparatus of claim 1, wherein the metric indicates the likelihood of success to localize the user in one or more viewing directions.

### Claim 5 (depends on 1)

The AR apparatus of claim 1, wherein the processing unit is configured to determine the metric based on a number of times a point of interest is detected from different camera positions.

### Claim 6 (depends on 1)

The AR apparatus of claim 1, wherein the processing unit is configured to determine the metric without determining any convex hull.

### Claim 7 (depends on 1)

The AR apparatus of claim 1, wherein the metric has a value that is based on directionality.

### Claim 8 (depends on 1)

The AR apparatus of claim 1, wherein the metric is for one of a plurality of cells, each of the cells representing a three dimensional space of a portion of the environment, and wherein the metric has a value that is based on a position within the one of the plurality of cells.

### Claim 9 (depends on 1)

The AR apparatus of claim 1, wherein the metric is for one of a plurality of cells, and each of the cells represents a three dimensional space of a portion of the environment.

### Claim 10 (depends on 9)

The AR apparatus of claim 9, wherein the camera system is configured to obtain multiple images, and wherein the processing unit is configured to determine the metric for one of the plurality of cells by:
identifying a subset of the images that belong to a same range of viewing directions;
determining respective scores for the images in the subset of the images; and
summing the scores to obtain a total score.

### Claim 11 (depends on 10)

The AR apparatus of claim 10, wherein the processing unit is also configured to determine an average score by dividing the total score by a number of the images in the subset of the images.

### Claim 12 (depends on 10)

The AR apparatus of claim 10, wherein the processing unit is configured to determine the respective scores by accessing a co-visibility graph that associates reference points with the multiple images.

### Claim 13 (depends on 12)

The AR apparatus of claim 12, wherein the co-visibility graph indicates which of the reference points is visible in which of the multiple images.

### Claim 14 (depends on 10)

The AR apparatus of claim 10, wherein the processing unit is configured to determine each of the respective scores by determining a number of reference point(s) that is detected in the corresponding one of the images in the subset of images.

### Claim 15 (depends on 1)

The AR apparatus of claim 1, wherein the processing unit is also configured to determine an area score indicating a degree of coverage by the map.

### Claim 16 (depends on 1)

The AR apparatus of claim 1, wherein the processing unit is configured to determine the metric by:
obtaining a plurality of images from the camera system; and
determining co-visibility values, wherein each of the co-visibility values indicating a number of reference points detected in a corresponding one of the plurality of images.

### Claim 17 (depends on 1)

The AR apparatus of claim 1, wherein the processing unit is configured to determine a desired viewing direction of the camera system for improving a value of the metric.

### Claim 18 (depends on 17)

The AR apparatus of claim 17, wherein the processing unit is configured to generate the graphics based on the determined desired viewing direction, the graphics configured to instruct the user to change a current viewing direction of the camera system to the desired viewing direction.

### Claim 19 (depends on 18)

The AR apparatus of claim 18, wherein the processing unit is configured to update the metric after the desired viewing direction is achieved.

### Claim 20 (depends on 1)

The AR apparatus of claim 1, wherein the processing unit is configured to perform a sanitization to remove or to disregard data that would otherwise provide an undesirable contribution for the map if the data is used to determine the map.

### Claim 21 (depends on 20)

The AR apparatus of claim 20, wherein the data comprises an image from the camera system, and wherein the processing unit is configured to perform the sanitization by (1) removing or disregarding the image, (2) disregarding an identification of a reference point in the image, and/or (3) disregarding a ray or a line that is associated with the image.

### Claim 22 (depends on 20)

The AR apparatus of claim 20, wherein the processing unit is configured to perform a bundle adjustment to adjust one or more rays associated with one or more images from the camera system, wherein the processing unit is configured to perform the bundle adjustment after performing the sanitization to remove the data.

### Claim 23 (depends on 1)

The AR apparatus of claim 1, wherein the processing unit is configured to perform an optimization based on images from the camera system, three-dimensional reference points, and a relative orientation between cameras of the camera system.

### Claim 24 (depends on 1)

The AR apparatus of claim 1, wherein the processing unit is configured to determine a score for an image obtained from the camera system.

### Claim 25 (depends on 24)

The AR apparatus of claim 24, wherein the score indicates how well the image is constrained.

### Claim 26 (depends on 24)

The AR apparatus of claim 24, wherein the processing unit is configured to determine the score based on a Jacobian of reference points measurements.

### Claim 27 (depends on 24)

The AR apparatus of claim 24, wherein the processing unit is configured to perform data sanitization based on the score; and
wherein the processing unit is configured to remove a constraint of the image, or to remove the image, when performing the data sanitization.

### Claim 28 (depends on 1)

The AR apparatus of claim 1, wherein the processing unit is configured to determine the map by:
determining multiple map segments; and
connecting the map segments.

### Claim 29 (depends on 28)

The AR apparatus of claim 28, wherein the processing unit is configured to determine a first map segment of the map segments by obtaining images from the camera system, and linking the images, wherein the images are generated in sequence by the camera system.

### Claim 30 (depends on 29)

The AR apparatus of claim 29, wherein the processing unit is configured to:
obtain an additional image from the camera system,
determine a score for the additional image, and
start a second map segment of the map segments in response to the score of the additional image from the camera system meeting a criterion.

### Claim 31 (depends on 30)

The AR apparatus of claim 30, wherein the processing unit is configured to start the second map segment when the score indicates that the image has a degree of constraint with respect to the first map segment that is below a threshold.

### Claim 32

A method performed by an AR apparatus that is configured to be worn on a head of a user, the AR apparatus having a processing unit, the method comprising:
determining a map by the processing unit based at least in part on an image, wherein the map is configured for use by the processing unit to localize the user with respect to the environment; and
obtaining, by the processing unit, a metric indicating a likelihood of success to localize the user using the map by computing the metric or by receiving the metric.
