# US 2023/0273676 A1 · Methods and apparatuses for determining and/or evaluating localizing maps of image display devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0273676 A1
- Document ID: US20230273676A1
- Status: Published application
- Kind: A1
- Application number: 18/314,289
- Filed: 2023-05-09
- Published: 2023-08-31
- Earliest claimed: 2018-07-24
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Divya SHARMA; Ali SHAHROKNI; Anush MOHAN; Prateek SINGHAL; Xuan ZHAO; Sergiu SIMA; Benjamin LANGMANN
- Web page: https://patentgrep.com/patent/US20230273676A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- G06F 1/1688: {the I/O peripheral being integrated loudspeakers}
- G06F 1/1694: {the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 15/005: {General purpose rendering architectures}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An apparatus configured to be worn on a head of a user, includes: a screen configured to present graphics to the user; a camera system configured to view an environment in which the user is located; and a processing unit configured to determine a map based at least in part on output(s) from the camera system, wherein the map is configured for use by the processing unit to localize the user with respect to the environment; wherein the processing unit of the apparatus is also configured to obtain a metric indicating a likelihood of success to localize the user using the map, and wherein the processing unit is configured to obtain the metric by computing the metric or by receiving the metric.

## Claims

### Claim 1

A method performed by an apparatus that is configured to be worn on a head of a user, the apparatus having a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit, the method comprising:
obtaining, by the processing unit, output from the camera system;
determining a map by the processing unit based at least in part on the output(s) from the camera system, wherein the map is configured for use by the processing unit to localize the user with respect to the environment; and
determining, by the processing unit, a score for an image obtained from the camera system, the score indicating how well the image is constrained with respect to a map segment for forming the map.

### Claim 2 (depends on 1)

The method of claim 1, wherein the score is determined based on a number of reference points detected in the image.

### Claim 3 (depends on 1)

The method of claim 1, further comprising performing, by the processing unit, data sanitization based on the score.

### Claim 4 (depends on 3)

The method of claim 3, wherein the act of performing the data sanitization comprises removing a constraint of the image, or removing the image.

### Claim 5 (depends on 4)

The method of claim 4, wherein the act of performing the data sanitization comprises removing the constraint of the image, or removing the image, when the score is below a threshold.

### Claim 6 (depends on 3)

The method of claim 3, wherein the sanitization is performed to remove or to disregard data that would otherwise provide an undesirable contribution for the map if the data is used to determine the map.

### Claim 7 (depends on 1)

The method of claim 1, wherein camera system comprises a plurality of cameras, wherein the data comprises a set of images generated by the respective cameras, and wherein the sanitization is performed to remove or disregard the set of images.

### Claim 8 (depends on 1)

The method of claim 1, further comprising performing a bundle adjustment to adjust one or more rays associated with one or more images from the camera system, wherein the bundle adjustment is performed after the sanitization is performed, wherein the image for which the score is determined is one of the one or more images, or is different from the one or more images,
wherein the bundle adjustment is performed as a part of a global optimization.

### Claim 9 (depends on 8)

The method of claim 8, further comprising performing, by the processing unit, the global optimization based on the one or more images from the camera system and three-dimensional reference points,
wherein the global optimization is performed also based on a relative orientation between cameras of the camera system.

### Claim 10 (depends on 1)

The method of claim 1, wherein the map is determined by:
determining multiple map segments, wherein the multiple map segment comprise the map segment; and
connecting the map segments,
wherein the portion of the map comprises one of the map segments.

### Claim 11 (depends on 1)

The method of claim 1, wherein the camera system is configured to provide additional images, the additional images generated by the camera system before the image for which the score is determined is generated,
wherein the act of determining the map comprises determining a first map segment of the map segments by linking the additional images, and wherein the additional images are generated in sequence by the camera system.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
determining respective scores of the additional images; and
starting, by the processing unit, a second map segment of the map segments in response to the score of the image from the camera system meeting a criterion,
wherein the second map segment is started when the score indicates that the image has a degree of constraint with respect to the first map segment that is below a threshold.

### Claim 13 (depends on 1)

The method of claim 1, wherein the map is determined by:
determining multiple map segments, wherein the multiple map segment comprise the map segment; and
connecting the map segments;
wherein the portion of the map comprises one of the map segments.

### Claim 14 (depends on 13)

The method of claim 13, wherein the camera system is configured to provide additional images, the additional images generated by the camera system before the image for which the score is determined is generated, wherein the act of determining the map comprises determining a first map segment of the map segments by linking the additional images, and wherein the additional images are generated in sequence by the camera system.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
determining respective scores of the additional images; and
starting, by the processing unit, a second map segment of the map segments in response to the score of the image from the camera system meeting a criterion,
wherein the second map segment is started when the score indicates that the image has a degree of constraint with respect to the first map segment that is below a threshold.

### Claim 16 (depends on 1)

The method of claim 1, further comprising determining a metric indicating a likelihood of success to localize the user using the map.

### Claim 17 (depends on 16)

The method of claim 16, wherein the metric is determined based on a co-visibility of a point of interest that is associated with different camera positions,

### Claim 18 (depends on 16)

The method of claim 16, wherein the metric is for one of a plurality of cells, and each of the cells represents a three dimensional space of a portion of the environment,
wherein the camera system is configured to obtain multiple images, the multiple images comprising the image for which the score is determined, and
wherein the metric is determined for one of the plurality of cells by:
identifying a subset of the images that belong to a same range of viewing directions;
determining respective scores for the images in the subset of the images; and
summing the scores to obtain a total score.

### Claim 19

A method performed by an apparatus that is configured to be worn on a head of a user, the apparatus having a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit, the method comprising:
obtaining, by the processing unit, output(s) from the camera system;
determining a map by the processing unit based at least in part on the output(s) from the camera system, wherein the map is configured for use by the processing unit to localize the user with respect to the environment; and
obtaining, by the processing unit, a score for an image obtained from the camera system, the score indicating how well the image is constrained with respect to a map segment for forming the map,
wherein the act of obtaining the score comprises computing the score or receiving the score.

### Claim 20

An apparatus configured to be worn on a head of a user, comprises:
a screen configured to present graphics to the user;
a camera system configured to view an environment in which the user is located; and
a processing unit configured to determine a map based at least in part on output(s) from the camera system,
wherein the map is configured for use by the processing unit to localize the user with respect to the environment, and
wherein the processing unit is configured to determine a score for an image obtained from the camera system, the score indicating how well the image is constrained with respect to a map segment for forming the map.
