# US 2021/0173215 A1 · Portable electronic device for mixed reality headset

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0173215 A1
- Document ID: US20210173215A1
- Status: Published application
- Kind: A1
- Application number: 17/181,236
- Filed: 2021-02-22
- Published: 2021-06-10
- Earliest claimed: 2021-02-22
- Assignees: Telefonaktiebolaget LM Ericsson (publ)
- Applicants: Telefonaktiebolaget LM Ericsson (publ)
- Inventors: José ARAÚJO; Lars ANDERSSON; Diego GONZALEZ MORIN; Ioannis KARAGIANNIS; AmirHossein TAHER KOUHESTANI
- Web page: https://patentgrep.com/patent/US20210173215A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G09G 5/38: with means for controlling the display position
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/0172: {characterised by optical features}
- G02B 27/0179: {Display position adjusting means not related to the information to be displayed}
- G06F 3/04812: Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 7/90: Determination of colour characteristics
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G09G 3/001: {using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background (slide projectors per se G03B23/00 = 42 HP)}
- G09G 5/02: characterised by the way in which colour is displayed {(details of colour display specific for CRTs G09G1/28; specific for flat matrix panels other than liquid crystal displays G09G3/2003; specific for liquid crystal displays G09G3/3607)}
- G09G 5/10: Intensity circuits
- G02B 2027/0118: {comprising devices for improving the contrast of the display / brillance control visibility}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06T 2213/12: Rule based animation
- G06T 2215/16: Using real world measurements to influence rendering
- G09G 2320/066: for control of contrast
- G09G 2320/0666: for control of colour parameters, e.g. colour temperature
- G09G 2320/0686: with two or more screen areas displaying information with different brightness or colours
- G09G 2320/0693: Calibration of display systems
- G09G 2354/00: Aspects of interface with display user
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A portable electronic device is used with a mixed reality (MR) headset. The MR headset includes a lens and a holder that retains the portable electronic device in a defined orientation relative to the lens. The portable electronic device includes a camera, a display device, and a processor. The camera outputs video frames and is arranged to view the lens. The display device is arranged to display information that is projected on the lens for reflection directly or indirectly toward the user's eyes and the camera. The processor performs operations that include processing the video frames from the camera to identify locations of at least one real-world feature, displaying information on the display device, and controlling at least one of the processing of the video frames and the displaying of the information on the display device to at least partially reduce occurrence in the video frames of the displayed information.

## Claims

### Claim 1

A portable electronic device for a mixed reality (MR) headset including a lens through which a user wearing the MR headset can view real-world features, the portable electronic device comprising:
a camera configured to output video frames and arranged to view at least a portion of the lens of the MR headset;
a display device arranged to display information that is to be projected on the lens for reflection directly or indirectly toward the user's eyes and the camera;
at least one processor operationally connected to the camera and to the display device, and configured to perform operations comprising:
processing the video frames from the camera to identify a location of a real-world feature captured in the video frames,
wherein the location of the real-world feature captured in the video frames from the camera is identified based on processing the video frames that were captured while the display device ceases to display information for projection on the lens and the processing prevents the identification of the location of the real-world feature from being performed on at least some of the video frames that were captured while the display device displays information for projection on the lens.

### Claim 2 (depends on 1)

The portable electronic device of claim 1, wherein the operations cease the processing of the video frames which were captured by the camera while the information is determined to be displayed on the display for projection on the lens, and the operations resume the processing of the video frames which were captured by the camera while the information is determined to not be displayed on the display.

### Claim 3 (depends on 2)

The portable electronic device of claim 2, wherein the operations to resume the processing of the video frames which were captured by the camera while the information is determined to not be displayed on the display, comprise skipping without processing to identify location of the real-world feature in at least some of the video frames that were captured by the camera while the information is determined to be displayed on the display for projection on the lens.

### Claim 4 (depends on 3)

The portable electronic device of claim 3, wherein the operations to skip without processing to identify location of the real-world feature in at least some of the video frames that were captured by the camera while the information is determined to be displayed on the display for projection on the lens, comprise skipping all of the video frames that were captured by the camera while the information is determined to be displayed on the display for projection on the lens.

### Claim 5 (depends on 1)

The portable electronic device of claim 1, wherein the operations repetitively alternate between displaying the information on the display device and then ceasing the display of the information while initiating the processing of the video frames from the camera, which are captured by the camera while the display of the information is ceased, to identify location of the real-world feature, wherein the processing of the video frames is controlled so at least some of the video frames captured by the camera while the information is displayed on the display device are skipped without being processed to identify location of the real-world feature.

### Claim 6 (depends on 5)

The portable electronic device of claim 5, wherein all the video frames that were captured by the camera while the information is displayed on the display device are skipped without being processed to identify location of the real-world feature.

### Claim 7 (depends on 5)

The portable electronic device of claim 5, wherein the operations further comprise:
determining when a display quality prioritization rule is satisfied; and
responsive to the display quality prioritization rule becoming satisfied, increasing a duty cycle of a first time duration during which the display device is repetitively refreshed to display the information relative to a second time duration during which the processing of the video frames from the camera, which are captured by the camera while the display of the information is ceased, is performed to identify location of the real-world feature.

### Claim 8 (depends on 5)

The portable electronic device of claim 5, wherein the operations further comprise:
determining when a camera video prioritization rule is satisfied; and
responsive to the camera video prioritization rule becoming satisfied, decreasing a duty cycle of a first time duration during which the display device is repetitively refreshed to display the information relative to a second time duration during which the processing of the video frames from the camera, which are captured by the camera while the display of the information is ceased, is performed to identify location of the real-world feature.

### Claim 9 (depends on 1)

The portable electronic device of claim 1, wherein the operations further comprise:
determining when a display quality prioritization rule is satisfied; and
responsive to the display quality prioritization rule becoming satisfied, increasing a frequency at which the display device is repetitively refreshed to display the information and/or decreasing a frequency at which the processing of the video frames from the camera, which are captured by the camera while the display of the information is ceased, is performed to identify location of the real-world feature.

### Claim 10 (depends on 1)

The portable electronic device of claim 1, wherein the operations further comprise:
determining when a camera video prioritization rule is satisfied; and
responsive to the camera video prioritization rule becoming satisfied, decreasing a frequency at which the display device is repetitively refreshed to display the information and/or increasing a frequency at which the processing of the video frames from the camera, which are captured by the camera while the display of the information is ceased, is performed to identify location of the real-world feature.

### Claim 11 (depends on 1)

The portable electronic device of claim 1, wherein the operations further comprise:
determining an amount of overlap occurring between: 1) a region of the video frames from the camera that captures the information being displayed by the display device; and 2) location of the real-world feature in the video frames; and
based on the amount of overlap satisfying a defined rule, initiating the operations to prevent the identification of the location of the real-world feature from being performed on at least some of the video frames that were captured while the display device displays information for projection on the lens.

### Claim 12 (depends on 1)

The portable electronic device of claim 1, wherein the operations further comprise:
displaying a calibration pattern on the display device;
processing at least one of the video frames to determine a region of the at least one of the video frames containing the calibration pattern;
ceasing display of the calibration pattern on the display device;
following cessation of display of the calibration pattern, processing the video frames subsequently captured by the camera to determine when location of the real-world feature enters the region of the video frames; and
responsive to the location of the real-world feature entering the region of the video frames, initiating the operations to prevent the identification of the location of the real-world feature from being performed on at least some of the video frames that were captured while the display device displays information for projection on the lens.

### Claim 13 (depends on 1)

The portable electronic device of claim 1, wherein:
the operations further comprise processing the video frames from the camera to identify a first region of the video frames from the camera that captures the information being displayed by the display device; and
the operations to prevent the identification of the location of the real-world feature from being performed on at least some of the video frames that were captured while the display device displays information for projection on the lens, further comprise:
processing the first region of the video frames at a lower frequency than a second region of the video frames, which is outside the first region of the video frames and does not capture the information being displayed by the display device, to identify locations of the real-world feature which can be in the first region and/or the second region.

### Claim 14 (depends on 1)

The portable electronic device of claim 1, wherein the operations further comprise:
processing the video frames from the camera to identify a region of the display device that displays a portion of the information and which at least partially overlaps a location of the real-world feature in the video frames; and
decreasing brightness and/or contrast of the region of the display device.

### Claim 15 (depends on 1)

The portable electronic device of claim 1, wherein the operations further comprise:
processing the video frames from the camera to identify a region of the display device that displays a portion of the information and which at least partially overlaps a location of the real-world feature in the video frames; and
determining a primary color of the real-world feature and selecting a color of the information that is then displayed in the region of the display device to differentiate from the primary color of the real-world feature.

### Claim 16 (depends on 1)

The portable electronic device of claim 1, wherein the operations further comprise:
processing the video frames from the camera to identify a first region of the display device that displays a portion of the information and which at least partially overlaps a location of the real-world feature in the video frames; and
increasing brightness and/or contrast of a second region of the display device that does not overlap the first region.

### Claim 17 (depends on 1)

The portable electronic device of claim 1, wherein the operations further comprise:
processing the video frames from the camera to identify a first region of the display device that displays a portion of the information and which at least partially overlaps a location of the real-world feature in the video frames; and
determining a primary color of the real-world feature and selecting a color of the information that is then displayed in a second region of the display device that does not overlap the first region to differentiate from the primary color of the real-world feature.

### Claim 18 (depends on 1)

The portable electronic device of claim 1, wherein the portable electronic device is connectable to a holder of the MR headset that is configured to releasably retain the portable electronic device in a defined orientation relative to the lens of the MR headset, wherein the camera of the portable electronic device is arranged by the holder to view at least a portion of the lens of the MR headset.

### Claim 19

A method by a portable electronic device for a mixed reality (MR) headset including a lens through which a user wearing the MR headset can view real-world features, wherein a display device of the portable electronic device is arranged to display information that is projected on the lens for reflection directly or indirectly toward the user's eyes and a camera of the portable electronic device, the method comprising:
process the video frames from the camera to identify a location of a real-world feature captured in the video frames, wherein the location of the real-world feature captured in the video frames from the camera is identified based on processing the video frames that were captured while the display device ceases to display information for projection on the lens and the processing prevents the identification of the location of the real-world feature from being performed on at least some of the video frames that were captured while the display device displays information for projection on the lens.

### Claim 20

A computer program product for a portable electronic device for a mixed reality (MR) headset having a lens through which a user wearing the MR headset can view real-world features, wherein a display device of the portable electronic device is arranged to display information that is projected on the lens for reflection directly or indirectly toward the user's eyes and a camera of the portable electronic device, the computer program product comprising:
a non-transitory computer readable medium storing program code executable by at least one processor of the portable electronic device to perform operations to:
process the video frames from the camera to identify a location of a real-world feature captured in the video frames, wherein the location of the real-world feature captured in the video frames from the camera is identified based on processing the video frames that were captured while the display device ceases to display information for projection on the lens and the processing prevents the identification of the location of the real-world feature from being performed on at least some of the video frames that were captured while the display device displays information for projection on the lens.
