# US 12,008,723 B2 · Depth plane selection for multi-depth plane display systems by user categorization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,723 B2
- Document ID: US12008723B2
- Status: Granted
- Kind: B2
- Application number: 17/962,289
- Filed: 2022-10-07
- Published: 2023-02-02
- Granted: 2024-06-11
- Earliest claimed: 2018-08-03
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Samuel A. Miller; Lomesh Agarwal; Lionel Ernest Edwin; Ivan Li Chuen Yeoh; Daniel Farmer; Sergey Fyodorovich Prokushkin; Yonatan Munk; Edwin Joseph Selker; Erik Fonseka; Paul M. Greco; Jeffrey Scott Sommers; Bradley Vincent Stuart; Shiuli Das; Suraj Manjunath Shanbhag
- Web page: https://patentgrep.com/patent/US12008723B2

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G02B 6/0076: {Stacked arrangements of multiple light guides of the same or different cross-sectional area}
- G02B 27/0172: {characterised by optical features}
- G02B 27/0179: {Display position adjusting means not related to the information to be displayed}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- H04N 13/344: with head-mounted left-right displays
- H04N 13/383: for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
- H04N 13/398: Synchronisation thereof; Control thereof
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0181: {Adaptation to the pilot/driver}
- G06T 2200/24: involving graphical user interfaces [GUIs]
- H04N 2213/001: Constructional or mechanical details
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A display system includes a head-mounted display configured to project light, having different amounts of wavefront divergence, to an eye of a user to display virtual image content appearing to be disposed at different depth planes. The wavefront divergence may be changed in discrete steps, with the change in steps being triggered based upon whether the user is fixating on a particular depth plane. The display system may be calibrated for switching depth planes for a main user. Upon determining that a guest user is utilizing the system, rather than undergoing a full calibration, the display system may be configured to switch depth planes based on a rough determination of the virtual content that the user is looking at. The virtual content has an associated depth plane and the display system may be configured to switch to the depth plane of that virtual content.

## Claims

### Claim 1

An augmented reality display system configured to present virtual image content on a plurality of depth planes, the augmented reality display system comprising:
a waveguide configured to present the virtual image content by outputting light to a wearer, the waveguide further configured to pass light from the world into an eye of the wearer;
an imaging device configured to capture images of eyes of the wearer; and
at least one processor configured to:
determine whether the wearer is a calibrated user or a guest user based at least in part on images of the eyes of the wearer from the imaging device;
based on determining that the wearer is a calibrated user:
load pre-existing user depth plane switching calibration information;
identify a fixation point of the user based upon the pre-existing user depth plane switching calibration information; and
switch the virtual image content to be presented at a depth plane that corresponds to the fixation point; and
based on determining that the wearer is a guest user:
determine the interpupillary distance of the guest user;
calculate an estimated fixation point of the eyes of the guest user based upon the determined interpupillary distance, wherein the estimated fixation point is a point in three-dimensional space on which the eyes of the guest user are focused;
determine a plurality of system-defined volumes such that a field of view of the guest user is divided into the plurality of system-defined volumes, wherein each system-defined volume spans a different three-dimensional portion of the field of view, and wherein at least one of a size or a shape of the volumes is determined based on weighted factors that include an application most recently utilized by the guest user;
determine that the estimated fixation point is within a particular one of the plurality of system-defined volumes that includes multiple depth planes of the plurality of depth planes;
responsive to determining that the system-defined volume includes at least two virtual objects of the virtual image content to be presented, switch the virtual image content, including the at least two virtual objects, to be presented at a depth plane that corresponds to the estimated fixation point, and
responsive to determining that the system-defined volume includes a single virtual object of the virtual image content to be presented, switch the virtual image content, including the single virtual object, to be presented at a depth plane that is specified by information included in the virtual image content.

### Claim 2 (depends on 1)

The augmented reality display system of claim 1, wherein the imaging device comprises a left eye tracking system and a right eye tracking system together configured to measure the wearer's interpupillary distance.

### Claim 3 (depends on 1)

The augmented reality display system of claim 1, wherein the processor is configured to determine whether the wearer is a calibrated user or a guest user based on the wearer's interpupillary distance.

### Claim 4 (depends on 3)

The augmented reality display system of claim 3, wherein the processor is configured to determine whether the wearer is a calibrated user or a guest user based on whether the wearer's interpupillary distance is within a predetermined threshold of the interpupillary distance of the calibrated user.

### Claim 5 (depends on 1)

The augmented reality display system of claim 1, wherein the processor is configured to determine whether the wearer is a calibrated user or a guest user based on whether the wearer's interpupillary distance is within 1.0 mm of the interpupillary distance of the calibrated user.

### Claim 6 (depends on 1)

The augmented reality display system of claim 1, wherein the waveguide is one of a plurality of waveguides forming a stack of waveguides.

### Claim 7 (depends on 6)

The augmented reality display system of claim 6, wherein one or more waveguides of the stack of waveguides are configured to output light to the eye of the wearer with a different amount of wavefront divergence than one or more other waveguides of the stack of waveguides, wherein the outputted light with different amounts of wavefront divergence forms virtual objects at different perceived depths away from the wearer.

### Claim 8

A method for depth plane switching in a display system configured to direct image light to eyes of a user to display virtual image content, the eyes separated by an interpupillary distance and the display system configured to present the virtual image content on a plurality of depth planes, the method comprising:
determining whether the user is a calibrated user or a guest user;
based on determining that the user is a calibrated user:
loading pre-existing user depth plane switching calibration information;
identifying a fixation point of the user based upon the pre-existing user depth plane switching calibration information; and
switching the virtual image content to be presented at a depth plane that corresponds to the fixation point; and
based on determining that the wearer is a guest user:
determining the interpupillary distance of the guest user;
calculating an estimated fixation point of the eyes of the guest user based upon the determined interpupillary distance, wherein the estimated fixation point is a point in three-dimensional (3D) space on which the eyes of the guest user are focused;
determining a plurality of system-defined volumes such that a field of view of the guest user is divided into the plurality of system-defined volumes, wherein each system-defined volume spans a different three-dimensional portion of the field of view, and wherein at least one of a size or a shape of the volumes is determined based on weighted factors that include an application most recently utilized by the guest user;
determining that the estimated fixation point is within a particular one of the plurality of system-defined volumes that includes multiple depth planes of the plurality of depth planes;
responsive to determining that the system-defined volume includes at least two virtual objects of the virtual image content to be presented, switching the virtual image content, including the at least two virtual objects, to be presented at a depth plane that corresponds to the estimated fixation point, and
responsive to determining that the system-defined volume includes a single virtual object of the virtual image content to be presented, switching the virtual image content, including the single virtual object, to be presented at a depth plane that is specified by information included in the virtual image content.

### Claim 9 (depends on 8)

The method of claim 8, wherein determining the interpupillary distance of the guest user comprises determining an interpupillary distance of eyes of the guest user focused at optical infinity.

### Claim 10 (depends on 8)

The method of claim 8, wherein determining whether the user is the calibrated user or is a guest user comprises determining the interpupillary distance of the user with one or more eye tracking cameras configured to image the eyes of the user.

### Claim 11 (depends on 8)

The method of claim 8, wherein determining whether the user is the calibrated user comprises determining the interpupillary distance of the user with one or more eye tracking cameras configured to image the eyes of the user and determining that the interpupillary distance of the user is within a predetermined range.

### Claim 12 (depends on 11)

The method of claim 11, wherein determining whether the user is the guest user comprises determining the interpupillary distance of the user with the one or more eye tracking cameras and determining that the interpupillary distance of the user is outside of the predetermined range.

### Claim 13 (depends on 8)

The method of claim 8, wherein the pre-existing user depth plane switching calibration information comprises a measured interpupillary distance of the calibrated user, wherein determining whether the user is the calibrated user or is a guest user comprises determining the interpupillary distance of the user, determining that the user is the calibrated user when the user's interpupillary distance is within 1.0 mm of the measured interpupillary distance of the calibrated user, and determining that the user is a guest user when the user's interpupillary distance is not within 1.0 mm of the measured interpupillary distance of the calibrated user.

### Claim 14 (depends on 8)

The method of claim 8, wherein determining whether the user is the calibrated user or is a guest user comprises identifying the user with at least one eye tracking camera.

### Claim 15 (depends on 8)

The method of claim 8, further comprising:
with an eye tracking system, determining an optical axis for each of the user's eyes; and
determining a vergence depth of the user based at least on the determined optical axes for the user's eyes.

### Claim 16 (depends on 8)

The method of claim 8, wherein each depth plane is associated with image light having a different amount of wavefront divergence, wherein the display system is configured to switch the virtual image content between different depth planes by changing a wavefront divergence of the image light.

### Claim 17 (depends on 16)

The method of claim 16, wherein the display system comprises a stack of waveguides, wherein each waveguide is configured to output the image light.

### Claim 18 (depends on 17)

The method of claim 17, wherein one or more waveguides of the stack of waveguides are configured to output image light to the eye of the wearer with a different amount of wavefront divergence than one or more other waveguides of the stack of waveguides, wherein the outputted image light with different amounts of wavefront divergence forms virtual objects at different perceived depths away from the wearer.

### Claim 19 (depends on 18)

The method of claim 18, wherein changing the wavefront divergence of the image light comprises switching the waveguide outputting the image light.

### Claim 20 (depends on 8)

The method of claim 8, further comprising:
based on determining that the wearer is the guest user, determining a distance between estimated 3D positions of centers of rotation of the eyes of the guest user;
wherein calculating the estimated fixation point includes comparing the interpupillary distance to the distance between the estimated 3D positions of the centers of rotation of the eyes of the guest user.