# US 12,182,945 B2 · Multi-depth plane display system with reduced switching between depth planes > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,945 B2 - Document ID: US12182945B2 - Status: Granted - Kind: B2 - Application number: 17/493,163 - Filed: 2021-10-04 - Published: 2022-01-27 - Granted: 2024-12-31 - Earliest claimed: 2016-02-11 - Assignees: Magic Leap, Inc. - Applicants: Magic Leap, Inc. - Inventors: Nicole Elizabeth Samec; Nastasja U. Robaina; Christopher M. Harrises; Mark Baerenrodt - Web page: https://patentgrep.com/patent/US12182945B2 ## CPC - A61B 3/113: for determining or recording eye movement - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G02B 6/0013: {Means for improving the coupling-in of light from the light source into the light guide (coupling light into light guides in general G02B6/42)} - G02B 6/0035: {provided on the surface of the light guide or in the bulk of it} - G02B 6/005: {provided by one optical element, or plurality thereof, placed on the light output side of the light guide} - G02B 6/0076: {Stacked arrangements of multiple light guides of the same or different cross-sectional area} - G02B 27/017: {Head mounted} - G02B 27/0172: {characterised by optical features} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence) - H04N 13/128: Adjusting depth or disparity - H04N 13/332: Displays for viewing with the aid of special glasses or head-mounted displays [HMD] - H04N 13/344: with head-mounted left-right displays - H04N 13/383: for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes - H04N 13/395: with depth sampling, i.e. the volume being constructed from a stack or sequence of two-dimensional [2D] image planes - G02B 2027/0112: {comprising device for genereting colour display} - G02B 2027/0127: {comprising devices increasing the depth of field} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G02B 2027/0181: {Adaptation to the pilot/driver} - G02B 2027/0185: {Displaying image at variable distance} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Methods and systems for reductions in switching between depth planes of a multi-depth plane display system are disclosed. The display system may be an augmented reality display system configured to provide virtual content on a plurality of depth planes using different wavefront divergence. The system may monitor the fixation points based upon the gaze of each of the user's eyes, with each fixation point being a three-dimensional location in the user's field of view. Location information of virtual objects to be presented to the user are obtained, with each virtual object being associated with a depth plane. In some embodiments, the depth plane on which the virtual object is to be presented is modified based upon the fixation point of the user's eyes. For example, where the user is switching their fixation between virtual objects on two different depth planes, the display system may be configured to modify the presentation of one of the objects such that both objects are placed on the same depth plane. ## Claims ### Claim 1 A system comprising: a display device comprising a stack of waveguides, the stack of waveguides forming a display area and providing a view of an ambient environment through the display area, wherein a first waveguide of the stack of waveguides comprises diffractive optical elements configured to output light with a first wavefront divergence corresponding to a first accommodation cue and wherein a second waveguide of the stack of waveguides comprises diffractive optical elements configured to output light with a second wavefront divergence corresponding to a second accommodation cue; one or more processors configured to provide access to information from a map of the ambient environment, the map comprising respective three-dimensional locations for a first virtual object and a second virtual object, wherein a location for the first virtual object is associated with the first wavefront divergence and has associated binocular depth cues, and wherein the second virtual object is associated with the second wavefront divergence and second binocular depth cues; and one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: determining an indication of a fixation point of a user's eyes, wherein the fixation point corresponds to a location of the first virtual object; obtaining, via the map, location information associated with the second virtual object, wherein the second virtual object is configured to be presented, via the display device, with the second wavefront divergence from the second waveguide and with the second binocular depth cues; subsequently changing the wavefront divergence associated with the second virtual object to correspond with the first wavefront divergence of the first virtual object, wherein the second binocular depth cues of the second virtual object are retained, wherein changing the wavefront divergence associated with the second virtual object causes an increase in an accommodation vergence mismatch of the second virtual object as long as the accommodation vergence mismatch of the second virtual object is below a threshold value, and wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue based on an indication of a user input for other virtual objects to be adjusted to be at a same depth as the second virtual object, wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue based on an indication of a user input for other virtual objects to be adjusted to be at a same depth as the second virtual object, the user input being independent of the determination of the fixation point of the user's eyes with respect to the first virtual object relative to the second virtual object, and wherein the user input includes at least one of a hand gesture, a voice input, an interaction with a totem, or a predetermined user preference; and causing presentation to the user, via the first waveguide of the display device, of the second virtual object with the adjusted second wavefront divergence, wherein, upon determining that the accommodation vergence mismatch exceeds the threshold value, the system is configured to change the second binocular depth cues to reduce the accommodation vergence mismatch to below the threshold value. ### Claim 2 (depends on 1) The system of claim 1, wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue further based on a determination that the first virtual object is being fixated on less than the second virtual object, and wherein determining the indication of the fixation point comprises determining that the user's eyes are fixating on the location for greater than a threshold duration. ### Claim 3 (depends on 1) The system of claim 1, wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue further based on a determination that the first virtual object is being fixated on less than the second virtual object, and wherein subsequently changing the wavefront divergence further comprises determining that the user is alternating fixation, at greater than a threshold frequency, between the first virtual object and the second virtual object. ### Claim 4 (depends on 1) The system of claim 1, wherein the accommodation-vergence mismatch threshold is 0.33 diopter. ### Claim 5 (depends on 1) The system of claim 1, wherein the display device is a vari-focal display device, and wherein at least some waveguides of the plurality of waveguides are configured to output light of different component colors than other waveguides. ### Claim 6 (depends on 1) The system of claim 1, wherein the depth cues further comprise size cues. ### Claim 7 A display system comprising: a display device configured to present virtual objects to a user on a plurality of depth planes, wherein each depth plane has a different associated accommodation cue; one or more processors; and one or more computer storage media storing instructions that when executed by the display system, cause the display system to perform operations comprising: accessing a map of the world that comprises respective locations for a first virtual object and a second virtual object, the first virtual object being presented on a first depth plane; determining an indication of a fixation point of the eyes of the user, the fixation point corresponding to the location of the first virtual object, wherein the determining the indication of the fixation point includes a user input indication of a user input for other virtual objects to be adjusted to be at a same depth as the second virtual object; the user input being independent of the determination of the fixation point of the user's eyes with respect to the first virtual object relative to the second virtual object, and wherein the user input includes at least one of a hand gesture, a voice input, an interaction with a totem, or a predetermined user preference; and based on the determined indication of the fixation point, changing display properties of the second virtual object such that the second virtual object is displayed on the first depth plane, wherein the second virtual object was configured to be presented on a second depth plane of the plurality of depth planes based on the map, wherein display of the second virtual object is changed to the first depth plane, wherein displaying the second virtual object on the first depth plane causes an increase in an accommodation vergence mismatch of the second virtual object to a threshold value, and wherein the system is configured to change the second binocular depth cues based on a determination that the accommodation vergence mismatch exceeds the threshold value. ### Claim 8 (depends on 7) The display system of claim 7, wherein the determining the indication of the fixation point further includes a determination that the first virtual object is being fixated on less than the second virtual object, and wherein determining the indication of the fixation point comprises determining that the fixation point falls on the depth plane for greater than a threshold metric before changing display properties of the second virtual object. ### Claim 9 (depends on 7) The display system of claim 7, wherein the determining the indication of the fixation point further includes a determination that the first virtual object is being fixated on less than the second virtual object, and wherein changing display properties of the second virtual object comprises determining that the second virtual object is being fixated upon less than the first virtual object. ### Claim 10 (depends on 7) The display system of claim 7, wherein the accommodation-vergence mismatch threshold is 0.33 diopter. ### Claim 11 (depends on 7) The display system of claim 7, wherein the display device is a vari-focal display device. ### Claim 12 A method comprising: by a system comprising one or more processors and a display device comprising a stack of waveguides forming a display area and providing a view of an ambient environment through the display area, the display device configured to present virtual objects to a user on a plurality of depth planes, wherein each depth plane has a different associated accommodation cue: accessing a map of the world that comprises respective positions for a first virtual object and a second virtual object, the first virtual object being presented via a first waveguide of the stack of waveguides, the first waveguide configured to output light with a first level of wavefront divergence, wherein the first virtual object has associated first binocular depth cues and the second virtual object has an associated second level of wavefront divergence and second binocular depth cues which are different from the first level of wavefront divergence and the first binocular depth cues; determining an indication of a fixation point of a user's eyes, wherein the fixation point corresponds to a location of the first virtual object; obtaining, via the map, location information associated with the second virtual object, wherein the second virtual object is configured to be presented, via the display device, with the second level of wavefront divergence and second binocular depth cues, and wherein a second waveguide of the stack of waveguides is configured to present virtual objects with the second accommodation cues; subsequently changing the wavefront divergence associated with the second virtual object to correspond with the first level of wavefront divergence, wherein the second binocular cues of the second virtual object are retained, wherein changing the wavefront divergence associated with the second virtual object causes an increase in an accommodation vergence mismatch of the second virtual object to a threshold value, wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue based on an indication of a user input for other virtual objects to be adjusted to be at a same depth as the second virtual object, wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue based on an indication of a user input for other virtual objects to be adjusted to be at a same depth as the second virtual object, the user input being independent of the determination of the fixation point of the user's eyes with respect to the first virtual object relative to the second virtual object, and wherein the user input includes at least one of a hand gesture, a voice input, an interaction with a totem, or a predetermined user preference, wherein, upon a determination that the accommodation vergence mismatch exceeds the threshold value, changing the second binocular depth cues such that the second virtual object is perceived by the user to be in a common plane as the first virtual object; and causing presentation to the user, via the first waveguide of the display device, of the second virtual object with the first level of wavefront divergence and the second binocular depth cues. ### Claim 13 (depends on 12) The method of claim 12, wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue further based on a determination that the first virtual object is being fixated on less than the second virtual object, and wherein determining the indication of the fixation point comprises determining that the user's eyes are fixating on the location for greater than a threshold duration. ### Claim 14 (depends on 12) The method of claim 12, wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue further based on a determination that the first virtual object is being fixated on less than the second virtual object, and wherein subsequently changing the wavefront divergence comprises: determining that the user is alternating fixation, at greater than a threshold frequency, between the first virtual object and the second virtual. ### Claim 15 (depends on 12) The method of claim 12, wherein the display device is a vari-focal display device, and wherein at least some waveguides of the stack of waveguides are configured to output light with different component colors than other waveguide. ### Claim 16 (depends on 1) The system of claim 1, wherein the subsequently changing the wavefront divergence includes adjusting the first accommodation cue further based on a determination that the first virtual object is being fixated on less than the second virtual object, and wherein the subsequently changing the wavefront divergence includes the determination that the first virtual object is being fixated on less than the second virtual object. ### Claim 17 (depends on 1) The system of claim 1, wherein the changing the second binocular depth cues upon determining that the accommodation vergence mismatch exceeds the threshold value includes changing the second binocular depth cues such that the second virtual object is perceived by the user to be in a common plane as the first virtual object. ### Claim 18 (depends on 1) The system of claim 1, wherein the changing the second binocular depth cues upon determining that the accommodation vergence mismatch exceeds the threshold value includes changing the second binocular depth cues such that the second virtual object is perceived by the user to be closer to the user but in a different plane than the first virtual object.