# US 11,112,863 B2 · Eye center of rotation determination, depth plane selection, and render camera positioning in display systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,112,863 B2
- Document ID: US11112863B2
- Status: Granted
- Kind: B2
- Application number: 16/250,931
- Filed: 2019-01-17
- Published: 2019-08-08
- Granted: 2021-09-07
- Earliest claimed: 2018-01-17
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Samuel A. Miller; Lomesh Agarwal; Lionel Ernest Edwin; Ivan Li Chuen Yeoh; Daniel Farmer; Sergey Fyodorovich Prokushkin; Yonatan Munk; Edwin Joseph Selker; Bradley Vincent Stuart; Jeffrey Scott Sommers
- Web page: https://patentgrep.com/patent/US11112863B2

## CPC

- A61B 3/113: for determining or recording eye movement
- A61B 3/111: {for measuring interpupillary distance}
- G02B 27/0081: {with means for altering, e.g. enlarging, the entrance or exit pupil}
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/017: {Head mounted}
- G02B 27/0172: {characterised by optical features}
- G02B 27/0176: {characterised by mechanical features}
- G02B 30/40: giving the observer of a single two-dimensional [2D] image a perception of depth
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06V 40/193: {Preprocessing; Feature extraction}
- H04N 13/344: with head-mounted left-right displays
- H04N 13/383: for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
- G02B 2027/0134: {of stereoscopic type}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/0174: {holographic}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A display system can include a head-mounted display configured to project light to an eye of a user to display virtual image content at different amounts of divergence and collimation. The display system can include an inward-facing imaging system images the user's eye and processing electronics that are in communication with the inward-facing imaging system and that are configured to obtain an estimate of a center of rotation of the user's eye. The display system may render virtual image content with a render camera positioned at the determined position of the center of rotation of said eye.

## Claims

### Claim 1

A display system configured to project light to an eye of a user to display virtual image content in a vision field of said user, said eye having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through said lens, pupil, and cornea, said display system comprising:
a frame configured to be supported on a head of the user;
a head-mounted display disposed on the frame, said display configured to project light into said user's eye to display virtual image content to the user's vision field such that the displayed virtual image content appears to originate from different depths;
one or more eye tracking cameras configured to image the user's eye; and
processing electronics in communication with the display and the one or more eye tracking cameras, the processing electronics configured to obtain an estimate of a center of rotation of said eye based on images of said eye obtained with said one or more eye tracking cameras, wherein to obtain the estimate of the center of rotation of said eye, the processing electronics are configured to:
determine an array of positions based on a plurality of spatial locations on an image of the user's eye obtained with said one or more eye tracking cameras, wherein said array of positions corresponds to at least a portion of an ellipse, and
determine a plurality of linear paths extending from a location on a first side of said array of positions through said array of positions to a second opposite side of said array of positions.

### Claim 2 (depends on 1)

The display system of claim 1, wherein said cornea has associated therewith a cornea sphere having a center of curvature and said processing electronics is configured to determine a location of said center of curvature of said cornea sphere.

### Claim 3 (depends on 1)

The display system claim 1, wherein said processing electronics is configured to determine a location and orientation of said optical axis.

### Claim 4 (depends on 3)

The display system of claim 3, wherein said processing electronics is configured to determine said location and orientation of said optical axis based on a location of a center of said pupil in three-dimensional space with respect to a center of curvature of said cornea.

### Claim 5 (depends on 1)

The display system of claim 1, wherein said processing electronics is configured to determine a location and orientation of a visual axis, offset from said optical axis, based on a location and orientation of said optical axis.

### Claim 6 (depends on 1)

The display system of claim 1, wherein the processing electronics is configured to obtain the estimate of the center of rotation by filtering a plurality of estimated center of rotation positions, averaging the plurality of estimated center of rotation positions, applying a Kalman filter to the plurality of estimated center of rotation positions, or any combinations thereof.

### Claim 7 (depends on 1)

The display system of claim 1, wherein said processing electronics is configured to present said virtual image content to said user's eye that is rendered as if captured by a camera having an aperture at the determined position of the center of rotation of said user's eye.

### Claim 8 (depends on 1)

The display system of claim 1, wherein said processing electronics is configured to use a render camera configured to render virtual images to be presented to said eye that are rendered as if captured by a camera having an aperture closer to said center of rotation than said retina of said eye.

### Claim 9 (depends on 1)

The display system of claim 1, wherein said processing electronics are configured to obtain the estimate of the center of rotation of said eye based on determination of multiple gaze directions of the user's eye over a period of time during which said eye is rotating based on images of said eye obtained by said one or more eye tracking cameras.

### Claim 10 (depends on 1)

The display system of claim 1, wherein said plurality of spatial locations on said image comprises spatial locations on a boundary between the cornea and the sclera of said user's eye in said image obtained with said one or more eye tracking cameras.

### Claim 11 (depends on 1)

The display system of claim 1, wherein said processing electronics are configured to determine a circular region based on said plurality of linear paths, said circular region having a radius, R.

### Claim 12 (depends on 11)

The display system of claim 11, wherein said radius, R, corresponds to an average radius of a limbus.

### Claim 13 (depends on 11)

The display system of claim 11, wherein said radius, R, corresponds to a measured radius of the limbus of the user's eye.

### Claim 14 (depends on 11)

The display system of claim 11, wherein said processing electronics are configured to determine respective locations and directions of a plurality of normals through central portions of respective circular regions based on a plurality of images of said eye previously obtained with said one or more eye tracking cameras.

### Claim 15 (depends on 14)

The display system of claim 14, wherein said processing electronics are configured to obtain the estimate of the center of rotation of said user's eye based on the locations and directions of multiple of said plurality of normals determined based on images of the user's eye obtained over a period of time during which said eye is rotating.

### Claim 16 (depends on 1)

The display system of claim 1, wherein said display is configured to project light into said user's eye to display virtual image content to the user's vision field at different amounts of divergence such that the displayed virtual image content appears to originate from different depths.

### Claim 17 (depends on 1)

The display system of claim 1, wherein said display is configured to project some light into said user's eye to display virtual image content to the user's vision field that is diverging such that the displayed virtual image content appears to originate from a first depth and project some light into said user's eye to display virtual image content to the user's vision field that is collimated such that the displayed virtual image content appears to originate from a second depth different than the first depth.

### Claim 18 (depends on 1)

The display system of claim 1, wherein said display system is configured to project light into said user's eye such that the displayed virtual image content appears to originate from different depths at different periods of time.

### Claim 19 (depends on 14)

The display system of claim 14, wherein said processing electronics are configured to obtain the estimate of the center of rotation of said user's eye based on an intersection of said plurality of normals.

### Claim 20 (depends on 14)

The display system of claim 14, wherein said processing electronics are configured to obtain the estimate of the center of rotation of said user's eye based on a region at which the plurality of normals converge.

### Claim 21

A method implemented by a display system configured to project light to left and right eyes of a user to display virtual image content in a vision field of said user, each of said eyes having a cornea, an iris, a pupil, a lens, a retina, and an optical axis extending through said lens, pupil, and cornea, said method comprising:
displaying, via the display system, virtual image content to the user's vision field such that the displayed virtual image content appears to originate from different depths at different periods of time; and
with one or more eye tracking cameras configured to image said eyes of the user to track movements of said eyes, determining a position of a center of rotation of said left eye and a position of a center of rotation of said right eye based on images of said eyes, wherein determining the position of the center of rotation of one of said left eye or said right eye comprises:
determining an array of positions based on a plurality of spatial locations on an image of the left eye or right eye obtained with said one or more eye tracking cameras, wherein said array of positions corresponds to at least a portion of an ellipse, and
determining a plurality of linear paths extending from a location on a first side of said array of positions through said array of positions to a second opposite side of said array of positions.

### Claim 22 (depends on 21)

The method of claim 21, wherein displaying, via the display system, virtual image content to the user's vision field comprises projecting light into said user's eye to display virtual image content to the user's vision field at different amounts of divergence such that the displayed virtual image content appears to originate from different depths.

### Claim 23 (depends on 21)

The method of claim 21, wherein displaying, via the display system, virtual image content to the user's vision field comprises projecting some light into said user's eyes to display virtual image content to the user's vision field that is diverging such that the displayed virtual image content appears to originate from a first depth and projecting some light into said user's eyes to display virtual image content to the user's vision field that is collimated such that the displayed virtual image content appears to originate from a second depth different than the first depth.

### Claim 24 (depends on 21)

The method of claim 21, wherein said cornea has associated therewith a cornea sphere having a center of curvature, and wherein the method further comprises determining a location of said center of curvature of said cornea sphere.

### Claim 25 (depends on 21)

The method of claim 21, wherein the method further comprises determining a location and orientation of said optical axis.

### Claim 26 (depends on 25)

The method of claim 25, wherein determining said location and orientation of said optical axis is based on a location of a center of said pupil in three-dimensional space with respect to a center of curvature of said cornea.

### Claim 27 (depends on 21)

The method of claim 21, wherein the method further comprises determining a location and orientation of a visual axis, offset from said optical axis, based on a location and orientation of said optical axis.

### Claim 28 (depends on 21)

The method of claim 21, wherein the method further comprises presenting said virtual image content to said user's left eye or right eye that is rendered as if captured by a camera having an aperture at the determined position of the center of rotation of said user's left eye or right eye.

### Claim 29 (depends on 21)

The method of claim 21, wherein the method further comprises obtaining the center of rotation by filtering a plurality of estimated center of rotation positions, averaging the plurality of estimated center of rotation positions, applying a Kalman filter to the plurality of estimated center of rotation positions, or any combinations thereof.

### Claim 30 (depends on 21)

The method of claim 21, wherein the method further comprises using a render camera configured to render virtual images to be presented to said left eye or right eye that are rendered as if captured by a camera having an aperture closer to said center of rotation than said retina of said left eye or right eye.

### Claim 31 (depends on 21)

The method of claim 21, wherein the method further comprises obtaining the estimate of the center of rotation of said left eye or right eye based on determination of multiple gaze directions of the user's left eye or right eye over a period of time during which said left eye or said right eye is rotating based on images of said left eye or said right eye obtained by said one or more eye tracking cameras.

### Claim 32 (depends on 21)

The method of claim 21, wherein said plurality of spatial locations on said image comprises spatial locations on a boundary between the cornea and the sclera of said left eye or said right eye in said image obtained with said one or more eye tracking cameras.

### Claim 33 (depends on 21)

The method of claim 21, wherein the method further comprises determining a circular region based on said plurality of linear paths, said circular region having a radius, R.

### Claim 34 (depends on 33)

The method of claim 33, wherein said radius, R, corresponds to an average radius of a limbus.

### Claim 35 (depends on 33)

The method of claim 33, wherein said radius, R, corresponds to a measured radius of the limbus of the left eye or right eye.

### Claim 36 (depends on 33)

The method of claim 33, wherein the method further comprises determining respective locations and directions of a plurality of normals through central portions of respective circular regions based on a plurality of images of said left eye or said right eye previously obtained with said one or more eye tracking cameras.

### Claim 37 (depends on 36)

The method of claim 36, wherein the method further comprises obtaining the estimate of the center of rotation of said left eye or right eye based on the locations and directions of multiple of said plurality of normals determined based on images of the said left eye or said right eye obtained over a period of time during which said left eye or said right eye is rotating.

### Claim 38 (depends on 36)

The method of claim 36, wherein the estimate of the center of rotation of said left eye or right eye is obtained based on an intersection of said plurality of normals.

### Claim 39 (depends on 36)

The method of claim 36, wherein the estimate of the center of rotation of said left eye or right eye is obtained based on a region at which the plurality of normals converge.