# US 10,120,442 B2 · Eye tracking using a light field camera on a head-mounted display > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,120,442 B2 - Document ID: US10120442B2 - Status: Granted - Kind: B2 - Application number: 15/387,439 - Filed: 2016-12-21 - Published: 2018-06-21 - Granted: 2018-11-06 - Earliest claimed: 2016-12-21 - Assignees: Oculus VR, LLC - Applicants: Oculus VR, LLC - Inventors: Xinqiao Liu; Elias Daniel Guestrin - Web page: https://patentgrep.com/patent/US10120442B2 ## CPC - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G02B 27/0025: {for optical correction, e.g. distorsion, aberration} - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 27/0172: {characterised by optical features} - G06F 1/163: {Wearable computers, e.g. on a belt} - G06F 1/1686: {the I/O peripheral being an integrated camera} - G06F 1/325: {Power saving in peripheral device} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06T 7/557: from light fields, e.g. from plenoptic cameras - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G02B 2027/0134: {of stereoscopic type} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G06T 2207/10052: Images from lightfield camera - G06T 2207/30041: Eye; Retina; Ophthalmic - G06T 2207/30201: Face - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A head mounted display (HMD) includes one or more light field cameras for tracking one or both eyes of a user wearing the HMD. The HMD optionally includes light sources positioned inside the HMD and that illuminate one or both of the eyes of the user. The light field camera captures a plenoptic image of the user's eye. The plenoptic image includes light intensity data and direction data of the captured light rays. An eye tracking system updates a 3D light field model of the user's eye based on depth information from the plenoptic image frame. The eye tracking system identifies an iris plane of the user's eye using the 3D light field model. The eye tracking system determines a gaze direction of the user's eye by identifying the normal to the center of the iris plane. ## Claims ### Claim 1 A head mounted display (HMD) comprising: a display element configured to display content to a user wearing the HMD; an optics block configured to direct light from the display element to an exit pupil of the HMD; a light field camera configured to capture a plenoptic image of an eye of the user; and a controller configured to: update a light field model of the eye based on the plenoptic image of the eye, identify a region of a plane that includes an iris of the eye in the light field model by: determining the plane by approximating a curvature of a surface of the eye based on depth information from the plenoptic image; and identifying the region of the plane based on a size of the iris of the eye, and determine a gaze direction of the eye based on the identified region. ### Claim 2 (depends on 1) The HMD of claim 1, wherein the light field camera is positioned outside of a line of sight of the user. ### Claim 3 (depends on 1) The HMD of claim 1, wherein determine the gaze direction of the eye based on the identified region comprises determine a normal vector to the plane at the center of the region, the normal vector being relative to a position of the light field camera. ### Claim 4 (depends on 1) The HMD of claim 1, further comprising a light source configured to illuminate the eye. ### Claim 5 (depends on 4) The HMD of claim 4, wherein the light field camera comprises a notch filter configured to only pass light from the light source. ### Claim 6 (depends on 1) The HMD of claim 1, wherein the display element is configured to, in response to the gaze direction facing a first portion of the display element, display a first image at a first resolution at the first portion and display a second image at a second resolution lower than the first resolution at a second portion of the display element. ### Claim 7 (depends on 1) The HMD of claim 1, wherein the display element is configured to, in response to the gaze direction facing a first portion of the display element, display an image at the first portion and not display any image at a second portion of the display element outside a line of sight of the user. ### Claim 8 A head mounted display (HMD) comprising: a display element configured to display content to a user wearing the HMD; a light source configured to illuminate an eye, inside the HMD, of the user; a light field camera configured to capture a plenoptic image of the eye; and a controller configured to: update a light field model of the eye based on the plenoptic image of the eye; identify a region of a plane that includes an iris of the eye in the light field model by: determining the plane by approximating a curvature of a surface of the eye based on depth information from the plenoptic image; and identifying the region of the plane based on a size of the iris of the eye; and determine a gaze direction of the eye based on the identified region. ### Claim 9 (depends on 8) The HMD of claim 8, wherein the light source and the light field camera are each positioned outside of a line of sight of the user. ### Claim 10 (depends on 8) The HMD of claim 8, wherein determining the gaze direction of the eye based on the identified region comprises determining a normal vector to the plane at the center of the region, the normal vector being relative to a position of the light field camera. ### Claim 11 (depends on 8) The HMD of claim 8, wherein the light source emits infrared light. ### Claim 12 (depends on 11) The HMD of claim 11, wherein the light field camera comprises a notch filter configured to only pass infrared light. ### Claim 13 (depends on 8) The HMD of claim 8, wherein the display element is configured to display, in response to the gaze direction facing a first portion of the display element, display a first image at a first resolution at the first portion and display a second image at a second resolution lower than the first resolution at a second portion of the display element. ### Claim 14 (depends on 8) The HMD of claim 8, wherein the display element is configured to, in response to the gaze direction facing a first portion of the display element, display an image at the first portion and not display any image at a second portion of the display element outside a line of sight of the user. ### Claim 15 A head mounted display (HMD) comprising: a display element configured to display content to a user wearing the HMD; an optics block configured to direct light from the display element to an exit pupil of the HMD; a light field camera configured to capture a plenoptic image of an eye of the user; and a controller configured to: receive the plenoptic image of the eye from the light field camera; provide the plenoptic image of the eye to a virtual reality (VR) console; receive, from the VR console, a light field model of the eye generated based on the plenoptic image of the eye, the VR console configured to: determine a plane of the eye in the light field model by approximating a curvature of a surface of the eye based on depth information from the plenoptic image; and identify a region of the plane, including an iris of the eye in the light field model, based on a size of the iris of the eye; and determine a gaze direction of the eye based on the identified region of the light field model. ### Claim 16 (depends on 15) The HMD of claim 15, wherein the light field camera is positioned outside of a line of sight of the user. ### Claim 17 (depends on 15) The HMD of claim 15, further comprising a light source configured to illuminate the eye, and wherein the light field camera comprises a notch filter configured to only pass light from the light source. ### Claim 18 (depends on 15) The HMD of claim 15, wherein the display element is configured to, in response to the gaze direction facing a first portion of the display element, display a first image at a first resolution at the first portion and display a second image at a second resolution lower than the first resolution at a second portion of the display element. ### Claim 19 An eye tracking system comprising: a light field camera configured to capture a plenoptic image of an eye of a user; and a controller configured to: update a light field model of the eye based on the plenoptic image of the eye; determine a plane in the light field model by approximating a curvature of a surface of the eye based on depth information from the plenoptic image; identify a region of the plane that includes an iris of the eye based on a size of the iris of the eye; and determine a gaze direction of the eye based on the identified region. ### Claim 20 (depends on 19) The eye tracking system of claim 19, wherein determine the gaze direction of the eye based on the identified region comprises determine a normal vector to the plane at the center of the region, the normal vector being relative to a position of the light field camera. ### Claim 21 (depends on 19) The eye tracking system of claim 19, wherein the controller is further configured to: in response to determining that the gaze direction is facing a first portion of a display element, (i) providing a first image for display by the display element at a first resolution at the first portion and (ii) providing a second image for display by the display element at a second resolution lower than the first resolution at a second portion of the display element. ### Claim 22 (depends on 19) The eye tracking system of claim 19, wherein the controller is further configured to: in response to determining that the gaze direction is facing a first portion of a display element, (i) providing an image for display by the display element at the first portion and (ii) determining to not provide any image for display by the display element at a second portion of the display element outside a line of sight of the user. ### Claim 23 (depends on 19) The eye tracking system of claim 19, wherein the light field camera is part of a head mounted display (HMD) worn by the user, the controller providing images or video for display by a display element of the HIVID based at least in part on the gaze direction.