# US 10,311,584 B1 · Estimation of absolute depth from polarization measurements > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,584 B1 - Document ID: US10311584B1 - Status: Granted - Kind: B1 - Application number: 15/808,758 - Filed: 2017-11-09 - Granted: 2019-06-04 - Earliest claimed: 2017-11-09 - Assignees: Facebook Technologies, LLC - Applicants: Facebook Technologies, LLC - Inventors: Michael Hall; Ruonan Yang - Web page: https://patentgrep.com/patent/US10311584B1 ## CPC - G06T 7/251: {involving models} - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/514: from specularities - G06V 40/19: Sensors therefor - G06V 40/193: {Preprocessing; Feature extraction} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/30201: Face - G06V 2201/07: Target detection - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A head mounted display comprises an eye tracking system configured to enable eye tracking using polarization. The eye tracking system includes one or more illumination sources and an optical detector comprising polarization sensitive pixels. The one or more illumination sources are configured to illuminate a user's eye and generate reflections directed towards the optical detector. The eye tracking system determines, for each polarization sensitive pixel in a subset of the polarization sensitive pixels, one or more estimation parameters. The eye tracking system determines, for the subset of the polarization sensitive pixels, depth information for one or more glints associated with one or more surfaces of the eye, based in part on the polarization of the reflections and the one or more estimation parameters. The determined depth information is used to update a model of the eye. The eye tracking system determines eye tracking information based on the updated model. ## Claims ### Claim 1 A head-mounted display (HMD) comprising: an eye tracking system that includes: one or more illumination sources configured to illuminate an eye of a user with light, and an optical detector comprising an array of a plurality of polarization sensitive pixels configured to: capture one or more images of the eye, and determine intensity and polarization information associated with the one or more images; and a controller configured to: determine, for each polarization sensitive pixel in a subset of the plurality of polarization sensitive pixels, one or more estimation parameters associated with the eye tracking system, determine, for the subset of the plurality of polarization sensitive pixels, depth information for one or more glints associated with one or more surfaces of a plurality of surfaces of the eye, based in part on the determined polarization information and the one or more estimation parameters, update a model of the eye based in part on the determined depth information, and determine eye tracking information associated with the updated model of the eye, the eye tracking information including an eye orientation of the eye. ### Claim 2 (depends on 1) The HMD of claim 1, wherein the controller is further configured to: update the model of the eye by fitting one or more model components of a plurality of model components in the model to the one or more surfaces of the eye, using the depth information for the one or more glints. ### Claim 3 (depends on 2) The HMD of claim 2, wherein the controller is further configured to: determine the eye tracking information based in part on the fitted one or more model components. ### Claim 4 (depends on 1) The HMD of claim 1, wherein: each polarization sensitive pixel in the plurality of polarization sensitive pixels includes a three or more native pixels; and each native pixel of the three or more native pixels comprises a detector coupled to a polarization filter. ### Claim 5 (depends on 1) The HMD of claim 1, wherein the controller is further configured to: determine a plurality of surface normals associated with the subset of the plurality of polarization sensitive pixels based on the determined intensity and polarization information; and determine a three-dimensional (3D) shape of the eye based in part on the determined surface normals. ### Claim 6 (depends on 1) The HMD of claim 1, wherein: the optical detector is configured to capture, for at least three polarization angles, an intensity of light reflected from at least one surface of the eye; and the controller is further configured to determine a degree of polarization of the reflected light based in part on the captured intensity. ### Claim 7 (depends on 6) The HMD of claim 6, wherein the controller is further configured to: determine a zenith angle associated with the reflected light based in part on the determined degree of polarization; and determine depth information for a glint of the one or more glints, based on the one or more estimation parameters and the zenith angle. ### Claim 8 (depends on 6) The HMD of claim 6, wherein the one or more estimation parameters determined for each polarization sensitive pixel in the subset include: a displacement between one of the one or more illumination sources and that polarization sensitive pixel, and an angle between a ray of light reflected from a surface of the eye and an axis of that polarization sensitive pixel capturing the ray of reflected light. ### Claim 9 (depends on 8) The HMD of claim 8, wherein the controller is further configured to: determine a zenith angle associated with the reflected light based in part on the determined degree of polarization; and determine depth information for a glint of the one or more glints based on the displacement, the angle and the zenith angle. ### Claim 10 (depends on 1) The HMD of claim 1, wherein the controller is further configured to: retrieve one or more model parameters associated with a stored model of the eye; extract one or more model parameter values from the determined depth information; update at least one of the one or more model parameters by replacing one or more stored model parameter values with the one or more extracted model parameter values, based in part on a determination of a difference between at least one of the one or more stored model parameter values and the one or more extracted model parameter values; and store the updated model in a data store associated with the eye tracking module. ### Claim 11 A head-mounted display (HMD) comprising: a display unit including one or more illumination sources, the display unit configured to emit content and illuminate an eye of a user, an eye tracking system that includes: an optical detector comprising an array of a plurality of polarization sensitive pixels configured to: capture one or more images of the eye, and determine intensity and polarization information associated with the one or more images; and a controller configured to: determine, for each polarization sensitive pixel in a subset of the plurality of polarization sensitive pixels, one or more estimation parameters associated with the eye tracking system, determine, for the subset of the plurality of polarization sensitive pixels, depth information for one or more glints associated with one or more surfaces of a plurality of surfaces of the eye, based in part on the determined polarization information, one or more distances between the optical detector and the one or more illumination sources, and the one or more estimation parameters, update a model of the eye based in part on the determined depth information, and determine eye tracking information associated with the updated model of the eye, the eye tracking information including an eye orientation of the eye. ### Claim 12 (depends on 11) The HMD of claim 11, wherein the controller is further configured to: update the model of the eye by fitting one or more model components of a plurality of model components in the model to the one or more surfaces of the eye, using the depth information for the one or more glints. ### Claim 13 (depends on 12) The HMD of claim 12, wherein the controller is further configured to: determine the eye tracking information based in part on the fitted one or more model components. ### Claim 14 (depends on 11) The HMD of claim 11, wherein the controller is further configured to: determine a plurality of surface normals associated the subset of the plurality of polarization sensitive pixels based on the determined intensity and polarization information; and determine a three-dimensional (3D) shape of the eye based in part on the determined surface normals. ### Claim 15 (depends on 11) The HMD of claim 11, wherein: the optical detector is configured to capture, for at least three polarization angles, an intensity of light reflected from at least one surface of the eye; and the controller is further configured to determine a degree of polarization of the reflected light based in part on the captured intensity. ### Claim 16 (depends on 15) The HMD of claim 15, wherein the controller is further configured to: determine a zenith angle associated with the reflected light based in part on the determined degree of polarization; and determine depth information for a glint of the one or more glints based on the one or more estimation parameters and the zenith angle. ### Claim 17 (depends on 15) The HMD of claim 15, wherein the one or more estimation parameters determined for each polarization sensitive pixel in the subset include: a displacement between one of the one or more illumination sources and that polarization sensitive pixel, and an angle between a ray of light reflected from a surface of the eye and an axis of that polarization sensitive pixel capturing the ray of reflected light. ### Claim 18 (depends on 17) The HMD of claim 17, wherein the controller is further configured to: determine a zenith angle associated with the reflected light based in part on the determined degree of polarization; and determine depth information for a glint of the one or more glints based on the displacement, the angle and the zenith angle. ### Claim 19 A head-mounted display (HMD) comprising: an eye tracking system that includes: one or more illumination sources configured to illuminate an eye of a user with unpolarized light, and an optical detector comprising an array of a plurality of polarization sensitive pixels coupled to a dynamic polarizer, each of the polarization sensitive pixels configured to: capture one or more images of the eye, and determine intensity and polarization information associated with the one or more images; and a controller configured to: determine, for each polarization sensitive pixel in a subset of the plurality of polarization sensitive pixels, one or more estimation parameters associated with the eye tracking system, determine, for the subset of the plurality of polarization sensitive pixels, depth information for one or more glints associated with one or more surfaces of a plurality of surfaces of the eye, based in part on the determined polarization information, one or more distances between the optical detector and the one or more illumination sources, and the one or more estimation parameters, update a model of the eye based in part on the determined depth information, and determine eye tracking information associated with the updated model of the eye, the eye tracking information including an eye orientation of the eye. ### Claim 20 (depends on 19) The HMD of claim 19, wherein the dynamic polarizer is an optical filter that is permeable to light waves of three or more polarization states.