# US 2013/0154918 A1 · Enhanced user eye gaze estimation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2013/0154918 A1 - Document ID: US20130154918A1 - Status: Published application - Kind: A1 - Application number: 13/331,285 - Filed: 2011-12-20 - Published: 2013-06-20 - Earliest claimed: 2011-12-20 - Assignees: Microsoft Technology Licensing, LLC - Inventors: BENJAMIN ISAAC VAUGHT; ROBERT L. CROCCO, JR.; JOHN LEWIS; JIAN SUN; YICHEN WEI - Web page: https://patentgrep.com/patent/US20130154918A1 ## CPC - G06V 40/197: {Matching; Classification} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06V 20/20: in augmented reality scenes - G02B 27/017: {Head mounted} - G06V 40/193: {Preprocessing; Feature extraction} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract Systems, methods, and computer media for estimating user eye gaze are provided. A plurality of images of a user's eye are acquired. At least one image of at least part of the user's field of view is acquired. At least one gaze target area in the user's field of view is determined based on the plurality of images of the user's eye. An enhanced user eye gaze is then estimated by narrowing a database of eye information and corresponding known gaze lines to a subset of the eye information having gaze lines corresponding to a gaze target area. User eye information derived from the images of the user's eye is then compared with the narrowed subset of the eye information, and an enhanced estimated user eye gaze is identified as the known gaze line of a matching eye image. ## Claims ### Claim 1 One or more computer-readable media storing computer-executable instructions for performing a method for estimating user eye gaze, the method comprising: acquiring a plurality of images of a user's eye; acquiring at least one image of at least part of the user's field of view; determining at least one gaze target area in the at least one image of at least part of the user's field of view based on at least one of the plurality of images of the user's eye; and estimating an enhanced user eye gaze by: narrowing a database of eye information and corresponding known gaze lines to a subset of the eye information having gaze lines corresponding to the at least one determined gaze target area, comparing user eye information derived from the plurality of images of the user's eye to the narrowed subset of the eye information, and identifying the known gaze line of matching eye information in the subset of the eye information as an enhanced estimated user eye gaze. ### Claim 2 (depends on 1) The media of claim 1, wherein the at least one gaze target area is determined by: determining a first estimated gaze based on at least one of the plurality of images of the user's eye; identifying at least one virtual or real visual target in the at least one image of at least part of the user's field of view corresponding to the first estimated gaze; and for each virtual or real visual target, determining a gaze target area in the user's field of view encompassing at least part of the corresponding virtual or real visual target. ### Claim 3 (depends on 2) The media of claim 2, wherein the first estimated gaze is determined based on an estimated pupil center and an estimated cornea center of the user's eye. ### Claim 4 (depends on 2) The media of claim 2, wherein a virtual target is an object viewable by the user on a head-mounted display. ### Claim 5 (depends on 2) The media of claim 2, further comprising: assigning the first estimated gaze a confidence score; assigning the enhanced estimated user eye gaze a confidence score; and providing the enhanced estimated user eye gaze when the combination of the first estimated gaze confidence score and the enhanced estimated user eye gaze confidence score exceed a predetermined threshold. ### Claim 6 (depends on 2) The media of claim 2, further comprising: comparing the enhanced estimated user eye gaze with the first estimated gaze; and confirming the enhanced estimated user eye gaze when the enhanced estimated user eye gaze varies less than a threshold from the first estimated gaze. ### Claim 7 (depends on 1) The media of claim 1, wherein comparing user eye information comprises segmenting the plurality of images of the user's eye and the region around the user's eye into a plurality of features according to a comprehensive segmentation model and comparing the plurality of features with the narrowed subset of eye information, and wherein the eye information stored in the database is also segmented according to the comprehensive segmentation model. ### Claim 8 (depends on 1) The media of claim 1, further comprising constructing a 3D model of the user's eye based on the plurality of images of the user's eye, and wherein comparing user eye information comprises comparing the 3D model of the user's eye with the narrowed subset of eye information. ### Claim 9 (depends on 1) The media of claim 1, further comprising determining a user intent based at least in part on the estimated enhanced user eye gaze. ### Claim 10 (depends on 9) The media of claim 9, wherein determining the user intent comprises determining a user eye acceleration or deceleration from the plurality of images of the user's eye. ### Claim 11 (depends on 10) The media of claim 10, wherein determining the user intent comprises (1) inferring a user activity based on the determined user eye acceleration or deceleration or (2) determining a user activity by comparing the determined user eye acceleration or deceleration to a database of user-specific movements. ### Claim 12 (depends on 10) The media of claim 10, wherein determining the user intent comprises comparing the determined user eye acceleration or deceleration to a database of involuntary or saccade movements to determine if the determined user eye acceleration or deceleration represents a deliberate user action. ### Claim 13 (depends on 9) The media of claim 9, further comprising initiating an action on a head-mounted display based at least in part on either the estimated enhanced user eye gaze or the determined user intent. ### Claim 14 One or more computer storage media having a system embodied thereon including computer-executable instructions that, when executed, perform a method for estimating user eye gaze, the system comprising: a data component that acquires a plurality of images of a user's eye and at least one image of at least part of the user's field of view; a gaze target component that determines at least one gaze target area in the at least one image of at least part of the user's field of view based on at least one of the plurality of images of the user's eye; an eye segmentation component that segments the plurality of images of the user's eye into a plurality of features according to a comprehensive segmentation model, the segmentation model including (1) the user's eye and (2) the region around the user's eye within approximately one and one half inches of the eyelids when open; a database of eye information and corresponding known gaze lines; and an estimation component that estimates an enhanced user eye gaze by: narrowing the database of eye information to a subset of the eye information having gaze lines corresponding to the at least one gaze target area determined by the gaze target component, comparing the segmented plurality of images of the user's eye with the narrowed subset of the eye information, wherein the eye information stored in the gaze database is also segmented according to the comprehensive segmentation model, and identifying the known gaze line of matching eye information in the subset of the eye information as an enhanced estimated user eye gaze. ### Claim 15 (depends on 14) The media of claim 14, wherein the gaze target component determines the at least one gaze target area by: determining a first estimated gaze based on at least one of the plurality of images of the user's eye; identifying at least one virtual or real visual target in the at least one image of at least part of the user's field of view corresponding to the first estimated gaze; and for each virtual or real visual target, determining a gaze target area in the user's field of view encompassing at least part of the corresponding virtual or real visual target. ### Claim 16 (depends on 14) The media of claim 14, further comprising an intent component that determines a user intent based at least in part on the enhanced user eye gaze estimated by the estimation component. ### Claim 17 (depends on 14) The media of claim 14, wherein the intent component determines a user eye acceleration or deceleration from the plurality of images of the user's eye and determines a user intent based on at least one of (1) the determined user eye acceleration or deceleration or (2) a comparison between the determined user eye acceleration or deceleration and a database of user-specific movements. ### Claim 18 A computer-implemented method for estimating user eye gaze in a head-mounted display system having a processor and memory, the method comprising: acquiring a plurality of images of a user's eye; acquiring at least one image of at least part of the user's field of view; determining a first estimated gaze based on at least one of the plurality of images of the user's eye; identifying at least one virtual or real visual target in the at least one image of at least part of the user's field of view corresponding to the first estimated gaze; for each virtual or real visual target, determining a gaze target area in the user's field of view encompassing at least part of the corresponding virtual or real visual target; segmenting at least one of the plurality of images of the user's eye into a plurality of features according to a comprehensive segmentation model, the segmentation model including (1) the user's eye and (2) the region around the user's eye within approximately one and one half inches of the eyelids when open; accessing a database of eye information and corresponding known gaze lines; identifying a subset of the eye information in the database having known gaze lines within at least one of the determined gaze target areas; comparing the segmented features of the at least one of the plurality of images of the user's eye with the subset of the eye information stored in the database; and identifying the known gaze line of matching eye information in the database as an enhanced estimated gaze. ### Claim 19 (depends on 18) The media of claim 18, further comprising: comparing the estimated enhanced user eye gaze with the first estimated gaze; and confirming the enhanced estimated user eye gaze when the enhanced estimated user eye gaze varies less than a threshold percentage from the first estimated gaze. ### Claim 20 (depends on 18) The media of claim 18, further comprising: determining a user eye acceleration or deceleration from the plurality of images of the user's eye, and determining a user intent based on at least one of (1) the determined user eye acceleration or deceleration or (2) a comparison between the determined user eye acceleration or deceleration and a database of user-specific movements.