# US 10,768,698 B1 · Method and device for eye tracking with content-embedded glints > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,768,698 B1 - Document ID: US10768698B1 - Status: Granted - Kind: B1 - Application number: 16/015,858 - Filed: 2018-06-22 - Granted: 2020-09-08 - Earliest claimed: 2017-09-28 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Jae Hwang Lee; Avi Bar-Zeev; Fletcher R. Rothkopf - Web page: https://patentgrep.com/patent/US10768698B1 ## CPC - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/012: {Head tracking input arrangements} - G06T 7/74: {involving reference images or patches} - G06V 10/141: Control of illumination - G06V 20/20: in augmented reality scenes - G06V 40/18: Eye characteristics, e.g. of the iris - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 27/0172: {characterised by optical features} - G02B 2027/014: {comprising information/image processing systems} - G06T 2207/10048: Infrared image - G06T 2207/30041: Eye; Retina; Ophthalmic - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract In one implementation, a method includes: synthesizing an AR/VR content stream by embedding a plurality of glints provided for eye tracking into one or more content frames of the AR/VR content stream; displaying, via the one or more AR/VR displays, the AR/VR content stream to a user of the HMD; obtaining, via the image sensor, light intensity data corresponding to the one or more content frames of the AR/VR content stream that include the plurality of glints, wherein the light intensity data includes a projection of an eye of the user of the HMD having projected thereon the plurality of glints; and determining an orientation of the eye of the user of the HMD based on the light intensity data. ## Claims ### Claim 1 A method comprising: at a head-mounted device (HMD) provided to deliver augmented reality/virtual reality (AR/VR) content, the HMD including with one or more processors, non-transitory memory, one or more AR/VR displays, and an image sensor: synthesizing an AR/VR content stream including a plurality of content frames by embedding a plurality of glints provided for eye tracking into the AR/VR content stream; displaying, via the one or more AR/VR displays, the AR/VR content stream to a user of the HMD; obtaining, via the image sensor, light intensity data corresponding to an eye of the user of the HMD having projected thereon the plurality of glints; and determining an orientation of the eye of the user of the HMD based on the light intensity data, wherein embedding the plurality of glints provided for eye tracking into the AR/VR content stream includes interleaving an eye tracking frame with the plurality of glints embedded thereon among the plurality of content frames according to a ratio of content frames to eye tracking frames. ### Claim 2 (depends on 1) The method of claim 1, further comprising determining the ratio based on at least one of a next content frame in the AR/VR content stream, ambient light conditions, or the orientation of the eye of the user of the HMD for a previous time period. ### Claim 3 (depends on 1) The method of claim 1, wherein embedding the plurality of glints provided for eye tracking into the AR/VR content stream includes embedding the plurality of glints into one or more content frames of the plurality of content frames, wherein the plurality of glints is displayed in the infrared (IR) domain. ### Claim 4 (depends on 1) The method of claim 1, further comprising determining a pattern for the plurality of glints based on at least one of a next content frame in the AR/VR content stream, ambient light conditions, or the orientation of the eye of the user of the HMD for a previous time period. ### Claim 5 (depends on 1) The method of claim 1, further comprising determining a shape for each of the plurality of glints based on at least one of a next content frame in the AR/VR content stream, ambient light conditions, or the orientation of the eye of the user of the HMD for a previous time period. ### Claim 6 (depends on 1) The method of claim 1, wherein the one or more AR/VR displays and the image sensor are synchronized in order to obtain, via the image sensor, the light intensity data. ### Claim 7 (depends on 1) The method of claim 1, wherein a first AR/VR display among the one or more AR/VR displays displays a first frame of the AR/VR content stream having embedded thereon the plurality of glints, and wherein a second AR/VR display among the one or more AR/VR displays displays a second frame of the AR/VR content stream having embedded thereon the plurality of glints, wherein the first and second frames are associated with temporally adjacent time periods. ### Claim 8 A device comprising: one or more processors; a non-transitory memory; an image sensor; one or more AR/VR displays; and one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to: synthesize an AR/VR content stream including a plurality of content frames by embedding a plurality of glints provided for eye tracking into the AR/VR content stream; display, via the one or more AR/VR displays, the AR/VR content stream to a user of the HMD; obtain, via the image sensor, light intensity data corresponding to an eye of the user of the HMD having projected thereon the plurality of glints; and determine an orientation of the eye of the user of the HMD based on the light intensity data, wherein embedding the plurality of glints provided for eye tracking into the AR/VR content stream includes interleaving an eye tracking frame with the plurality of glints embedded thereon among the plurality of content frames according to a ratio of content frames to eye tracking frames. ### Claim 9 (depends on 8) The device of claim 8, wherein the one or more AR/VR displays include an RGB display and plurality of glints are displayed in the RGB domain. ### Claim 10 (depends on 8) The device of claim 8, wherein embedding the plurality of glints provided for eye tracking into the AR/VR content stream includes embedding the plurality of glints into one or more content frames of the plurality of content frames, wherein the plurality of glints is displayed in the infrared (IR) domain. ### Claim 11 (depends on 10) The device of claim 10, wherein the one or more AR/VR displays include an RGB display, an IR illuminator, and an IR filter. ### Claim 12 (depends on 10) The device of claim 10, wherein the one or more AR/VR displays include an RGBI display. ### Claim 13 (depends on 8) The device of claim 8, wherein the one or more programs further cause the device to determine a pattern for the plurality of glints based on at least one of a next content frame in the AR/VR content stream, ambient light conditions, or the orientation of the eye of the user of the HMD for a previous time period. ### Claim 14 (depends on 8) The device of claim 8, wherein the one or more programs further cause the device to determine a shape for each of the plurality of glints based on at least one of a next content frame in the AR/VR content stream, ambient light conditions, or the orientation of the eye of the user of the HMD for a previous time period. ### Claim 15 (depends on 8) The device of claim 8, wherein the one or more AR/VR displays include a first AR/VR display and a second AR/VR display, wherein the first AR/VR display displays a first frame of the AR/VR content stream having embedded thereon the plurality of glints, wherein the second AR/VR display displays a second frame of the AR/VR content stream having embedded thereon the plurality of glints, and wherein the first and second frames are associated with temporally adjacent time periods. ### Claim 16 A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device with one or more AR/VR displays and an image sensor, cause the device to: synthesize an AR/VR content stream including a plurality of content frames by embedding a plurality of glints provided for eye tracking into the AR/VR content stream; display, via the one or more AR/VR displays, the AR/VR content stream to a user of the HMD; obtain, via the image sensor, light intensity data corresponding to an eye of the user of the HMD having projected thereon the plurality of glints; and determine an orientation of the eye of the user of the HMD based on the light intensity data, wherein embedding the plurality of glints provided for eye tracking into the AR/VR content stream includes interleaving an eye tracking frame with the plurality of glints embedded thereon among the plurality of content frames according to a ratio of content frames to eye tracking frames. ### Claim 17 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein embedding the plurality of glints provided for eye tracking into the AR/VR content stream includes embedding the plurality of glints into one or more content frames of the plurality of content frames, wherein the plurality of glints is displayed in the infrared (IR) domain.