# US 11,442,540 B2 · Eye tracking using low resolution images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,442,540 B2 - Document ID: US11442540B2 - Status: Granted - Kind: B2 - Application number: 17/021,920 - Filed: 2020-09-15 - Published: 2021-03-18 - Granted: 2022-09-13 - Earliest claimed: 2019-09-18 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Tom Sengelaub - Web page: https://patentgrep.com/patent/US11442540B2 ## CPC - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 27/017: {Head mounted} - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - H04N 23/20: for generating image signals from infrared radiation only - H04N 23/651: {for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera} - H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G01S 5/16: using electromagnetic waves other than radio waves - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G06T 2207/10048: Infrared image - G06T 2207/30196: Human being; Person - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES ## Abstract Low-power eye tracking for detecting position and movements of a user's eyes in a head-mounted device (HMD). An eye tracking system for an HMD may include eye tracking cameras. The eye tracking cameras may be used to capture low-resolution frames between capturing high-resolution frames, for example by binning pixels on the camera sensor or by capturing horizontal and vertical stripes or lines of pixels on the camera sensor rather than entire frames. This low-resolution information may be used to track relative movement of the user's eyes with respect to the device in intervals between the processing of full, high-resolution frames captured by the eye tracking cameras. ## Claims ### Claim 1 A system, comprising: a head-mounted device (HMD) configured to display visual content for viewing by a user, wherein the HMD comprises at least one camera configured to switch between a high-resolution mode and a low-resolution mode to respectively capture high-resolution images of the user's eyes and low-resolution images of the user's eyes at a frame rate of N frames per second; and a controller comprising one or more processors configured to iteratively perform: process a high-resolution image captured by the at least one camera in the high-resolution mode using three-dimensional (3D) image processing techniques to track movement of the user's eyes with respect to the HMD; signal the at least one camera to switch to the low-resolution mode; and process one or more low-resolution images subsequently captured by the at least one camera in the low-resolution mode using two-dimensional (2D) image processing techniques to track movement of the user's eyes with respect to the HMD in intervals between high-resolution frames captured by the at least one camera and processed by the controller. ### Claim 2 (depends on 1) The system as recited in claim 1, wherein the low-resolution frames are captured by binning blocks of pixels on a camera sensor. ### Claim 3 (depends on 1) The system as recited in claim 1, wherein the low-resolution frames are captured by capturing horizontal and vertical lines of pixels on a camera sensor. ### Claim 4 (depends on 1) The system as recited in claim 1, wherein the controller is further configured to perform a three-dimensional (3D) reconstruction based on at least one high-resolution image captured by the at least one camera to generate 3D models of the user's eyes, wherein the 3D models indicate position of the user's eyes with respect to the at least one camera. ### Claim 5 (depends on 1) The system as recited in claim 1, wherein the controller is further configured to, upon detecting that the HMD has shifted on the user's head: signal the at least one camera to switch to the high-resolution mode; and process one or more subsequent high-resolution images captured by the at least one camera to update the 3D models to indicate a new position of the user's eyes with respect to the HMD. ### Claim 6 (depends on 1) The system as recited in claim 1, wherein the controller is configured to apply eye movement information obtained by processing the low-resolution images to update a position of the user's eyes with respect to the at least one camera determined by processing the high-resolution images in the intervals between the processing of high-resolution images captured by the at least one camera. ### Claim 7 (depends on 1) The system as recited in claim 1, wherein the controller is a component of the HMD. ### Claim 8 (depends on 1) The system as recited in claim 1, wherein the HMD further comprises: at least one display screen configured to display frames containing the visual content for viewing by the user, wherein the controller is further configured to render the frames containing the visual content for display by the at least one display screen; and one or more light sources configured to emit light towards the user's eyes, wherein the at least one camera captures a portion of the light reflected off the user's eyes. ### Claim 9 (depends on 6) The system as recited in claim 6, wherein the HMD further comprises left and right optical lenses located between the at least one display screen and the user's eyes. ### Claim 10 A method, comprising: performing, by a controller comprising one or more processors: processing a high-resolution image captured by at least one camera of a head-mounted device (HMD) in high-resolution mode using three-dimensional (3D) image processing techniques to track movement of the user's eyes with respect to the HMD; signaling the at least one camera to switch to low-resolution mode; and processing one or more low-resolution images subsequently captured by the at least one camera in the low-resolution mode using two-dimensional (2D) image processing techniques to track movement of the user's eyes with respect to the HMD in intervals between high-resolution frames captured by the at least one camera and processed by the controller. ### Claim 11 (depends on 10) The method as recited in claim 10, wherein the low-resolution frames are captured by binning blocks of pixels on a camera sensor. ### Claim 12 (depends on 10) The method as recited in claim 10, wherein the low-resolution frames are captured by capturing horizontal and vertical lines of pixels on a camera sensor. ### Claim 13 (depends on 10) The method as recited in claim 10, further comprising perform a three-dimensional (3D) reconstruction based on at least one high-resolution image captured by the at least one camera to generate 3D models of the user's eyes, wherein the 3D models indicate position of the user's eyes with respect to the at least one camera. ### Claim 14 (depends on 10) The method as recited in claim 10, further comprising, upon detecting that the HMD has shifted on the user's head: signaling the at least one camera to switch to the high-resolution mode; and processing one or more subsequent high-resolution images captured by the at least one camera to update the 3D models to indicate a new position of the user's eyes with respect to the HMD. ### Claim 15 (depends on 10) The method as recited in claim 10, further comprising applying eye movement information obtained by processing the low-resolution images to update a position of the user's eyes with respect to the at least one camera determined by processing the high-resolution images in the intervals between the processing of high-resolution images captured by the at least one camera. ### Claim 16 One or more non-transitory computer-readable storage media storing program instructions that when executed on or across one or more processors cause the one or more processors to: process a high-resolution image captured by at least one camera of a head-mounted device (HMD) in high-resolution mode using three-dimensional (3D) image processing techniques to track movement of the user's eyes with respect to the HMD; signaling the at least one camera to switch to low-resolution mode; and process one or more low-resolution images subsequently captured by the at least one camera in the low-resolution mode using two-dimensional (2D) image processing techniques to track movement of the user's eyes with respect to the HMD in intervals between high-resolution frames captured by the at least one camera and processed by the controller.