# US 11,307,659 B2 · Low-power eye tracking system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,307,659 B2 - Document ID: US11307659B2 - Status: Granted - Kind: B2 - Application number: 17/021,901 - Filed: 2020-09-15 - Published: 2021-03-18 - Granted: 2022-04-19 - Earliest claimed: 2019-09-18 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Tom Sengelaub - Web page: https://patentgrep.com/patent/US11307659B2 ## CPC - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 27/017: {Head mounted} - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - H04N 23/20: for generating image signals from infrared radiation only - H04N 23/651: {for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera} - H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G01S 5/16: using electromagnetic waves other than radio waves - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G06T 2207/10048: Infrared image - G06T 2207/30196: Human being; Person - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES ## Abstract Low-power eye tracking for detecting position and movements of a user's eyes in a head-mounted device (HMD). An eye tracking system for an HMD may include head odometers, eye odometers, or both head odometers and eye odometers to reduce power consumption of the system. Head odometers may be used to detect movement of the device with respect to the user's eyes, which may allow 3D reconstruction to generate 3D models of the user's eyes used in eye tracking to be performed only when movement of the device with respect to the user's eyes has been detected. Eye odometers may be used as a low-power component to track relative movement of the user's eyes with respect to the device between frames captured by the eye tracking cameras, which may allow the frame rate of the eye tracking cameras to be reduced. ## Claims ### Claim 1 A system, comprising: a head-mounted device (HMD) comprising: at least one camera configured to capture images of a user's eyes at a frame rate of N frames per second; and one or more head motion sensors configured to detect motion of the HMD with respect to the user's head; and a controller comprising one or more processors configured to: perform an initial three-dimensional (3D) reconstruction based on at least one image captured by the at least one camera to generate initial 3D models of the user's eyes, wherein the initial 3D models indicate position of the user's eyes with respect to the at least one camera; perform two-dimensional (2D) image processing of one or more subsequent images captured by the at least one camera to track movement of the user's eyes with respect to the HMD until detecting that the HMD has shifted on the user's head based on signals from the one or more head motion sensors; and upon detecting that the HMD has shifted on the user's head based on the signals from the one or more head motion sensors, update the 3D models to indicate a new position of the user's eyes with respect to the at least one camera. ### Claim 2 (depends on 1) The system as recited in claim 1, wherein, to update the 3D models to indicate a new position of the user's eyes with respect to the at least one camera, the controller is configured to perform the 3D reconstruction based on at least one subsequent image captured by the at least one camera to generate new 3D models of the user's eyes. ### Claim 3 (depends on 1) The system as recited in claim 1, wherein, to update the 3D models to indicate a new position of the user's eyes with respect to the at least one camera, the controller is configured to adjust the initial 3D models based on magnitude and direction of movement of the HMD with respect to the user's head determined from the signals from the one or more head motion sensors. ### Claim 4 (depends on 1) The system as recited in claim 1, wherein the one or more head motion sensors include at least one sensor located in the HMD at or near the bridge of the user's nose and at least one sensor located in the HMD at or near the user's temples. ### Claim 5 (depends on 1) The system as recited in claim 1, wherein the HMD further comprises one or more eye motion sensors configured to detect movement of the user's eyes with respect to the HMD, wherein the controller is further configured to process signals from the eye motion sensors to track movement of the user's eyes with respect to the HMD in intervals between processing frames captured by the at least one camera. ### Claim 6 (depends on 1) The system as recited in claim 1, wherein the one or more cameras are configured to capture high-resolution frames and low-resolution frames. ### Claim 7 (depends on 6) The system as recited in claim 6, wherein the 3D reconstruction to generate 3D models of the user's eyes is performed on high-resolution frames captured by the one or more cameras; and wherein the 2D image processing is performed on low-resolution frames captured by the one or more cameras. ### Claim 8 (depends on 6) The system as recited in claim 6, wherein the controller is further configured to: process high-resolution images captured by the at least one camera to track movement of the user's eyes with respect to the HMD; and track movement of the user's eyes with respect to the HMD in intervals between the high-resolution frames captured by the at least one camera and processed by the controller. ### Claim 9 (depends on 6) The system as recited in claim 6, wherein the low-resolution frames are captured by binning blocks of pixels on a camera sensor or by capturing horizontal and vertical lines of pixels on the camera sensor. ### Claim 10 (depends on 1) The system as recited in claim 1, wherein the controller is a component of the HMD. ### Claim 11 (depends on 1) The system as recited in claim 1, wherein the HMD further comprises: at least one display screen configured to display frames containing the visual content for viewing by the user, wherein the controller is further configured to render the frames containing the visual content for display by the at least one display screen; and one or more light sources configured to emit light towards the user's eyes, wherein the at least one camera captures a portion of the light reflected off the user's eyes. ### Claim 12 (depends on 11) The system as recited in claim 11, wherein the HMD further comprises left and right optical lenses located between the at least one display screen and the user's eyes. ### Claim 13 A method, comprising: performing, by one or more processors: generating an initial three-dimensional (3D) model of a user's eye based on one or more images of the user's eye captured by a camera of a head-mounted device (HMD); processing subsequent images captured by the camera using two-dimensional (2D) image processing to track movement of the user's eye with respect to the HMD until detecting that the HMD has shifted on the user's head based on signals from one or more head motion sensors of the HMD; and updating the 3D model of the user's eye to indicate a new position of the user's eye with respect to the camera upon detecting that the HMD has shifted on the user's head based on the signals from the one or more head motion sensors of the HMD. ### Claim 14 (depends on 13) The method as recited in claim 13, wherein updating the 3D model to indicate a new position of the user's eye with respect to the camera comprises generating a new 3D model of the user's eye based on at least one subsequent image captured by the camera. ### Claim 15 (depends on 13) The method as recited in claim 13, wherein updating the 3D model to indicate a new position of the user's eye with respect to the camera comprises adjusting the 3D model based on magnitude and direction of movement of the HMD with respect to the user's head determined from the signals from the one or more head motion sensors. ### Claim 16 (depends on 13) The method as recited in claim 13, further comprising tracking movement of the user's eye with respect to the HMD in intervals between frames captured by the camera based on signals from one or more eye motion sensors of the HMD. ### Claim 17 (depends on 13) The method as recited in claim 13, wherein the camera is configured to capture high-resolution images of the user's eye and low-resolution images of the user's eye, and wherein the one or more processors are further configured to iteratively perform: processing a high-resolution image captured by the camera to track movement of the user's eye with respect to the HID; and processing one or more low-resolution images captured by the camera to track movement of the user's eye with respect to the HMD between high-resolution frames captured by the camera, wherein the low-resolution frames are captured by binning blocks of pixels on the camera's sensor or by capturing horizontal and vertical lines of pixels on the camera's sensor. ### Claim 18 A device, comprising: at least one camera configured to capture high-resolution images and low-resolution images of a user's eyes; at least one head motion sensor configured to detect motion of the device with respect to the user's head; and a controller comprising one or more processors configured to: process one or more high-resolution images captured by the at least one camera to generate initial three-dimensional (3D) models of the user's eyes, wherein the 3D models indicate position of the user's eyes with respect to the device; signal the at least one camera to switch to low-resolution mode; process one or more subsequent low-resolution images captured by the at least one camera to track relative movement of the user's eyes with respect to the device; and upon detecting that the device has shifted on the user's head based on signals from the one or more head motion sensors: signal the at least one camera to switch to high-resolution mode; and process one or more subsequent high-resolution images captured by the at least one camera to update the 3D models to indicate a new position of the user's eyes with respect to the device. ### Claim 19 (depends on 18) The device as recited in claim 18, wherein the low-resolution images are captured in low-resolution mode by binning blocks of pixels on a camera sensor or by capturing horizontal and vertical lines of pixels on the camera sensor. ### Claim 20 (depends on 18) The device as recited in claim 18, wherein the high-resolution images are processed using 3D image processing, and wherein the low-resolution images are processed using two-dimensional (2D) image processing.