Binocular eye tracking from video frame sequences

A process for tracking 3D positions and angles of gaze of the eyes in space from facial information in video or other image sequences of said eye region of the face of an individual, whereby said information comprises a 2D projection of each of a plurality of 3D configurations of one or more of the following facial features:

• i) an eyeball having a spherical radius and a 3D position,
• ii) an upper eyelid having an angle of closure,
• iii) a lower eyelid having an angle of closure,
• iv) an iris having a circular radius and center with a two-parameter location relative to the center of said eyeball defining the angle of gaze,
• v) a pupil having a circular radius,
• vi) a scleral region bounded by an iris, an upper eyelid and a lower eyelid,
• said process comprising the steps of:
• obtaining a video image of the eyes of a subject, said video image comprising an array of image pixels, each of said pixels having an image pixel trichromatic value,
• storing said video image in a storage buffer,
• providing a standard 3D geometric model of said facial features, said 3D geometric model incorporating a plurality of adjustable parameters of said 3D configurations of said facial features, including said 3D locations, angles of gaze, and pupil radii,
• deriving a projected 2D view of said 3D geometric model of said facial features, said 2D view comprised of model pixels;
• defining a plurality of zones of said projected 2D view, each corresponding to one of said facial features, each of said zones containing a respective plurality of model pixels, each of said model pixels having a model pixel trichromatic value,
• determining the average trichromatic value of a plurality of said model pixels in each of said zones,
• determining the deviation of the image pixel trichromatic value of each of said plurality of said image pixels from said average trichromatic value of said plurality of said model pixels in each of said zones to determine a trichromatic pixel error value for each of said image pixels,
• combining said trichromatic pixel error values across said plurality of said zones to calculate a combined error value over a plurality of said image zones constrained by said 3D geometric model,
• analyzing histogramically the distributions of said image pixel trichromatic values of said pixels within said zones of scleral and iris features to trim and remove outlier values due to specular glint reflections of incident light sources,
• varying said adjustable parameters of said geometric 3D model and repeating the steps of deriving a 2D view, determining the deviation of the image pixel trichromatic values and determining the trichromatic pixel error values and calculating the combined error values over a plurality of the image zones in an iterative loop until said combined error value across said plurality of image zones converges to a minimum value,
• storing in a trajectory accumulator said values of said adjustable parameters, including said 3D positions, angles of gaze, eyelid angles of closure and pupil sizes of said eyes, under conditions providing said minimum values of said combined error values, to form trajectories of said adjustable parameters over time,
• whereby the trajectories of said eyeball positions, angles of gaze, eyelid angles of closure, and pupil radii, are tracked over time, where said tracking is done without calibration and without the use of any specialized equipment except widely available consumer devices incorporating a camera selected from the group including:
• i) video cameras,
• ii) television monitors,
• iii) smartphones
• iv) tablet computers
• v) laptop computers
• vi) desktop computers
• vii) gaming consoles
• viii) virtual reality devices
• ix) wearable electronics
• x) inbuilt architectural displays.