Apparatus for tracking the human eye with a retinal scanning display, and method thereof

A method for processing an image of a human eye to compute the locations of points of interest on a line-structure contained within a predefined area comprising the steps of:

• a) isolating the image area containing the line structure of interest by:
• (1) writing "ones" into a memory array for the array elements contained within a template defining the area and "zeros" into the remaining,
• (2) executing en masse the multiplication of the template array and the image array, and
• (3) writing the products into a masked memory array for matching elements;
• b) computing a histogram for the isolated image by:
• (1) writing reference values into the columns (rows) of a comparator array, where the reference values are the same for each column (row) and the values are indexed from zero to the highest intensity level,
• (2) counting en masse the comparator's "equal conditions" with a counter array as the contents of the masked memory plane are shifted along the rows (columns) of the array in a toroidal manner,
• (3) transferring en masse the counted values in the counter array to a memory array,
• (4) summing en masse the counted values in an accumulator as the memory array values are shifted along the columns (rows), and
• (5) reading the bottom row (column) of the accumulator;
• c) enhancing the intensity contrast of the isolated image with histogram equalization by:
• (1) computing a histogram equalization mapping from the intensity histogram,
• (2) writing reference values into the columns (rows) of a comparator array, where the reference values are the same for each column (row) and the values are indexed from zero to the highest intensity level,
• (3) writing the corresponding mapping function values to the elements of an accumulator array, and
• (4) writing the mapping function value to an enhanced memory array for the comparator's "equal condition" as the contents of the masked memory array and the enhanced memory array are shifted in step along the rows (columns) of the array in a toroidal manner;
• d) enhancing the lines within the isolated image with a gradient template matching of the spatial contents for edges and lines by first zeroing a destination accumulator array, and then:
• (1) performing a template matching process using the enhanced image array and a matching accumulator array to accumulate the matched contents,
• (2) adding absolute values of the contents for the template matching to the destination accumulator array,
• (3) repeating the above template matching operation for each template in the orthogonal set of line matching templates, and
• (4) writing the contents of the destination accumulator to a line-enhanced memory array, where for each template, the template matching process is performed by first zeroing a matching accumulator array, and then:
• (a) writing template weight values to an accumulator array, (b) multiplying enhanced image array by template weights, (c) adding products to contents of matching accumulator array, (d) shifting enhanced image array in turn to left, right, down, up, and each diagonal, and repeating above template weighting process for each shift;
• e) segmenting the line-structure of interest by binary thresholding, as follows:
• (1) computing the histogram for the edge enhanced image,
• (2) selecting a threshold value,
• (3) writing the threshold as a reference value to the elements of a comparator array, and
• (4) writing an "off" to a memory array if the intensity of an element of the edge enhanced image is less or equal to the comparator's threshold value and an "on" if it is greater;
• f) isolating the points of the line-segments by:
• (1) counting the number with an accumulator array of neighboring pixels that are "on" for each on-pixel in the segmented image,
• (2) comparing the count to a threshold count with a comparator array, and
• (3) writing a non-count value to a memory array for a count less than the threshold, and a count value for equal or more;
• and
• g) saving in cache memory the image coordinates of the points of the line segments, by reading the isolation array with the CPU as the elements are shifted by rows (columns) and columns (rows) until all elements are read, and storing the row and column addresses of those elements that are "on".

A method for processing an image of the human eye to compute the moments of the pupil image and the locations of:

• (1) the capillary junctions of the retinal fundus, and (2) the cusps of the sphincteral pattern on the iris, comprising the steps of:
• a) executing the following upon receipt by the CPU of the clocked display field refresh pulse of a video display;
• b) computing the moments of the pupil image and an isolation template by:
• (1) initializing a clocked transfer mechanism for transfer of the accumulated image contained by a CCD to an analog-to-digital converter;
• initializing a moment generator circuit for analysis of the output of the same with the threshold of a comparator within the moment generator preset to pass only the pupil image;
• reading the serial digital output from a clocked analog-to-digital converter and writing the values to a digital image memory array cache of the stack;
• reading the output from the moment generator circuit upon acquisition of the digitized image from the clocked transfer mechanism;
• and computing a template from the ellipsoidal moments for isolating the pupil image from the digital image memory array cache;
• c) determining the number of capillary junctions within the fundus of the image, by:
• isolating the pupil image with the said isolation template, enhancing the isolated pupil image by histogram equalization, enhancing the capillary network on the pupil fundus with application of spatial gradient templates for edges and lines, segmenting the network with binary thresholding, and isolating the junction points of the network segments with a count threshold of three, a non-count value of "off", and a count value of "on";
• d) computing an isolation template for the iris by:
• calculating a threshold reference value with the histogram technique applied to the digitized eye image without the pupil image, intensity thresholding of the reduced image, and binary equalization of the same;
• e) determining the number of cusps of the sphincteral pattern on the iris, by:
• isolating the iris image with the said isolation template, enhancing the isolated iris image by histogram equalization, enhancing the outline of the sphincteral pattern on the iris with application of spatial gradient templates for edges and lines, segmenting the outline with binary thresholding, and isolating the cusps of the pattern with a count threshold of two a non-count value of "on", and a count value of "off";
• and
• f) outputting the pupil image moments and the image coordinates of the capillary junctions of the retinal fundus and the cusps of the sphincteral pattern on the iris, to a memory mapped area.

A method using an ophthalmomatric ray tracing approach for computing the intraocular location of an ocular feature from an eye image location a source-to-sensor pairing table the corneal optic center-to-feature distance, and the corneal center, comprising the steps of:

• (a) finding a set of corneal reflection points which listed in a source-to-sensor pairing table are at or near the eye image location of the ocular feature so as to form a nearest-neighbor grid surrounding and overlapping the image of the feature;
• (b) computing the optical locations of the corneal reflection points in the said nearest-neighbor set from the locations and orientations of the corresponding sensors and paired source excitations;
• (c) computing the location of the corneal refraction point for the ocular feature with numerical interpolation by fitting a local surface to the set of neighboring reflection points;
• (d) computing the surface normal for the said refraction point from the said location and the corneal optic center;
• (e) computing the angle of incident for the light ray from the feature, from the location and surface normal of the said refraction point and the location of the corresponding sensor in the image plane of the accumulator;
• (f) computing the directional cosines for the transmitted ray at the said refraction point from the corneal index of refraction and Snell's law relating the angle of incidence to the angle of refraction;
• (g) tracing a ray from the said refraction point back to the ocular feature in the eye including transmission through the crystalline lens and the internal vitreous body;
• (h) computing the intersection points of the ray with an imaginary sphere centered on the corneal optic center and having a radius equal to the intraocular distance between the center and the ocular feature as determined in calibration;
• and
• (i) determining the location of the feature from the said intersection points with the values of the vector dot products formed from the directional cosines of the back traced ray and the outer directed surface normals to the said imaginary sphere at the points of intersection from the corneal optic center.

A method for computing the gaze point of regard and fixation duration for each eye, and the binocular convergence point for both eyes of the viewer, using a digital computer program, comprising the steps of:

• a) an interrupt service routine which in response to the clocked display field refresh pulse output from a video display, calls in turn, upon the following routines;
• b) a routine for reading the position and orientation of a headset sensor;
• c) a routine using a three dimensional model of corneal reflections, which computes the corneal center of each eye and the surface point locations of the reflections, from a source-to-sensor pairing table and embedded memory tables of the geometry's for light source locations and directions and a sensor array;
• d) a routine which smooths the computed corneal reflections for each eye by clustering together surface points with nearly the same orientation relative to the corresponding corneal center, and computes the surface center, the surface normal at the center, and the median axes of the cornea from the smoothed surface;
• e) a routine which computes the locations of the pupil optical center for each eye using an ophthalmomatric ray tracing approach and the corneal optic center to pupil center distance, from the corresponding said corneal center, a memory mapped pupil image center, and a source to sensor pairing table;
• f) a routine which computes the locations of the capillary junctions of the retinal fundus and the cusps of the sphincteral pattern on the iris for each eye using an ophthalmomatric ray tracing approach and the corneal optic center distances, from the corresponding said corneal center, memory mapped eye image locations, and the source to sensor pairing table;
• g) a routine which computes the optical origin, and the optical and median axes for each eye, by a best fit to a three dimensional model of the computed values for the corneal optic center, the corneal surface center and axes, the pupil optical center, the pupil orientation, the capillary network of the retinal fundus, and the sphincteral pattern of the iris;
• h) a routine which computes the viewing origin and axis for each eye from the corresponding said optical origin and axis and the median axes;
• j) a routine which converts the viewing origin and axis for each eye to that for the viewer's workspace from the said headset position and orientation sensor reading;
• k) a routine which classifies the viewing status of each eye as saccadic, fixation, or pursuit from the corresponding immediate eye movement history;
• l) a routine which classifies the gaze status of each eye by grouping local fixations, and computes the gaze point of regard in the viewer's workspace and fixation duration, from the corresponding viewing origin and axis, and an embedded memory table of the workspace geometry;
• and
• m) a routine to compute the binocular convergence point from the viewing origins and axes of both eyes for the viewer's workspace.