# US 10,311,583 B2 · Eye motion detection method, program, program storage medium, and eye motion detection device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,583 B2 - Document ID: US10311583B2 - Status: Granted - Kind: B2 - Application number: 15/579,241 - Filed: 2016-06-03 - Published: 2018-06-21 - Granted: 2019-06-04 - Earliest claimed: 2015-06-05 - Applicants: Kiyoshi Hoshino - Inventors: Kiyoshi Hoshino - Web page: https://patentgrep.com/patent/US10311583B2 ## CPC - G06T 7/248: {involving reference images or patches} - A61B 3/113: for determining or recording eye movement - G06T 7/11: Region-based segmentation - G06T 7/74: {involving reference images or patches} - G06T 2207/30041: Eye; Retina; Ophthalmic - G06T 2207/30201: Face - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) ## Abstract Provided is a method to easily detect eye motions in the eye moving direction, which has resistance to a fluctuation in background light and to a displacement of the position of a camera. The method includes the steps of: detecting a white area of the eye from the eye area; determining a blood vessel image to be used for a template blood vessel image in the white area of the eye; detecting a nth blood vessel image, which matches the template blood vessel image; calculating a nth moving amount of the eye based on a difference between the reference position and the detection position of the nth blood vessel image, which matches the template blood vessel image; and calculating a nth rotation angle of the eye to detect eye motions in the information processing apparatus. ## Claims ### Claim 1 An eye motion detection method, which is incorporated in an information processing apparatus, comprising the steps of: (a) entering a reference image of a predetermined area including an area around at least one of eyes, which is captured by an imaging device installed in the vicinity of the front side of at least the one of eyes or at its optically equivalent position; (b) detecting at least an eye inner corner and an eye outer corner of the eye from the reference image; (c) detecting the white area of the eye from the eye area between the eye inner corner and the eye outer coroner of the reference image; (d) extracting all discriminable blood vessel images from the white area; (e) determining a blood vessel image to be used for a template blood vessel image among the all blood vessel images; (f) recording a reference position and the reference angle of a horizontal reference of the template blood vessel image in the white area of the eye in the coordinate system; (g) entering an image for detecting a nth (n is a one or larger integer starting from 1 and gradually increasing in 1 (one) increments) blood vessel detecting image captured by the imaging device; (h) detecting the nth blood vessel image, which matches the template blood vessel image (hereinafter, simply referred to as the nth matched blood vessel image), from the nth blood vessel detecting image; (i) recording the nth detected position and the nth detected angle of the detected nth matched blood vessel image in the white area of the eye in the coordinate system; (j) calculating a nth moving amount of the eye based on a difference between the reference position and the nth detected position in the coordinate system and calculate a nth eye rotation angle based on a difference between the reference angle and the nth detected angle; and (k) repetitively, executing the above step (g) to step (j) with setting the n=n+1, detecting a (n+1)th matched blood vessel image from an entered (n+1)th blood vessel detecting image and calculating a (n+1)th moving amount and a (n+1)th rotation angle of the (n+1)th matched blood vessel image. ### Claim 2 (depends on 1) The eye motion detection method defined in claim 1, wherein step (b) of detecting the eye inner corner and eye outer corner and step (c) of detecting the white area of the eye feature vector values including the brightness values of the binarized image and detecting areas whose binarized brightness value is higher than or equal to its threshold value. ### Claim 3 (depends on 1) The eye motion detection method defined in claim 1, wherein in step (e) of determining the template blood vessel image and step in (h) of detecting the nth matched blood vessel image, erosion and dilation are executed at least once, respectively, on the template blood vessel image and the individual nth blood vessel detection images. ### Claim 4 (depends on 1) The eye motion detection method defined in claim 1, wherein in step (h) of detecting the nth matched blood vessel image, the nth matched blood vessel image is matched with the template blood vessel image in accordance with a moving amount in the vertical, horizontal and/or rotational direction. ### Claim 5 (depends on 1) The eye motion detection method defined in claim 1, wherein in step (c) of detecting the white area from the reference image, an iris area which are not included in the white area of the eye and the center of a pupil are detected, wherein in step (f) of recording a reference position and a reference angle of the template blood vessel image in the coordinate system, a distance from the iris area to the template blood vessel image in the coordinate system and the angle from the horizontal reference line passing through the center of the pupil, wherein in step (e) of determining the template blood vessel image and in step (h) of detecting the nth matched blood vessel image, termination or detection with selection of a blood vessel image, which is found in the vicinity area of the iris area on an ear side in the white area of the eye and in the vicinity area in the lower side area of the horizontal reference line. ### Claim 6 (depends on 1) The eye motion detection method defined in claim 1, wherein in step (e) of determining the template blood vessel image and in step (h) of detecting the nth matched blood vessel image, labeling the individual pixels of the white area of the eye and individual blood vessel images, wherein the blood vessel image is determined or detected; (1) by calculating the concatenated number of long-side concatenated pixel area with same label in each blood vessel image and select a blood vessel image which has the largest concatenated number, or (2) by calculating a peripheral length of the concatenated pixel area with same label in each blood vessel image and select a blood vessel image which has the largest peripheral length, or (3) by obtaining the first principal component which indicates a displacement range relative to the first component axis in the concatenated pixel area with same label, and the second principal component which indicates a displacement range relative to the second component axis in the concatenated pixel area with same label, and selecting a blood vessel image which has the largest sum value of the displacement range of the first component axis and the displacement range of the second component axis. ### Claim 7 (depends on 1) The eye motion detection method defined in claim 1, wherein obtaining at least the eye inner corner and the eye outer corner as the two feature points from the reference image, and determining the third feature in accordance with a predetermined triangle, and suppressing blur in the nth blood vessel detecting image by setting the triangle formed by the three feature points are similar form. ### Claim 8 (depends on 1) A program executing the steps in the eye motion detection method of claim 1. ### Claim 9 (depends on 8) A non-transitory storage medium storing the program in claim 8. ### Claim 10 An eye motion detection device comprising: (A) an imaging device installed in the vicinity of at least one of eyes on the front side, or at its optically equivalent position; and (B) an information processing apparatus, which at least executes the steps of; entering the reference image of the predetermined area including the peripheral of the eye entered from the imaging device (a); detecting at least the eye inner corner and the eye outer corner from the reference image (b); detecting the white area of the eye from the area between the eye inner corner and the eye outer corner on the reference image (c); extracting all the blood vessel images discriminable from the white area of the eye (d); determining a blood vessel image to be used for a template blood vessel image among all the blood vessel (e); recording a reference position and a reference angle of a horizontal reference in the coordinate system of the template blood vessel image in the white area of the eye (f); entering an image for detecting a nth (n is a one or larger integer starting from 1 and gradually increasing in 1 (one) increments) blood vessel image captured by the imaging device (g); detecting a nth blood vessel image, which matches the template blood vessel image (h); recording the detection point and detection angle of the nth blood vessel image, which matches the template blood vessel image, in the white area of the eye in the coordinate system (i); calculating a moving amount of the nth blood vessel based on the reference point and detection position of the nth blood vessel in the coordinate system and calculating an eye rotation angle of the nth blood vessel based on a difference between the reference angle and detection angle of the nth blood vessel (j); and executing the aforementioned steps (g) to (j) with setting the n=n+1 to detect the (n+1)th blood vessel image, which matched the template blood vessel image from the image for detecting the (n+1)th blood vessel image entered; and step (k) of repeating the calculation of the moving amount and rotation angle of the nth blood vessel. ### Claim 11 (depends on 2) The eye motion detection method defined in claim 2, wherein in step (e) of determining the template blood vessel image and in step (h) of detecting the nth matched blood vessel image, erosion and dilation are executed at least once, respectively, on the template blood vessel image and the individual nth blood vessel detection images. ### Claim 12 (depends on 2) The eye motion detection method defined in claim 2, wherein in step (h) of detecting the nth matched blood vessel image, the nth matched blood vessel image is matched with the template blood vessel image in accordance with a moving amount in the vertical, horizontal and/or rotational direction. ### Claim 13 (depends on 3) The eye motion detection method defined in claim 3, wherein in step (h) of detecting the nth matched blood vessel image, the nth matched blood vessel image is matched with the template blood vessel image in accordance with a moving amount in the vertical, horizontal and/or rotational direction. ### Claim 14 (depends on 11) The eye motion detection method defined in claim 11, wherein in step (h) of detecting the nth matched blood vessel image, the nth matched blood vessel image is matched with the template blood vessel image in accordance with a moving amount in the vertical, horizontal and/or rotational direction. ### Claim 15 (depends on 2) The eye motion detection method defined in claim 2, wherein in step (c) of detecting the white area from the reference image, an iris area which are not included in the white area of the eye and the center of a pupil are detected, wherein in step (f) of recording a reference position and a reference angle of the template blood vessel image in the coordinate system, a distance from the iris area to the template blood vessel image in the coordinate system and the angle from the horizontal reference line passing through the center of the pupil, wherein in step (e) of determining the template blood vessel image and in step (h) of detecting the nth matched blood vessel image, termination or detection with selection of a blood vessel image, which is found in the vicinity area of the iris area on an ear side in the white area of the eye and in the vicinity area in the lower side area of the horizontal reference line. ### Claim 16 (depends on 3) The eye motion detection method defined in claim 3, wherein in step (c) of detecting the white area from the reference image, an iris area which are not included in the white area of the eye and the center of a pupil are detected, wherein in step (f) of recording a reference position and a reference angle of the template blood vessel image in the coordinate system, a distance from the iris area to the template blood vessel image in the coordinate system and the angle from the horizontal reference line passing through the center of the pupil, wherein in step (e) of determining the template blood vessel image and in step (h) of detecting the nth matched blood vessel image, termination or detection with selection of a blood vessel image, which is found in the vicinity area of the iris area on an ear side in the white area of the eye and in the vicinity area in the lower side area of the horizontal reference line. ### Claim 17 (depends on 4) The eye motion detection method defined in claim 4, wherein in step (c) of detecting the white area from the reference image, an iris area which are not included in the white area of the eye and the center of a pupil are detected, wherein in step (f) of recording a reference position and a reference angle of the template blood vessel image in the coordinate system, a distance from the iris area to the template blood vessel image in the coordinate system and the angle from the horizontal reference line passing through the center of the pupil, wherein in step (e) of determining the template blood vessel image and in step (h) of detecting the nth matched blood vessel image, termination or detection with selection of a blood vessel image, which is found in the vicinity area of the iris area on an ear side in the white area of the eye and in the vicinity area in the lower side area of the horizontal reference line. ### Claim 18 (depends on 11) The eye motion detection method defined in claim 11, wherein in step (c) of detecting the white area from the reference image, an iris area which are not included in the white area of the eye and the center of a pupil are detected, wherein in step (f) of recording a reference position and a reference angle of the template blood vessel image in the coordinate system, a distance from the iris area to the template blood vessel image in the coordinate system and the angle from the horizontal reference line passing through the center of the pupil, wherein in step (e) of determining the template blood vessel image and in step (h) of detecting the nth matched blood vessel image, termination or detection with selection of a blood vessel image, which is found in the vicinity area of the iris area on an ear side in the white area of the eye and in the vicinity area in the lower side area of the horizontal reference line. ### Claim 19 (depends on 12) The eye motion detection method defined in claim 12, wherein in step (c) of detecting the white area from the reference image, an iris area which are not included in the white area of the eye and the center of a pupil are detected, wherein in step (f) of recording a reference position and a reference angle of the template blood vessel image in the coordinate system, a distance from the iris area to the template blood vessel image in the coordinate system and the angle from the horizontal reference line passing through the center of the pupil, wherein in step (e) of determining the template blood vessel image and in step (h) of detecting the nth matched blood vessel image, termination or detection with selection of a blood vessel image, which is found in the vicinity area of the iris area on an ear side in the white area of the eye and in the vicinity area in the lower side area of the horizontal reference line. ### Claim 20 (depends on 13) The eye motion detection method defined in claim 13, wherein in step (c) of detecting the white area from the reference image, an iris area which are not included in the white area of the eye and the center of a pupil are detected, wherein in step (f) of recording a reference position and a reference angle of the template blood vessel image in the coordinate system, a distance from the iris area to the template blood vessel image in the coordinate system and the angle from the horizontal reference line passing through the center of the pupil, wherein in step (e) of determining the template blood vessel image and in step (h) of detecting the nth matched blood vessel image, termination or detection with selection of a blood vessel image, which is found in the vicinity area of the iris area on an ear side in the white area of the eye and in the vicinity area in the lower side area of the horizontal reference line.