# US 9,521,325 B2 · Terminal device and line of sight detection method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,521,325 B2 - Document ID: US9521325B2 - Status: Granted - Kind: B2 - Application number: 14/284,937 - Filed: 2014-05-22 - Published: 2014-12-18 - Granted: 2016-12-13 - Earliest claimed: 2013-06-14 - Assignees: Fujitsu Limited - Applicants: Fujitsu Limited - Inventors: Shanshan Yu; Satoshi Nakashima - Web page: https://patentgrep.com/patent/US9521325B2 ## CPC - H04N 23/68: for stable pick-up of the scene, e.g. compensating for camera body vibrations - A61B 3/113: for determining or recording eye movement - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06V 40/19: Sensors therefor - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A terminal device includes, a processor; and a memory which stores a plurality of instructions, which when executed by the processor, cause the processor to execute, obtaining information that indicates a state of an imaging unit when photographing a subject; determining whether an image photographed by the imaging unit includes blurriness due to shaking of the imaging unit based on the obtained information; and using an image determined as not including blurriness in the determining to detect a line of sight of the photographed subject. ## Claims ### Claim 1 A terminal device comprising: a processor; and a memory which stores a plurality of instructions, which when executed by the processor, cause the processor to execute, obtaining information that indicates each state of an imaging unit in each timing of photographing a subject; determining whether each image photographed by the imaging unit includes blurriness based on the obtained information; detecting a region including an eye in a first image and detecting a line of sight of the photographed subject in the detected region when the first image is determined as not including blurriness; when a second image photographed after the first image is determined as including blurriness, detecting a region including an eye in the second image based on a first corresponding region in an image taken before the second image, the first corresponding region corresponding to the detected region in the first image; and when a third image photographed after the first image is determined as not including blurriness, detecting a region including an eye in the third image based on a second corresponding region in an image taken before the third image and detecting a line of sight of the photographed subject in the detected region based on the detected region in the third image, the second corresponding region corresponding to the detected region in the first image. ### Claim 2 (depends on 1) The device according to claim 1, wherein the obtaining involves obtaining information indicating a state of the terminal device as the information indicating the each state of the imaging unit; and wherein the determining involves determining that the each image photographed by the imaging unit includes blurriness if a magnitude of a change of the indicated state in the information that indicates the state is equal to or greater than a certain value. ### Claim 3 (depends on 1) The device according to claim 1, further comprising specifying the position of the subject in a third image based on the position of the subject in the first image when the third image photographed at a time after the first image is determined as not including blurriness. ### Claim 4 (depends on 1) A terminal device according to claim 1, further comprising detecting a region including an eye in a first image without using the each image photographed before the first image. ### Claim 5 A line of sight detection method comprising: obtaining information that indicates each state of an imaging unit in each timing of photographing a subject; determining, by a computer processor, whether each image photographed by the imaging unit includes blurriness based on the obtained information; detecting a region including an eye in a first image and detecting a line of sight of the photographed subject in the detected region when the first image is determined as not including blurriness; when a second image photographed after the first image is determined as including blurriness, detecting a region including an eye in the second image based on a first corresponding region in an image taken before the second image, the first corresponding region corresponding to the detected region in the first image; and when a third image photographed after the first image is determined as not including blurriness, detecting a region including an eye in the third image based on a second corresponding region in an image taken before the third image and detecting a line of sight of the photographed subject in the detected region based on the detected region in the third image, the second corresponding region corresponding to the detected region in the first image. ### Claim 6 (depends on 5) The method according to claim 5, wherein the obtaining involves obtaining information indicating a state of the terminal device as the information indicating the each state of the imaging unit; and wherein the determining involves determining that the each image photographed by the imaging unit includes blurriness if a magnitude of a change of the indicated state in the information that indicates the state is equal to or greater than a certain value. ### Claim 7 (depends on 5) The method according to claim 5, wherein the obtaining involves associating pixels included in a first image and a second image photographed before the first image among a plurality of images generated by the imaging unit continuously photographing the subject, and obtains a magnitude of an image region in which are dispersed pixels for which a difference between extracted feature quantities of the first image and extracted feature quantities of the second image is equal to or greater than a certain threshold, as the information indicating the each state of the imaging unit; and wherein the determining involves determining that the first image includes blurriness if the magnitude of the image region is equal to or greater than a certain value. ### Claim 8 A non-transitory computer-readable storage medium storing a line of sight detection program that causes a computer to execute a process comprising: obtaining information that indicates each state of an imaging unit in each timing of photographing a subject; determining whether each image photographed by the imaging unit includes blurriness based on the obtained information; detecting a region including an eye in a first image and detecting a line of sight of the photographed subject in the detected region when the first image is determined as not including blurriness; when a second image photographed after the first image is determined as including blurriness, detecting a region including an eye in the second image based on a first corresponding region in an image taken before the second image, the first corresponding region corresponding to the detected region in the first image; and when a third image photographed after the first image is determined as not including blurriness, detecting a region including an eye in the third image based on a second corresponding region in an image taken before the third image and detecting a line of sight of the photographed subject in the detected region based on the detected region in the third image, the second corresponding region corresponding to the detected region in the first image. ### Claim 9 (depends on 8) The non-transitory computer-readable storage medium according to claim 8, wherein the obtaining involves obtaining information indicating a state of the terminal device as the information indicating the each state of the imaging unit; and wherein the determining involves determining that the each image photographed by the imaging unit includes blurriness if a magnitude of a change of the indicated state in the information that indicates the state is equal to or greater than a certain value. ### Claim 10 (depends on 8) The non-transitory computer-readable storage medium according to claim 8, wherein the obtaining involves associating pixels included in a first image and a second image photographed before the first image among a plurality of images generated by the imaging unit continuously photographing the subject, and obtains a magnitude of an image region in which are dispersed pixels for which a difference between extracted feature quantities of the first image and extracted feature quantities of the second image is equal to or greater than a certain threshold, as the information indicating the each state of the imaging unit; and wherein the determining involves determining that the first image includes blurriness if the magnitude of the image region is equal to or greater than a certain value.