# US 12,112,570 B2 · Direction estimation device, direction estimation method, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,570 B2 - Document ID: US12112570B2 - Status: Granted - Kind: B2 - Application number: 17/680,829 - Filed: 2022-02-25 - Published: 2022-06-09 - Granted: 2024-10-08 - Earliest claimed: 2022-02-25 - Assignees: NEC Corporation - Applicants: NEC Corporation - Inventors: Yusuke Morishita - Web page: https://patentgrep.com/patent/US12112570B2 ## CPC - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - A61B 5/163: {by tracking eye movement, gaze, or pupil change} - G06T 7/73: using feature-based methods - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 40/168: {Feature extraction; Face representation} - G06V 40/197: {Matching; Classification} - A61B 5/18: for vehicle drivers {or machine operators} - G06T 2207/30201: Face - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An estimation device according to one aspect of the present disclosure includes: at least one memory storing a set of instructions; and at least one processor configured to execute the set of instructions to: generate a plurality of extraction regions by adding a perturbation to an extraction region of a partial image determined based on positions of feature points extracted from a face image; estimate a plurality of directions of at least one of a face and a line of sight and a reliability of each of the plurality of directions based on a plurality of partial images in the plurality of extraction regions of the face image; and calculate an integrated direction obtained by integrating the plurality of directions based on the estimated reliability. ## Claims ### Claim 1 An estimation device comprising: at least one memory storing instructions; and at least one processor configured to execute the instructions to: extract a first partial image from a face image from an extraction region in the face image, the extraction region being determined based on a position of a feature point extracted from the face image; extract a second partial image from a noised face image from the extraction region in the noised face image, the noised face image being generated by adding noise to the face image; extract a third partial image to which a perturbation is added from the face image, the perturbation includes a parallel translation of the extraction region, the third partial image being extracted from a perturbated extraction range in the face image, the perturbated extraction region being acquired by adding the perturbation to the extraction region; extract a fourth partial image to which the perturbation is added from the noised face image, the fourth partial image being extracted from the perturbated region in the noised image; estimate a plurality of directions and reliabilities of the plurality of directions from a plurality of partial images including the first partial image, the second partial image, the third partial image and the fourth partial image, the plurality of directions being a plurality of face directions or a plurality of directions of lines of sight; and calculate an integrated direction based on the reliabilities, the integrated direction being a direction into which the plurality of directions are integrated. ### Claim 2 (depends on 1) The estimation device according to claim 1, wherein the at least one processor is further configured to execute the set of instructions to: determine, based on the position of the feature point, the perturbation added to the extraction region determined based on the position of the feature point. ### Claim 3 (depends on 1) The estimation device according to claim 1, wherein the at least one processor is further configured to execute the set of instructions to: extract a face region that is a region of the face from the face image; extract the feature point from the face region; estimate a size of the face based on the position of the extracted feature point; and determine the perturbation based on the estimated size. ### Claim 4 (depends on 1) The estimation device according to claim 1, wherein the at least one processor is further configured to execute the set of instructions to: acquire an input image and extracting the face image from the input image; extract the feature point from the face image; and output the integrated direction. ### Claim 5 (depends on 4) The estimation device according to claim 4, wherein the at least one processor is further configured to execute the set of instructions to: estimate the plurality of directions of the face and the plurality of directions of the line of sight; calculate the integrated face direction obtained by integrating the plurality of directions of the face and the integrated line of sight direction obtained by integrating the plurality of directions of the line of sight; superimpose a first mark indicating the integrated face direction and a second mark indicating the integrated line of sight direction on the input image; and output the input image on which the first mark and the second mark are superimposed. ### Claim 6 An estimation method comprising: extracting a first partial image from a face image from an extraction region in the face image, the extraction region being determined based on a position of a feature point extracted from the face image; extracting a second partial image from a noised face image from the extraction region in the noised face image, the noised face image being generated by adding noise to the face image; extracting a third partial image to which a perturbation is added from the face image, the perturbation including a parallel translation of the extraction region, the third partial image being extracted from a perturbated extraction range in the face image, the perturbated extraction region being acquired by adding the perturbation to the extraction region; extracting a fourth partial image to which the perturbation is added from the noised face image, the fourth partial image being extracted from the perturbated region in the noised image; estimating a plurality of directions and reliabilities of the plurality of directions from a plurality of partial images including the first partial image, the second partial image, the third partial image and the fourth partial image, the plurality of directions being a plurality of face directions or a plurality of directions of lines of sight; and calculating an integrated direction based on the reliabilities, the integrated direction being a direction into which the plurality of directions are integrated. ### Claim 7 (depends on 6) The estimation method according to claim 6, further comprising determining, based on the position of the feature point, the perturbation added to the extraction region determined based on the position of the feature point. ### Claim 8 (depends on 6) The estimation method according to claim 6, further comprising: extracting a face region that is a region of the face from the face image; extracting the feature point from the face region; estimating a size of the face based on the position of the extracted feature point; and determining the perturbation based on the estimated size. ### Claim 9 (depends on 6) The estimation method according to claim 6, further comprising: acquiring an input image and extracting the face image from the input image; extracting the feature point from the face image; and outputting the integrated direction. ### Claim 10 (depends on 9) The estimation method according to claim 9, further comprising: estimating the plurality of directions of the face and the plurality of directions of the line of sight; calculating the integrated face direction obtained by integrating the plurality of directions of the face and the integrated line of sight direction obtained by integrating the plurality of directions of the line of sight; superimposing a first mark indicating the integrated face direction and a second mark indicating the integrated line of sight direction on the input image; and outputting the input image on which the first mark and the second mark are superimposed. ### Claim 11 A non-transitory computer readable storage medium storing a program causing a computer to execute processing of: extracting a first partial image from a face image from an extraction region in the face image, the extraction region being determined based on a position of a feature point extracted from the face image; extracting a second partial image from a noised face image from the extraction region in the noised face image, the noised face image being generated by adding noise to the face image; extracting a third partial image to which a perturbation is added from the face image, the perturbation including a parallel translation of the extraction region, the third partial image being extracted from a perturbated extraction range in the face image, the perturbated extraction region being acquired by adding the perturbation to the extraction region; extracting a fourth partial image to which the perturbation is added from the noised face image, the fourth partial image being extracted from the perturbated region in the noised image; estimating a plurality of directions and reliabilities of the plurality of directions from a plurality of partial images including the first partial image, the second partial image, the third partial image and the fourth partial image, the plurality of directions being a plurality of face directions or a plurality of directions of lines of sight; and calculating an integrated direction based on the reliabilities, the integrated direction being a direction into which the plurality of directions are integrated. ### Claim 12 (depends on 11) The storage medium according to claim 11, wherein the program further causes a computer to execute processing of determining, based on the position of the feature point, the perturbation added to the extraction region determined based on the position of the feature point. ### Claim 13 (depends on 4) The storage medium according to claim 4, wherein the program further causes a computer to execute processing of: extracting a face region that is a region of the face from the face extracting the feature point from the face region; estimating a size of the face based on the position of the extracted feature point; and determining the perturbation based on the estimated size. ### Claim 14 (depends on 11) The storage medium according to claim 11, wherein the program further causes a computer to execute processing of: acquiring an input image and extracting the face image from the input image; extracting the feature point from the face image; and outputting the integrated direction. ### Claim 15 (depends on 1) The estimation device according to claim 1, wherein the perturbation further includes enlargement of an extracted partial image, the extracted partial image being the third partial image and the fourth partial image. ### Claim 16 (depends on 1) The estimation device according to claim 1, wherein the perturbation further includes a rotation of an extracted partial image, the extracted partial image being the third partial image and the fourth partial image. ### Claim 17 (depends on 1) The estimation device according to claim 1, wherein the perturbation further includes smoothing the face image. ### Claim 18 (depends on 1) The estimation device according to claim 1, wherein the at least one processor configured to execute the instructions to: evaluate an image quality of the face image; and change a magnitude of the perturbation based on the image quality. ### Claim 19 (depends on 1) The estimation device according to claim 1, wherein the at least one processor configured to execute the instructions to: calculate a detection score of the face image; and change a magnitude of the perturbation based on the detection score.