# US 10,769,635 B2 · Authentication techniques including speech and/or lip movement analysis > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,635 B2 - Document ID: US10769635B2 - Status: Granted - Kind: B2 - Application number: 15/229,233 - Filed: 2016-08-05 - Published: 2018-02-08 - Granted: 2020-09-08 - Earliest claimed: 2016-08-05 - Assignees: Nok Nok Labs, Inc. - Applicants: Nok Nok Labs, Inc. - Inventors: Rolf Lindemann - Web page: https://patentgrep.com/patent/US10769635B2 ## CPC - G06Q 20/40145: {Biometric identity checks} - G06F 21/31: User authentication - G06F 21/32: using biometric data, e.g. fingerprints, iris scans or voiceprints - G06F 21/44: Program or device authentication - G06Q 20/3224: {Transactions dependent on location of M-devices} - G06Q 20/4014: {Identity check for transactions} - G06V 20/80: Recognising image objects characterised by unique random patterns - G06V 40/167: {using comparisons between temporally consecutive images} - G06V 40/19: Sensors therefor - G06V 40/70: Multimodal biometrics, e.g. combining information from different biometric modalities - G10L 15/25: using position of the lips, movement of the lips or face analysis - G10L 17/04: Training, enrolment or model building - G10L 17/06: Decision making techniques; Pattern matching strategies - H04L 63/0861: {using biometrical features, e.g. fingerprint, retina-scan (cryptographic mechanisms or cryptographic arrangements for entity authentication using biological data H04L9/3231)} - G06F 2221/2111: Location-sensitive, e.g. geographical location, GPS - G06F 2221/2113: Multi-level security, e.g. mandatory access control - G10L 17/00: Speaker identification or verification techniques - G10L 17/24: the user being prompted to utter a password or a predefined phrase - H04L 63/107: {wherein the security policies are location-dependent, e.g. entities privileges depend on current location or allowing specific operations only from locally connected terminals} - H04L 63/1433: {Vulnerability analysis} - H04L 2463/102: applying security measure for e-commerce - H04W 12/06: Authentication - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract A system, apparatus, method, and machine readable medium are described for performing eye tracking during authentication. For example, one embodiment of a method comprises: receiving a request to authenticate a user; presenting one or more screen layouts to the user; capturing a sequence of images which include the user's eyes as the one or more screen layouts are displayed; and (a) performing eye movement detection across the sequence of images to identify a correlation between motion of the user's eyes as the one or more screen layouts are presented and an expected motion of the user's eyes as the one or more screen layouts are presented and/or (b) measuring the eye's pupil size to identify a correlation between the effective light intensity of the screen and its effect on the user's eye pupil size; capturing audio of the user's voice; and performing voice recognition techniques to determine a correlation between the captured audio of the user's voice and one or more voice prints. ## Claims ### Claim 1 A method comprising: receiving a request to authenticate a user; presenting one or more screen layouts to the user; capturing a sequence of images which include the user's eyes as the one or more screen layouts are displayed; performing eye movement detection across the sequence of images to identify a correlation between motion of the user's eyes as the one or more screen layouts are presented and an expected motion of the user's eyes as the one or more screen layouts are presented; capturing audio of the user's voice; performing voice recognition techniques to determine a correlation between the captured audio of the user's voice and one or more voice prints; training a lip movement analysis module to associate particular phonetics and volume levels with particular lip or mouth positions and/or movements over time; performing lip-movement analysis using the trained lip movement analysis module to determine a correlation between the audio of the user's voice and detected motion of the user's lips across the sequence of images; detecting motion of the user's lips across the sequence of images; correlating the lip motion with a reference lip motion captured during an enrollment of the user; and generating an assurance level based on (a) a correlation between the images of the user's face and facial template data associated with the user, (b) a correlation between the motion of the user's eyes and an expected motion of the user's eyes as the one or more screen layouts are presented, (c) a correlation between the captured audio of the user's voice and the one or more voice prints, and (d) a correlation between the audio of the user's voice and detected motion of the user's lips across the sequence of images. ### Claim 2 (depends on 1) The method as in claim 1 further comprising: performing facial recognition to identify the correlation between one or more images of the user's face and facial template data associated with the user. ### Claim 3 (depends on 2) The method as in claim 2 wherein presenting the one or more screen layouts comprises displaying one or more graphics images, photographs, or motion video images in designated regions of the display, wherein the expected motion characteristic of the user's eyes depends on the contents, the size, the appearance, the context and the region where these elements are shown. ### Claim 4 (depends on 3) The method as in claim 3 wherein facial recognition is performed first, resulting in a first correlation and eye movement detection is performed only if the facial recognition results in a correlation above a specified threshold. ### Claim 5 (depends on 3) The method as in claim 3 wherein the designated regions of the display for displaying the graphics images, photographs, or motion video images are selected randomly. ### Claim 6 (depends on 3) The method as in claim 3 wherein presenting the one or more screen layouts comprises displaying a plurality of graphics images, photographs, or motion video images in a specified sequence using a specified timing. ### Claim 7 (depends on 1) The method as in claim 1 further comprising: determining whether the assurance level is sufficient to authenticate the user for a particular transaction; and permitting the transaction if the assurance level is sufficient. ### Claim 8 (depends on 7) The method as in claim 7 wherein the transaction is an online transaction with a remote relying party. ### Claim 9 (depends on 7) The method as in claim 7 further comprising: if the assurance level is not sufficient by itself to authenticate the user for a particular transaction, then combining the assurance level with authentication data generated using one or more additional authentication techniques to authenticate the user for the particular transaction. ### Claim 10 (depends on 9) The method as in claim 9 wherein one of the additional authentication techniques comprises performing an explicit authentication in which the end user is required to enter a password or PIN or to swipe a finger on a fingerprint authentication device. ### Claim 11 (depends on 9) The method as in claim 9 wherein one of the additional authentication techniques comprises one or more non-intrusive authentication techniques in which sensor data collected from one or more device sensors is used for authentication. ### Claim 12 (depends on 11) The method as in claim 11 wherein the sensor data comprises location data. ### Claim 13 (depends on 11) The method as in claim 11 wherein one of the non-intrusive techniques comprises determining elapsed time from the last explicit user authentication and responsively increasing the assurance level of explicit user authentication has occurred within a specified time period. ### Claim 14 (depends on 1) The method as in claim 1 wherein the expected motion of the user's eyes is based on learned characteristics of eye-movement triggered by different types of images. ### Claim 15 (depends on 1) The method as in claim 1 wherein capturing audio of the user's voice comprises prompting the user to speak one or more words and/or phrases and receiving responses from the user via a microphone. ### Claim 16 An apparatus comprising: an authentication engine receiving a request to authenticate a user; a camera capturing a sequence of images which include the user's eyes as one or more screen layouts are displayed; an eye tracking module performing eye movement detection across the sequence of images to identify a correlation between motion of the user's eyes as the one or more screen layouts are presented and an expected motion of the user's eyes as the one or more screen layouts are presented; a voice recognition module to perform voice recognition techniques to determine a correlation between captured audio of the user's voice and one or more voice prints; and a lip movement analysis module, trained to associate particular phonetics and volume levels with particular lip or mouth positions and/or movements over time, to perform lip-movement analysis to determine a correlation between the audio of the user's voice and detected motion of the user's lips across the sequence of images, the lip movement analysis module further configured to detect motion of the user's lips across a sequence of images and correlate the lip motion with a reference lip motion captured during an enrollment of the user; and the authentication engine generating an assurance level based on (a) a correlation between the images of the user's face and facial template data associated with the user, (b) the correlation between the motion of the user's eyes and an expected motion of the user's eyes as the one or more screen layouts are presented; and (c) the correlation between the captured audio of the user's voice and the one or more voice prints, and (d) the correlation between the audio of the user's voice and detected motion of the user's lips across the sequence of images. ### Claim 17 (depends on 16) The apparatus as in claim 16 further comprising: a facial recognition engine to perform facial recognition to identify the correlation between one or more images of the user's face and facial template data associated with the user. ### Claim 18 (depends on 17) The apparatus as in claim 17 wherein presenting the one or more screen layouts comprises displaying one or more graphics images, photographs, or motion video images in designated regions of the display, wherein the expected motion of the user's eyes is in the direction of the graphics images, photographs, or motion video images to different degrees. ### Claim 19 (depends on 18) The apparatus as in claim 18 wherein facial recognition is performed first, resulting in a first correlation and eye movement detection is performed only if the facial recognition results in a correlation above a specified threshold. ### Claim 20 (depends on 18) The apparatus as in claim 18 wherein the designated regions of the display for displaying the graphics images, photographs, or motion video images are selected randomly. ### Claim 21 (depends on 18) The apparatus as in claim 18 wherein presenting the one or more screen layouts comprises displaying a plurality of graphics images, photographs, or motion video images in a specified sequence using a specified timing. ### Claim 22 (depends on 21) The apparatus as in claim 21 wherein the transaction is an online transaction with a remote relying party. ### Claim 23 (depends on 16) The apparatus as in claim 16 further comprising: the authentication engine determining whether the assurance level is sufficient to authenticate the user for a particular transaction; and permitting the transaction if the assurance level is sufficient. ### Claim 24 (depends on 23) The apparatus as in claim 23 further comprising: the authentication engine combining the assurance level with authentication data generated using one or more additional authentication techniques to authenticate the user for the particular transaction if the assurance level is not sufficient by itself to authenticate the user for a particular transaction. ### Claim 25 (depends on 23) The apparatus as in claim 23 wherein one of the additional authentication techniques comprises one or more non-intrusive authentication techniques in which sensor data collected from one or more device sensors is used for authentication. ### Claim 26 (depends on 25) The apparatus as in claim 25 wherein one of the additional authentication techniques comprises performing an explicit authentication in which the end user is required to enter a password or PIN or to swipe a finger on a fingerprint authentication device. ### Claim 27 (depends on 25) The apparatus as in claim 25 wherein the sensor data comprises location data. ### Claim 28 (depends on 25) The apparatus as in claim 25 wherein one of the non-intrusive techniques comprises determining elapsed time from the last explicit user authentication and responsively increasing the assurance level of explicit user authentication has occurred within a specified time period. ### Claim 29 (depends on 16) The apparatus as in claim 16 wherein the expected motion of the user's eyes is based on learned characteristics of eye-movement triggered by different types of images. ### Claim 30 (depends on 16) The apparatus as in claim 16 wherein capturing audio of the user's voice comprises prompting the user to speak one or more words and/or phrases and receiving responses from the user via a microphone.