# US 11,030,294 B2 · Computerized systems and methods for determining authenticity using micro expressions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,030,294 B2 - Document ID: US11030294B2 - Status: Granted - Kind: B2 - Application number: 16/536,181 - Filed: 2019-08-08 - Published: 2020-10-15 - Granted: 2021-06-08 - Earliest claimed: 2019-04-12 - Assignees: Coupang Corp. - Applicants: Coupang, Corp. - Inventors: Xiaojun Huang - Web page: https://patentgrep.com/patent/US11030294B2 ## CPC - G06F 21/32: using biometric data, e.g. fingerprints, iris scans or voiceprints - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/063: using electronic means - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/048: Activation functions - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems and methods are provided for calculating authenticity of a human user. One method comprises receiving, via a network, an electronic request from a user device, instantiating a video connection with the user device; generating, using a database of questions, a first question; providing, via the network, the generated question to the user device; analyzing video and audio data received via the connection to extract facial expressions, calculating, using convolutional neural networks, first data and second data corresponding predetermined emotions based on facial expressions and audio data; generating candidate emotion data using the first and second data; determining whether the candidate emotion data predicts a predetermined emotion, and generating a second question to collect additional data for aggregating with the first and second data or determining the authenticity of the user and using the determined authenticity to decide on the user request. ## Claims ### Claim 1 A method for calculating authenticity of a human user, performed by a device having at least one processor, the method comprising: receiving, via a network, a request, the request instantiating a connection with a user device; providing, via the network, a first question to the user device; extracting a plurality of facial expressions from data received via the connection; calculating, using a first convolutional neural network, first data corresponding to one or more predetermined emotions based on at least one extracted facial expression, and using a second convolutional neural network, second data corresponding to the one or more predetermined emotions based on at least two extracted facial expressions and the received data; generating candidate emotion data using the first and second data; determining whether the candidate emotion data predicts a predetermined emotion; and based on determining whether the candidate emotion data predicts the predetermined emotion: generating a second question to collect additional data for aggregating with the first and second data; or determining the authenticity of the human user and using the determined authenticity to decide on the request. ### Claim 2 (depends on 1) The method of claim 1, wherein the first convolutional neural network comprises a spatial convolutional neural network and the second convolutional neural network comprises a temporal convolutional neural network. ### Claim 3 (depends on 2) The method of claim 2, further comprising training at least one of the convolutional neural networks using a database of resources, the database of resources comprising face data labeled to establish ground truth of the one or more pre-determined emotions. ### Claim 4 (depends on 3) The method of claim 3, further comprising: normalizing the database of resources using time interpolation; and training the temporal convolutional neural network using the normalized database of resources. ### Claim 5 (depends on 2) The method of claim 2, wherein the temporal convolutional neural network analyzes the received data. ### Claim 6 (depends on 1) The method of claim 1, wherein the two convolutional neural networks analyze the human user's micro expressions. ### Claim 7 (depends on 1) The method of claim 1, wherein the one or more predetermined emotions comprises surprise, disgust, or repression. ### Claim 8 (depends on 1) The method of claim 1, wherein generating the second question is based on a received answer to at least the first question. ### Claim 9 (depends on 1) The method of claim 1, wherein the human user is a loan applicant. ### Claim 10 A computing system for calculating authenticity of a human user, comprising: at least one processor; at least one memory storing instructions, wherein the instructions cause the at least one processor to: receive, via a network, a request, the request instantiating a connection with a user device; provide, via the network, a first question to the user device; extract a plurality of facial expressions from data received via the connection; calculate, using a first convolutional neural network, first data corresponding to one or more predetermined emotions based on at least one extracted facial expression, and using a second convolutional neural network, second data corresponding to the one or more predetermined emotions based on at least two extracted facial expressions and the received data; generate candidate emotion data using the first and second data; determine whether the candidate emotion data predicts a predetermined emotion; and based on determining whether the candidate emotion data predicts the predetermined emotion: generate a second question to collect additional data for aggregating with the first and second data; or determine the authenticity of the human user and using the determined authenticity to decide on the request. ### Claim 11 (depends on 10) The system of claim 10, wherein the first convolutional neural network comprises a spatial convolutional neural network and the second convolutional neural network comprises a temporal convolutional neural network. ### Claim 12 (depends on 11) The system of claim 11, wherein the instructions further cause the at least one processor to train at least one of the convolutional neural networks using a database of resources, the database of resources comprising face data labeled to establish ground truth of the one or more pre-determined emotions. ### Claim 13 (depends on 12) The system of claim 12, wherein the instructions further cause the at least one processor to: normalize the database of resources using time interpolation; and train the temporal convolutional neural network using the normalized database of resources. ### Claim 14 (depends on 11) The system of claim 11, wherein the temporal convolutional neural network analyzes the received data. ### Claim 15 (depends on 10) The system of claim 10, wherein the two convolutional neural networks analyze the human user's micro expressions. ### Claim 16 (depends on 10) The system of claim 10, wherein the one or more predetermined emotions comprises surprise, disgust, or repression. ### Claim 17 (depends on 10) The system of claim 10, wherein generating the second question is based on a received answer to at least the first question. ### Claim 18 (depends on 10) The system of claim 10, wherein the human user is a loan applicant. ### Claim 19 (depends on 10) The system of claim 10, wherein the user device provides at least seven questions to the human user. ### Claim 20 A method for determining authenticity of a loan applicant, performed by a user device having at least one processor, the method comprising: receiving, via a network, a request, the request instantiating a connection with a user device; providing, via the network, a first question to the user device; extracting a plurality of facial expressions from data received via the connection; calculating, using a first convolutional neural network, first data corresponding to one or more predetermined emotions based on at least one extracted facial expression, and using a second convolutional neural network, second data corresponding to the one or more predetermined emotions based on at least two extracted facial expressions and the received data; generating candidate emotion data using the first and second data; determining whether the candidate emotion data predicts a predetermined emotion; and based on determining whether the candidate emotion data predicts the predetermined emotion: generating a second question to collect additional data for aggregating with the first and second data; or determining the authenticity of the human user and using the determined authenticity to decide on the request.