# US 12,002,295 B2 · System and method for video authentication

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,295 B2
- Document ID: US12002295B2
- Status: Granted
- Kind: B2
- Application number: 17/401,609
- Filed: 2021-08-13
- Published: 2023-02-23
- Granted: 2024-06-04
- Earliest claimed: 2021-08-13
- Assignees: Accenture Global Solutions Limited
- Applicants: Accenture Global Solutions Limited
- Inventors: Ankit Suneja; Rajeev Divakaran Nair; S. Abishek Kumar
- Web page: https://patentgrep.com/patent/US12002295B2

## CPC

- G06V 40/45: {Detection of the body part being alive}
- G06F 18/253: {of extracted features}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 20/00: Machine learning
- G06Q 30/0185: {Product, service or business identity fraud}
- G06Q 30/0609: {Qualifying participants for shopping transactions (payment transaction verification G06Q20/401)}
- G06Q 50/265: {Personal security, identity or safety}
- G06V 40/172: {Classification, e.g. identification}
- G06V 40/70: Multimodal biometrics, e.g. combining information from different biometric modalities
- G10L 17/06: Decision making techniques; Pattern matching strategies
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06V 40/178: {estimating age from face image; using age information for improving recognition}
- G10L 25/48: specially adapted for particular use
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A system and method for video authentication may apply machine learning to analyze whether a person's face captured by live video matches a face in a photo ID captured by live video and to analyze other features based on a video session with the person. For example, machine learning may be applied to analyze a set of features indicating whether the person is a real, live person (as opposed to a photo image held up over the person's face in the video, etc.). Finally, the machine learning may be applied to analyze a set of features to determine whether a lower probability prediction that the person's face captured by live video matches a face in a photo ID captured by live video should be either pass authentication (due to one or more features/circumstances mitigating the lower probability) or fail authentication (due to one or more features not mitigating the lower probability). In such a situation, the set of features may indicate that mitigating factors/conditions exist that can offset the lower probability.

## Claims

### Claim 1

A computer implemented method of applying machine learning to authenticate a customer's identity via live video, comprising:
building convolutional neural network (“CNN”) models for at least two of a sentiment module, a face identity document match (“face ID match”), a liveness module, a voice module, and a politically exposed person (“PEP”) module on at least two different computation nodes by:
receiving, at a model building planner, module data;
creating, by the model building planner, dispatchable transaction packages at a module data building pool;
sharing the dispatchable transaction packages with the at least two computation nodes; and
dynamically coordinating, by a model building scheduler, training progress among the at least two computation nodes;
receiving captured live video comprising a person's voice and images including a person's face and an image of a photo identity document (“photo ID”);
processing the images through the sentiment module to generate a sentiment score based on the person's face as it appears in the live video;
processing the images through the face ID match module to generate a face ID score based on the person's face as it appears in the live video;
processing the images through the liveness module to generate a liveness score based on the person's face as it appears in the live video;
processing the person's voice through the voice module to generate a voice score based on the person's voice as it sounds in the live video;
processing the images and the person's voice through the PEP module to generate a PEP score based on the person's face and the person's voice; and
in response to determining that one or both of the face ID score and the liveness score is “fail,” determining whether a condition offsets the failing face ID score and/or the failing liveness score;
processing the images through a machine learning model to determine, based on the photo ID, whether the person's age is above a predetermined threshold, wherein the person's age is determined to be above the predetermined threshold and the person's age is the condition; and
in response to determining that PEP score is “pass,” disallowing an offset based on whether a condition offsets the score of “fail.”

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
in response to determining that a condition offsets the failing face ID score and/or the failing liveness score, verifying that the customer's identity passes authentication.

### Claim 3 (depends on 1)

The method of claim 1, wherein the sentiment score is the condition.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
in response to determining that a condition does not offset the failing face ID score and/or the failing liveness score, verifying that the customer's identity fails authentication.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
pre-training at least one of the CNN models of the sentiment module, the face ID match module, the liveness module, the voice module, and the PEP module by a heuristic unsupervised pre-training process including:
starting a training session;
initializing neuron weights with random values;
pre-training the CNN model unsupervised;
training the CNN model supervised;
validating the CNN model;
testing performance of the CNN model; and
completing the training session.

### Claim 6 (depends on 1)

The method of claim 1, wherein processing the images through at least one of the sentiment module, the face ID match module, the liveness module, and the PEP module includes extracting features from the images and wherein the method further comprises:
fusing user moment information with the features extracted from the images when processing the images through at least one of the sentiment module, the face ID match module, the liveness module, and the PEP module.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
simultaneously building the at least two of the sentiment module, the face ID match module, the liveness module, the voice module, and the PEP module on the at least two different computation nodes.

### Claim 8 (depends on 1)

The method of claim 1, further comprising monitoring and controlling local processes of the at least two computation nodes by local managers on the at least two computation nodes.

### Claim 9 (depends on 1)

The method of claim 1, further comprising creating an auxiliary decoder neuron layer for each neuron of the respective CNN model.

### Claim 10

A system for applying machine learning to authenticate a customer's identity via live video, comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to:
build convolutional neural network (“CNN”) models for at least two of a sentiment module, a face identity document match (“face ID match”), a liveness module, a voice module, and a politically exposed person (“PEP”) module on at least two different computation nodes by:
receiving, at a model building planner, module data;
creating, by the model building planner, dispatchable transaction packages at a module data building pool;
sharing the dispatchable transaction packages with the at least two computation nodes; and
dynamically coordinating, by a model building scheduler, training progress among the at least two computation nodes;
receive captured live video comprising a person's voice and images including a person's face and an image of a photo identity document (“photo ID”);
process the images through the sentiment module to generate a sentiment score based on the person's face as it appears in the live video;
process the images through the face ID match module to generate a face ID score based on the person's face as it appears in the live video;
process the images through the liveness module to generate a liveness score based on the person's face as it appears in the live video;
process the person's voice through the voice module to generate a voice score based on the person's voice as it sounds in the live video;
process the images and the person's voice through the PEP module to generate a PEP score based on the person's face and the person's voice; and
in response to determining that one or both of the face ID score and the liveness score is “fail,” determine whether a condition offsets the failing face ID score and/or the failing liveness score;
process the images through a machine learning model to determine, based on the photo ID, whether the person's age is above a predetermined threshold, wherein the person's age is determined to be above the predetermined threshold and the person's age is the condition; and
in response to determining that PEP score is “pass,” disallow an offset based on whether a condition offsets the score of “fail.”

### Claim 11 (depends on 10)

The system of claim 10, further comprising:
in response to determining that a condition offsets the failing face ID score and/or the failing liveness score, verify that the customer's identity passes authentication.

### Claim 12 (depends on 10)

The system of claim 10, wherein causing the one or more computers to apply machine learning to authenticate a customer's identity via live video further includes:
simultaneously building the at least two of the sentiment module, the face ID match module, the liveness module, the voice module, and the PEP module on the at least two different computation nodes.

### Claim 13

A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to apply machine learning to authenticate a customer's identity via live video by:
building convolutional neural network (“CNN”) models for at least two of a sentiment module, a face identity document match (“face ID match”), a liveness module, a voice module, and a politically exposed person (“PEP”) module on at least two different computation nodes by:
receiving, at a model building planner, module data;
creating, by the model building planner, dispatchable transaction packages at a module data building pool;
sharing the dispatchable transaction packages with the at least two computation nodes; and
dynamically coordinating, by a model building scheduler, training progress among the at least two computation nodes;
receiving captured live video comprising a person's voice and images including a person's face and an image of a photo identity document (“photo ID”);
processing the images through the sentiment module to generate a sentiment score based on the person's face as it appears in the live video;
processing the images through the face ID match module to generate a face ID score based on the person's face as it appears in the live video;
processing the images through the liveness module to generate a liveness score based on the person's face as it appears in the live video;
processing the person's voice through the voice module to generate a voice score based on the person's voice as it sounds in the live video;
processing the images and the person's voice through the PEP module to generate a PEP score based on the person's face and the person's voice; and
in response to determining that one or both of the face ID score and the liveness score is “fail,” determining whether a condition offsets the failing face ID score and/or the failing liveness score;
processing the images through a machine learning model to determine, based on the photo ID, whether the person's age is above a predetermined threshold, wherein the person's age is determined to be above the predetermined threshold and the person's age is the condition; and
in response to determining that PEP score is “pass,” disallowing an offset based on whether a condition offsets the score of “fail.”

### Claim 14 (depends on 13)

The non-transitory computer-readable medium of claim 13, wherein causing the one or more computers to apply machine learning to authenticate a customer's identity via live video further includes:
in response to determining that a condition offsets the failing face ID score and/or the failing liveness score, verifying that the customer's identity passes authentication.

### Claim 15 (depends on 13)

The non-transitory computer-readable medium of claim 13, wherein causing the one or more computers to apply machine learning to authenticate a customer's identity via live video further includes:
simultaneously building the at least two of the sentiment module, the face ID match module, the liveness module, the voice module, and the PEP module on the at least two different computation nodes.

### Claim 16 (depends on 13)

The non-transitory computer-readable medium of claim 13, wherein processing the images through at least one of the sentiment module, the face ID match module, the liveness module, and the PEP module includes extracting features from the images and wherein causing the one or more computers to apply machine learning to authenticate a customer's identity via live video further includes:
fusing user moment information with the features extracted from the images when processing the images through at least one of the sentiment module, the face ID match module, the liveness module, and the PEP module.