# US 11,443,559 B2 · Facial liveness detection with a mobile device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,559 B2
- Document ID: US11443559B2
- Status: Granted
- Kind: B2
- Application number: 17/006,256
- Filed: 2020-08-28
- Published: 2021-03-04
- Granted: 2022-09-13
- Earliest claimed: 2019-08-29
- Assignees: PXL Vision AG
- Applicants: PXL Vision AG
- Inventors: Mikhail Vorobiev; Nevena Shamoska; Magdalena Polac; Benjamin Fankhauser; Michael Goettlicher; Marcus Hudritsch; Stamatios Georgoulis; Suman Saha; Luc van Gool
- Web page: https://patentgrep.com/patent/US11443559B2

## CPC

- G06F 21/32: using biometric data, e.g. fingerprints, iris scans or voiceprints
- B42D 25/23: Identity cards
- B42D 25/328: Diffraction gratings; Holograms
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 21/34: involving the use of external additional devices, e.g. dongles or smart cards
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06V 10/225: {based on a marking or identifier characterising the area}
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06V 40/168: {Feature extraction; Face representation}
- G06V 40/172: {Classification, e.g. identification}
- G06V 40/45: {Detection of the body part being alive}
- G06V 40/67: {by interactive indications to the user}
- G07D 7/0032: {using holograms}
- G07D 7/12: Visible light, infrared or ultraviolet radiation
- G07D 7/206: Matching template patterns
- G06V 10/16: {using multiple overlapping images; Image stitching}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- B42D: BOOKS; BOOK COVERS; LOOSE LEAVES; PRINTED MATTER CHARACTERISED BY IDENTIFICATION OR SECURITY FEATURES; PRINTED MATTER OF SPECIAL FORMAT OR STYLE NOT OTHERWISE PROVIDED FOR; DEVICES FOR USE THEREWITH AND NOT OTHERWISE PROVIDED FOR; MOVABLE-STRIP WRITING OR READING APPARATUS
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07D: HANDLING OF COINS OR VALUABLE PAPERS, e.g. TESTING, SORTING BY DENOMINATIONS, COUNTING, DISPENSING, CHANGING OR DEPOSITING

## Abstract

A system for remote identification of users. The system uses deep learning techniques for authenticating a user from an identification document, using automated verification of identification documents and detection that a live person identified by the document is present. Liveness of a user indicated by the identification document may be determined with a deep learning model trained for identification of facial spoofing attacks. The deep learning model may be trained using training data extracted from facial feature locations of training images.

## Claims

### Claim 1

A non-transitory computer-readable medium comprising computer-executable instructions which, when executed by a computing device, cause the computing device to carry out a method, the method comprising:
using at least one processor to perform:
accessing a plurality of images comprising a face obtained by a camera;
providing the plurality of images to a trained deep learning model to obtain output indicating one or more likelihoods that the plurality of images comprise images of a live user and one or more likelihoods that the plurality of images comprise images of a spoof attack; and
identifying the plurality of images as comprising at least one of a live user and a spoof attack based on the output obtained from the trained deep learning model;
wherein the trained deep learning model comprises at least one convolutional neural network, the at least one convolutional neural network being trained using at least one feedback network configured to improve performance of the at least one convolutional neural network in classifying images from multiple domains at least in part by generating, during training, one or more domain classifier scores representing one or more probabilities of an input training data example belonging to one or more domains.

### Claim 2 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the trained deep learning model is trained based on any one of:
training data comprising facial feature locations extracted from training images; and
feedback from the at least one feedback network.

### Claim 3 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the trained deep learning model is configured to identify spoof attacks in images from the multiple domains, the multiple domains including pre-recorded videos comprising a face, still images comprising a face, and live users wearing a mask.

### Claim 4 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein during training, the at least one convolutional neural network is trained using the one or more domain classifier scores during backpropagation.

### Claim 5 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the camera and the trained deep learning model are disposed in a user device.

### Claim 6 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the at least one convolutional neural network comprises a residual network.

### Claim 7

A computing system comprising:
a camera;
a server;
at least one processor; and
at least one non-transitory computer-readable medium comprising instructions which, when executed by the at least one processor, cause the computing system to perform a method of:
using the at least one processor to perform:
accessing, from the server, a plurality of images comprising a face obtained by the camera;
providing the plurality of images to a trained deep learning model to obtain output indicating one or more likelihoods that the plurality of images comprise images of a live user and one or more likelihoods that the plurality of images comprise images of a spoof attack; and
identifying the plurality of images as comprising at least one of a live user and a spoof attack based on the output obtained from the trained deep learning model,
wherein the trained deep learning model comprises at least one convolutional neural network, the at least one convolutional neural network being trained using at least one feedback network configured to improve performance of the at least one convolutional neural network in classifying images from multiple domains at least in part by generating, during training, one or more domain classifier scores representing one or more probabilities of an input training data example belonging to one or more domains.

### Claim 8 (depends on 7)

The computing system of claim 7, wherein the trained deep learning model is trained based on any one of:
training data comprising facial feature locations extracted from training images; and
feedback from the at least one feedback network.

### Claim 9 (depends on 7)

The computing system of claim 7, wherein the trained deep learning model is configured to identify spoof attacks in images from the multiple domains, the multiple domains including pre-recorded videos comprising a face, still images comprising a face, and live users wearing a mask.

### Claim 10 (depends on 9)

The computing system of claim 9, wherein the camera is disposed in a user device and the trained deep learning model is stored on computer-readable medium of the server.

### Claim 11 (depends on 9)

The computing system of claim 9, wherein the at least one convolutional neural network comprises a residual network.

### Claim 12

A method of training a deep learning model for identifying facial spoofing, the method comprising:
using at least one computer hardware processor to perform:
accessing training data obtained by extracting facial feature locations from training images; and
training the deep learning model using the training data, wherein the deep learning model comprises at least one residual network, and wherein training the deep learning model comprises using at least one feedback network to improve performance of the at least one residual network in classifying images from multiple domains at least in part by generating, during training, one or more domain classifier scores representing one or more probabilities of an input training data example belonging to one or more domains.

### Claim 13 (depends on 12)

The method of claim 12, wherein the at least one residual network is configured to pass a feature vector to a label classifier network and a gradient reversal layer.

### Claim 14 (depends on 13)

The method of claim 13, wherein the label classifier network is configured to process the feature vector into a classification score vector and to classify whether the at least one image is an image of a live user or a spoof attack based on the classification score vector.

### Claim 15 (depends on 13)

The method of claim 13, wherein the gradient reversal layer is configured to optimize parameters of the at least one residual network during training.

### Claim 16 (depends on 13)

The method of claim 13, wherein the gradient reversal layer is configured to pass the feature vector to the at least one feedback network during forward propagation.

### Claim 17

A method of training a deep learning model for identifying facial spoofing, the method comprising:
using at least one computer hardware processor to perform:
accessing training data obtained by extracting facial feature locations from training images; and
training the deep learning model using the training data, wherein:
the deep learning model comprises at least one residual network,
training the deep learning model comprises using at least one feedback network to improve performance of the at least one residual network in classifying images from multiple domains at least in part by generating, during training, one or more domain classifier scores representing one or more probabilities of an input training data example belonging to one or more domains, and
the at least one feedback network comprises a domain discriminator network configured to produce a domain classification loss vector, wherein during training:
the domain discriminator network minimizes the domain classification loss vector; and
the at least one residual network maximizes the domain classification loss vector until an optimization between the domain discriminator network and the at least one residual network is reached.

### Claim 18 (depends on 12)

The method of claim 12, wherein the deep learning model is trained to identify spoof attacks including pre-recorded videos comprising a face, still images comprising a face, and live users wearing a mask.