# US 2021/0233541 A1 · Robust spoofing detection system using deep residual neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0233541 A1
- Document ID: US20210233541A1
- Status: Published application
- Kind: A1
- Application number: 17/155,851
- Filed: 2021-01-22
- Published: 2021-07-29
- Earliest claimed: 2020-01-27
- Assignees: Pindrop Security, Inc.
- Applicants: Pindrop Security, Inc.
- Inventors: Tianxiang CHEN; Elie KHOURY
- Web page: https://patentgrep.com/patent/US20210233541A1

## CPC

- G10L 17/18: Artificial neural networks; Connectionist approaches
- G10L 17/02: Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
- G10L 17/04: Training, enrolment or model building
- G10L 17/08: Use of distortion metrics or a particular distance between probe pattern and reference templates
- G10L 17/22: Interactive procedures; Man-machine interfaces
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Embodiments described herein provide for systems and methods for implementing a neural network architecture for spoof detection in audio signals. The neural network architecture contains a layers defining embedding extractors that extract embeddings from input audio signals. Spoofprint embeddings are generated for particular system enrollees to detect attempts to spoof the enrollee's voice. Optionally, voiceprint embeddings are generated for the system enrollees to recognize the enrollee's voice. The voiceprints are extracted using features related to the enrollee's voice. The spoofprints are extracted using features related to features of how the enrollee speaks and other artifacts. The spoofprints facilitate detection of efforts to fool voice biometrics using synthesized speech (e.g., deepfakes) that spoof and emulate the enrollee's voice.

## Claims

### Claim 1

A computer-implemented method for spoofing countermeasures, the method comprising:
generating, by a computer, an enrollee spoofprint for an enrollee based upon a first set of one or more features extracted from one or more enrollee audio signals for the enrollee, wherein the first set of one or more features includes one or more audio characteristics of the enrollee;
applying, by the computer, a neural network architecture to an inbound audio signal, the neural network architecture trained to detect spoofing artifacts occurring in an audio signal;
generating, by the computer, an inbound spoofprint for an inbound speaker by applying the neural network architecture to the inbound audio signal for the inbound speaker; and
generating, by the computer, a spoof likelihood score for the inbound audio signal based upon one or more similarities between the inbound spoofprint and the enrollee spoofprint.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising:
extracting, by the computer, a plurality of features from a plurality of training audio signals, the plurality of training audio signals comprising one or more simulated audio signals and one or more clean audio signals; and
training, by the computer, the neural network architecture to detect speech by applying the neural network architecture to the plurality of features.

### Claim 3 (depends on 2)

The method according to claim 2, further comprising generating, by the computer, the one or more simulated audio signals by executing one or more data augmentation operations.

### Claim 4 (depends on 2)

The method according to claim 2, further comprising during a training phase:
executing, by the computer, a loss function of the neural network architecture for the spoof likelihood score outputted by the neural network architecture, the loss function instructing the computer to update one or more hyperparameters of one or more layers of the neural network architecture based on maximizing inter-class variance and minimizing intra-class variance.

### Claim 5 (depends on 1)

The method according to claim 1, wherein generating the enrollee spoofprint for the enrollee includes:
applying, by the computer, the neural network architecture to the first set of one or more features extracted from the one or more enrollee audio signals to generate a feature vector corresponding to the enrollee spoofprint.

### Claim 6 (depends on 5)

The method according to claim 5, further comprising, during an enrollment phase, generating, by the computer, one or more simulated enrollee audio signals by executing one or more data augmentation operations on the one or more enrollee audio signals.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the one or more data augmentation operations includes a frequency masking data augmentation operation.

### Claim 8 (depends on 1)

The method according to claim 1, further comprising:
generating, by the computer, an enrollee voiceprint for the enrollee by applying the neural network architecture to a second set of one or more features extracted from the one or more enrollee audio signals for the enrollee, wherein the second set of one or more features includes one or more voice characteristics of the enrollee;
generating, by the computer, an inbound voiceprint for the inbound speaker by applying the neural network architecture to the second set of one or more features extracted from the inbound audio signal; and
generating, by the computer, a voice similarity score for the inbound audio signal based upon one or more similarities between the inbound voiceprint and the enrollee voiceprint; and
generating, by the computer, a combined similarity score based upon the voice similarity score and the spoof likelihood score.

### Claim 9 (depends on 1)

The method according to claim 1, further comprising:
generating, by the computer, an enrollee combined embedding based upon the enrollee spoofprint and an enrollee voiceprint;
generating, by the computer, an inbound combined embedding based upon the inbound spoofprint and an inbound voiceprint; and
generating, by the computer, a similarity score for the inbound audio signal based upon a similarity between the enrollee combined embedding and the inbound combined embedding.

### Claim 10 (depends on 1)

The method according to claim 1, wherein the neural network architecture comprises one or more layers of one or more embedding extractors, including at least one of a spoofprint embedding extractor and a voiceprint embedding extractor.

### Claim 11

A computer-implemented method for spoofing countermeasures, the method comprising:
obtaining, by a computer, a plurality of training audio signals including one or more clean audio signals and one or more simulated audio signals;
training, by the computer, a neural network architecture to extract a spoofprint embedding from an audio signal and classify the audio signal, the neural network architecture trained by applying the neural network architecture on a plurality of features of the plurality of training audio signals;
extracting, by the computer, an inbound spoofprint for the inbound speaker by applying the neural network architecture on the plurality of features of an inbound audio signal; and
generating, by the computer, a classification for the inbound audio signal based upon applying the neural network architecture on the inbound spoofprint.

### Claim 12 (depends on 11)

The method according to claim 11, further comprising generating, by the computer, the one or more simulated audio signals by executing one or more data augmentation operations.

### Claim 13 (depends on 11)

The method according to claim 11, further comprising, for each training audio signal:
extracting, by the computer, a training spoofprint for a corresponding training audio signal by applying an embedding extractor of the neural network architecture on the corresponding training audio signal; and
executing, by the computer, a loss function of the neural network architecture according to the training spoofprint outputted by the embedding extractor for the corresponding training audio signal, the loss function instructing the computer to update one or more hyperparameters of one or more layers of the neural network architecture, the one or more hyperparameters updated based on maximizing inter-class variance and minimizing intra-class variance.

### Claim 14

A system comprising:
a non-transitory machine readable memory; and
a computer comprising a processor configured to:
generate an enrollee spoofprint for an enrollee based upon a first set of one or more features extracted from one or more enrollee audio signals for the enrollee, wherein the first set of one or more features includes one or more types of audio characteristics of the enrollee;
store the enrollee spoofprint into the memory;
apply a neural network architecture to an inbound audio signal, the neural network architecture trained to detect spoofing artifacts occurring in an audio signal;
generate an inbound spoofprint for an inbound speaker by applying the neural network architecture to an inbound audio signal for the inbound speaker; and
generate a spoof likelihood score for the inbound audio signal based upon one or more similarities between the inbound spoofprint and the enrollee spoofprint.

### Claim 15 (depends on 14)

The system according to claim 14, wherein the computer is further configured to:
extract a plurality of features from a plurality of training audio signals, the plurality of training audio signals comprising one or more simulated audio signals and one or more clean audio signals; and
training, by the computer, the neural network architecture to detect speech by applying the neural network architecture to the plurality of features.

### Claim 16 (depends on 15)

The system according to claim 15 wherein the computer is further configured to:
generate the one or more simulated audio signals by executing one or more one or more data augmentation operations.

### Claim 17 (depends on 15)

The system according to claim 15, wherein the computer is further configured to, during a training phase:
execute a loss function of the neural network architecture for the spoof likelihood score outputted by the neural network architecture, the loss function instructing the computer to update hyperparameters of the neural network architecture based on maximizing inter-class variance and minimizing intra-class variance.

### Claim 18 (depends on 14)

The system according to claim 14, wherein the computer is further configured to:
apply the neural network architecture to the first set of one or more features extracted from the one or more enrollee audio signals to generate a feature vector corresponding to the enrollee spoofprint.

### Claim 19 (depends on 14)

The system according to claim 14, wherein the computer is further configured to:
generate an enrollee voiceprint for the enrollee by applying the neural network architecture to a second set of one or more features extracted from the one or more enrollee audio signals for the enrollee, wherein the second set of one or more features includes one or more voice characteristics of the enrollee;
generate an inbound voiceprint for the inbound speaker by applying the neural network architecture to the second set of one or more features extracted from the inbound audio signal; and
generate a voice similarity score for the inbound audio signal based upon one or more similarities between the inbound voiceprint and the enrollee voiceprint; and
generate a combined similarity score based upon the voice similarity score and the spoof likelihood score.

### Claim 20 (depends on 14)

The system according to claim 14, wherein the neural network architecture comprises one or more layers of one or more embedding extractors, including at least one of a spoofprint embedding extractor and a voiceprint embedding extractor.
