# US 12,015,637 B2 · Systems and methods for end-to-end architectures for voice spoofing detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,015,637 B2
- Document ID: US12015637B2
- Status: Granted
- Kind: B2
- Application number: 16/841,473
- Filed: 2020-04-06
- Published: 2020-10-08
- Granted: 2024-06-18
- Earliest claimed: 2019-04-08
- Assignees: Pindrop Security, Inc.
- Applicants: Pindrop Security, Inc.
- Inventors: Khaled Lakhdhar; Parav Nagarsheth; Tianxiang Chen; Elie Khoury
- Web page: https://patentgrep.com/patent/US12015637B2

## CPC

- H04L 65/1079: {of unsolicited session attempts, e.g. SPIT}
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G10L 17/00: Speaker identification or verification techniques
- G10L 17/04: Training, enrolment or model building
- G10L 17/18: Artificial neural networks; Connectionist approaches
- G10L 17/26: Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
- G10L 19/26: Pre-filtering or post-filtering
- H04L 63/1466: {Active attacks involving interception, injection, modification, spoofing of data unit addresses, e.g. hijacking, packet injection or TCP sequence number attacks}
- H04L 65/1083: In-session procedures
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Embodiments described herein provide for automatically detecting whether an audio signal is a spoofed audio signal or a genuine audio signal. A spoof detection system can include an audio signal transforming front end and a classification back end. Both the front end and the back end can include neural networks that can be trained using the same set of labeled audio signals. The audio signal transforming front end can include a one or more neural networks for per-channel energy normalization transformation of the audio signal, and the back end can include a convolution neural network for classification into spoofed or genuine audio signal. In some embodiments, the transforming audio signal front end can include one or more neural networks for bandpass filtering of the audio signals, and the back end can include a residual neural network for audio signal classification into spoofed or genuine audio signal.

## Claims

### Claim 1

A computer-implemented method for implementing an anti-spoofing end-to-end neural network architecture, the method comprising:
receiving, by a computer, a raw audio signal of a call purportedly associated with a verified speaker identity;
generating, by the computer executing a convolution layer of a signal processing frontend of a neural network architecture, a two-dimensional transformed representation from the received raw audio signal based upon a set of bandpass filter parameters, thereby resulting in a processed audio signal;
filtering, by the computer executing the signal processing frontend of the neural network architecture, the processed audio signal into frequency channels by applying a filterbank; and
generating, by the computer executing the neural network architecture, a prediction score for the processed audio signal, the prediction score indicating a likelihood the processed audio signal of the call is associated with a spoof of the verified speaker identity;
wherein the neural network architecture is trained according to a plurality of raw audio signals having at least one raw audio signal for at least one verified call associated with the verified speaker identity and at least one raw audio signal for at least one spoofed call.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising detecting, by the computer executing a classifier backend of the neural network architecture, that the received raw audio signal is a spoofed audio signal in response to determining that the prediction score fails to satisfy a threshold value.

### Claim 3 (depends on 1)

The method according to claim 1, further comprising modifying, by the computer, one or more parameters of the neural network architecture based upon back-propagation.

### Claim 4 (depends on 1)

The method according to claim 1, wherein generating the transformed representation of the received raw audio signal comprises:
transforming, by the computer executing a one-dimensional convolution neural network of a signal processing frontend of the neural network architecture, the raw audio signal into a time-frequency representation.

### Claim 5 (depends on 4)

The method according to claim 4, wherein, the computer executes a per-channel transformation neural network to generate the processed audio signal based upon the time-frequency representation of the audio signal.

### Claim 6 (depends on 1)

The method according to claim 1, further comprising parsing, by the computer, the processed audio signal into a plurality of subframes, wherein each subframe has a fixed duration and overlaps in time with at least one adjacent subframe of the plurality of subframes, and wherein the computer generates the prediction score based upon the plurality of subframes.

### Claim 7 (depends on 1)

The method according to claim 1, further comprising generating, by the computer executing a convolution neural network of a classifier backend of the neural network architecture, a L2 normalized vector based upon concatenation of mean and standard deviation of a last max-pooling layer of a plurality of max-pooling layers.

### Claim 8 (depends on 1)

The method according to claim 1, further comprising:
receiving, by the computer, from a computing device, an updated set of bandpass filter parameters, wherein the computer generates the two-dimensional matrix and the prediction score based upon the updated set of bandpass filter parameters.

### Claim 9 (depends on 1)

The method according to claim 1, wherein the computer executes a residual neural network of a classifier backend of the neural network architecture to generate the prediction score based upon the two-dimensional matrix and a set of residual neural network parameters.

### Claim 10 (depends on 1)

The method according to claim 1, further comprising:
generating, by the computer executing a residual neural network of a classifier backend of the neural network architecture, a pooled feature map based upon a feature map generated by a statistics layer of the residual neural network; and
executing, by the computer executing the residual neural network, two fully connected layers of the residual neural network using the pooled feature map.

### Claim 11 (depends on 1)

The method according to claim 1, wherein the received raw audio signal comprises at least one of speech and background noise.

### Claim 12

A computer-implemented method for implementing an anti-spoofing end-to-end neural network architecture, the method comprising:
executing, by a computer, a neural network architecture on a plurality of audio signals having at least one audio signal associated with at least one spoofed call, the neural network architecture configured to process each audio signal and classify whether each audio signal is associated with a spoofed call, wherein a signal processing frontend of the neural network architecture is configured to process each audio signal based upon a filterbank of bandpass filters to generate a transformed representation of each audio signal, and wherein a classifier backend of the neural network architecture is configured to classify whether the transformed representation of each audio signal is associated with the spoofed call; and
training, by the computer, a first set of parameters for signal processing based upon one or more loss functions generated for the audio signals by training, by the computer, a one-dimensional convolution layer and a residual neural network based upon each of the loss functions generated for the plurality of audio signals.

### Claim 13 (depends on 12)

The method according to claim 12, further comprising training, by the computer, a second set of parameters for classifying whether each audio signal is associated with a spoofed call.

### Claim 14 (depends on 12)

The method according to claim 12, wherein training the first set of parameters further comprises:
modifying, by the computer, at least one set of parameters based upon back-propagation of executing one or more neural networks of the neural network architecture.

### Claim 15 (depends on 12)

The method according to claim 12, wherein executing the neural network architecture on the plurality of audio signals further comprises, for each respective audio signal:
generating, by the computer executing the classifier backend, a loss function based upon a prediction score and a predetermined verification status of the respective audio signal, wherein the verification status indicates whether the respective audio signal is associated with a spoofed call.

### Claim 16 (depends on 12)

The method according to claim 12, wherein executing the neural network architecture on the plurality of audio signals further comprises, for each respective audio signal:
generating, by the computer executing the signal processing frontend, a transformed audio signal based upon a raw audio signal of the audio signal and the first set of parameters.

### Claim 17 (depends on 12)

The method according to claim 12, wherein executing the neural network architecture on the plurality of audio signals further comprises, for each respective audio signal:
generating, by the computer executing the classifier backend, a prediction score based upon the audio signal and a second set of parameters, wherein the prediction score indicates a likelihood the audio signal is associated with a spoofed call.

### Claim 18 (depends on 17)

The method according to claim 17, wherein executing the neural network architecture on the plurality of audio signals further comprises, for each respective audio signal:
parsing, by the computer, the audio signal into a plurality of frames, wherein the computer generates the prediction score for the audio signal using the plurality of frames.

### Claim 19 (depends on 12)

The method according to claim 12, wherein executing the neural network architecture on the plurality of audio signals further comprises, for each respective audio signal:
generating, by the computer executing the one-dimensional convolution layer of the signal processing frontend, a two-dimensional matrix of the audio signal and a set of bandpass filter parameters in the first set of parameters.

### Claim 20 (depends on 12)

The method according to claim 12, wherein executing the neural network architecture on the plurality of audio signals further comprises, for each respective audio signal:
generating, by the computer executing a residual neural network of the classifier backend, a prediction score based upon a two-dimensional matrix of the audio signal and a set of residual neural network parameters in a second set of parameters, wherein the prediction score indicates a likelihood that the audio signal is a genuine audio signal.

### Claim 21 (depends on 12)

The method according to claim 12, wherein executing the neural network architecture on the plurality of audio signals further comprises, for each respective audio signal:
generating, by the computer executing a residual neural network of the classifier backend, a loss function based upon a prediction score and a predetermined verification status of the audio signal.

### Claim 22 (depends on 12)

The method according to claim 12, wherein training the first set of parameters for signal processing further comprises:
modifying a set of bandpass filter parameters in the first set of parameters and a set of residual neural network parameters in a second set of parameters for classifying whether each audio signal is associated with a spoofed call.