US 10,540,961 B2 · Convolutional recurrent neural networks for small-footprint keyword spotting

Abstract

Described herein are systems and methods for creating and using Convolutional Recurrent Neural Networks (CRNNs) for small-footprint keyword spotting (KWS) systems. Inspired by the large-scale state-of-the-art speech recognition systems, in embodiments, the strengths of convolutional layers to utilize the structure in the data in time and frequency domains are combined with recurrent layers to utilize context for the entire processed frame. The effect of architecture parameters were examined to determine preferred model embodiments given the performance versus model size tradeoff. Various training strategies are provided to improve performance. In embodiments, using only ˜230 k parameters and yielding acceptably low latency, a CRNN model embodiment demonstrated high accuracy and robust performance in a wide range of environments.

Claims20

Dependent claims

A computer-implemented method for training a small-footprint keyword-spotting model architecture, the method comprising:

receiving a set of audio samples that each comprises a keyword, the keywords having been aligned by using an alignment method that enables use of a cross-entropy loss function;
converting the set of audio samples into spectrograms;
inputting the spectrograms to one or more convolutional layers that apply one or more multi-dimensional filters to the spectrograms to generate a convolutional layer output;
using the convolutional layer output in at least one recurrent layer to generate a recurrent layer output;
providing the recurrent layer output to one or more fully connected layers;
performing a nonlinear function on an output of the one or more fully connected layers to obtain a probability score that corresponds to a probability that a sample in the set of audio samples comprises the keyword;
and
using the probability score in a cross-entropy loss function to adjust one or more model parameters of a convolutional layer.

5 more elements

A method for using a small-footprint keyword spotting model to detect a keyword, the method comprising:

receiving audio data;
inputting the audio data into a convolutional recurrent neural network (CRNN) that has been trained using cross-entropy loss to detect a target keyword in an audio sample by using a training method that comprises:
receiving a set of audio samples that each comprises the target keyword, the target keywords having been aligned by using an alignment method that enables use of a cross-entropy loss function;
aligning the target keywords in the audio samples to enable a use of a cross-entropy loss function;
converting the set of audio samples into spectrograms to obtain features;
using the features to a convolutional layer that applies one or more multi-dimensional filters to the spectrograms to generate a convolutional layer output;
using the convolutional layer output in at least one recurrent layer to generate a recurrent layer output;
providing the recurrent layer output to one or more fully connected layers;
performing a nonlinear function on an output of the one or more fully connected layers to obtain a probability score that corresponds to a probability that a sample in the set of audio samples comprises the target keyword;
and
using the probability score in a cross-entropy loss function to adjust a model parameter of a convolutional layer.

8 more elements

A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by one or more microprocessors, causes steps to be performed comprising:

receiving a set of audio samples that each comprises a keyword, the keywords having been aligned by using an alignment method that enables use of a cross-entropy loss function;
converting the set of audio samples into spectrograms;
inputting the spectrograms to a convolutional layer that applies one or more multi-dimensional filters to the spectrograms to generate a convolutional layer output;
using the convolutional layer output in at least one recurrent layer to generate a recurrent layer output;
providing the recurrent layer output to one or more fully connected layers;
performing a nonlinear function on an output of the one or more fully connected layers to obtain a probability score that corresponds to a probability that a sample in the set of audio samples comprises the keyword;
and
using the probability score in a cross-entropy loss function to adjust one or more model parameters of a convolutional layer.

5 more elements

Dependencies are inferred from the claim text.

Convolutional recurrent neural networks for small-footprint keyword spotting

Life of this patent

Convolutional recurrent neural networks for small-footprint keyword spotting

Life of this patent

Abstract

Claims20