US 10,380,997 B1 · Deep learning internal state index-based search and classification

Abstract

Systems and methods are disclosed for generating internal state representations of a neural network during processing and using the internal state representations for classification or search. In some embodiments, the internal state representations are generated from the output activation functions of a subset of nodes of the neural network. The internal state representations may be used for classification by training a classification model using internal state representations and corresponding classifications. The internal state representations may be used for search, by producing a search feature from an search input and comparing the search feature with one or more feature representations to find the feature representation with the highest degree of similarity.

Drawings

Drawings are temporarily unavailable. Open the PDF at USPTO

Drawings

Claims20

Dependent claims

A non-transitory computer-readable medium comprising instructions for:

providing a trained speech recognition neural network, the speech recognition neural network including a plurality of layers each having a plurality of nodes;
transcribing speech audio by the speech recognition neural network;
while the speech recognition neural network is transcribing the speech audio, generating one or more feature representations from a subset of nodes, the one or more feature representations representing an internal state of the speech recognition neural network at a plurality of timestamps during transcription, each of the feature representations comprising a vector of quantized values where each of the quantized values is obtained by quantizing an activation output of a node in the subset of nodes;
storing the one or more feature representations;
receiving a first set of classifications for a first portion of the speech audio;
training a classification model, the classification model being different than the trained speech recognition neural network, on a first set of feature representations corresponding to the first portion of the speech audio and the first set of classifications, the first set of feature representations comprising a first subset of the feature representations generated during the speech audio transcription;
and
determining a second set of classifications for a second portion of the speech audio by inputting a second set of feature representations corresponding to the second portion of the speech audio into the trained classification model, the second set of feature representations comprising a second subset of the feature representations generated during the speech audio transcription.

5 more elements

A non-transitory computer-readable medium comprising instructions for:

providing a trained speech recognition neural network, the speech recognition neural network including a plurality of layers each having a plurality of nodes;
performing inference by the speech recognition neural network on input data, wherein the input data comprises speech audio and performing inference comprises transcribing the speech audio;
while the speech recognition neural network is performing inference on the input data, generating one or more feature representations from a subset of nodes, the one or more feature representations representing an internal state of the speech recognition neural network at a plurality of timesteps, each of the feature representations comprising a vector of quantized values where each of the quantized values is obtained by quantizing an activation output of a node in the subset of nodes;
storing the one or more feature representations;
receiving a first set of classifications for a first portion of the input data;
training a classification model, the classification model being different than the trained speech recognition neural network, on a first set of feature representations corresponding to the first portion of the input data and the first set of classifications, each of the first set of feature representations comprising one of the vectors of quantized values generated during inference on the first portion of the input data;
and
determining a second set of classifications for a second portion of the input data by inputting a second set of feature representations corresponding to the second portion of the input data into the trained classification model, each of the second set of feature representations comprising one of the vectors of quantized values generated during inference on the second portion of the input data.

5 more elements

A non-transitory computer-readable medium comprising instructions for:

providing a trained speech recognition neural network, the speech recognition neural network including a plurality of layers each having a plurality of nodes;
transcribing speech audio by the speech recognition neural network;
while the speech recognition neural network is transcribing the speech audio, generating one or more feature representations from a subset of nodes, the one or more feature representations representing an internal state of the speech recognition neural network at a plurality of timestamps during transcription, each of the feature representations comprising a vector of quantized values where each of the quantized values is obtained by quantizing an activation output of a node in the subset of nodes;
storing the one or more feature representations;
training a second neural network to generate a search feature based on a text input, the search feature comprising a vector of values, wherein the training is performed based on one or more training examples, each training example comprising a portion of the speech audio and a corresponding subset of the one or more feature representations generated during the speech audio transcription, and wherein the second neural network is different than the trained speech recognition neural network;
receiving a text query and inputting the text query to the second neural network to generate the search feature;
determining a similarity between the search feature and each of the one or more feature representations;
selecting a feature representation with the greatest similarity with the search feature;
and
outputting an indication of a portion of the speech audio corresponding to the feature representation with the greatest similarity with the search feature.

7 more elements

A non-transitory computer-readable medium comprising instructions for:

providing a trained speech recognition neural network, the speech recognition neural network including a plurality of layers each having a plurality of nodes;
performing inference by the speech recognition neural network on input data, wherein the input data comprises speech audio and performing inference comprises transcribing the speech audio;
while the speech recognition neural network is performing inference on the input data, generating one or more feature representations from a subset of nodes, the one or more feature representations representing an internal state of the speech recognition neural network at a plurality of timesteps, each of the feature representations comprising a vector of quantized values where each of the quantized values is obtained by quantizing an activation output of a node in the subset of nodes;
storing the one or more feature representations;
training a second neural network to generate a search feature based on a text input, the search feature comprising a vector of values, wherein the training is performed based on one or more training examples, each training example comprising a portion of the speech audio and the corresponding one or more vectors of quantized values generated during the transcription of the portion of the speech audio, and wherein the second neural network is different than the trained speech recognition neural network;
receiving a text query and inputting the text query to the second neural network to generate the search feature;
determining a similarity between the search feature and each of the one or more feature representations;
selecting a feature representation with the greatest similarity with the search feature;
and
outputting an indication of a portion of the speech audio corresponding to the feature representation with the greatest similarity with the search feature.

7 more elements

Dependencies are inferred from the claim text.

Description22,783 words

Summary569 words

#Deep learning using neural networks is a useful paradigm for machine learning. However, one disadvantage of deep neural networks is that it may take a long time to train and use a full machine learning model. Meanwhile, when deep neural networks are used, generally only the output of the neural network is saved. Transient information inferred by the neural network during use is generally discarded. It would be desirable to find a way to use information generated by the neural network without having to train and use a full model.

#In an embodiment, an internal state representation of a neural network is captured during the processing of input. The internal state representation may be derived from the pattern of neural network output activations during processing. The internal state representation may be stored with the input and a generated output for later processing. In one embodiment, the internal state representations may be used for a search task. In another embodiment, the internal state representations may be used for later machine learning tasks, such as classification.

#In some embodiments, the neural network is an end-to-end speech recognition system, end-to-end speech classification system, or end-to-end phoneme recognition system. In other embodiments, the neural network is unrelated to speech recognition.

Brief description of the drawings20 figures

#The present disclosure will become more fully understood from the detailed description and the accompanying drawings, wherein:

#FIG. 1 illustrates an exemplary network environment where some embodiments of the invention may operate

#FIG. 2 illustrates an end-to-end speech recognition system according to an embodiment;

Detailed description21,911 words

#In the drawings, reference numbers may be reused to identify similar and/or identical elements.

#Embodiments described herein relate to end-to-end neural network speech recognition systems. Some disclosed embodiments form a single neural network from input to output. Because of this unitary architecture, the disclosed speech recognition systems are able to be trained solely by data driven techniques, eschewing laborious hand-tuning and increasing accuracy.

#Traditional speech pipelines need tens of people working together to build a model over several months. If one portion of the pipeline is altered, then all interfaces with the standard pipeline may be affected. Embodiments disclosed herein are trained by data-driven techniques only, without the need for human intervention.

Facts

Application

16/108,109Filed 22 Aug 2018 · us provisional application

Priority

27 Jul 2018

Granted

13 Aug 2019

Term ends

22 Aug 2038

Estimated · how

20 years from earliest known nonprovisional domestic filing plus recorded patent term adjustment. Maintenance-fee lapses, terminal disclaimers and statutory extensions have not been checked. The continuity chain may be incomplete; this is not an enforceability determination.

Classification30 codes

G10L 15/16
Speech classification or search: using artificial neural networks
G06F 18/214
Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation: generating training…
G06F 18/24133
Based on distances to training or reference patterns: distances to prototypes
G06N 3/044
Architecture, e.g. interconnection topology: recurrent networks, e.g. Hopfield networks

Deep learning internal state index-based search and classification

Life of this patent

Deep learning internal state index-based search and classification

Life of this patent

Life of this patent

Abstract

Drawings

Drawings

Claims20

Description22,783 words

Drawings

Classification30 codes

Life of this patent

Description22,783 words