User specified keyword spotting using neural network feature extractor

#Automatic speech recognition is one technology that is used in mobile devices. One task that is a common goal for this technology is to be able to use voice commands to wake up a mobile device or to have basic spoken interactions with the mobile device. For example, it may be desirable for the mobile device to recognize a “hotword” that signals that the mobile device should activate when the mobile device is in a sleep state.

#In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of receiving, by a device for each of multiple variable length enrollment audio signals each encoding a respective spoken utterance of an enrollment phrase, a respective plurality of enrollment feature vectors that represent features of the respective variable length enrollment audio signal, wherein when the device determines that another audio signal encodes another spoken utterance of the enrollment phrase, the device performs a particular action assigned to the enrollment phrase, and for each of the multiple variable length enrollment audio signals: processing, by the device, each of the plurality of enrollment feature vectors for the respective variable length enrollment audio signal using a long short term memory (LSTM) neural network to generate a respective enrollment LSTM output vector for each enrollment feature vector, and generating, by the device for the respective variable length enrollment audio signal, a template fixed length representation for use in determining whether the other audio signal encodes another spoken utterance of the enrollment phrase by combining at most a quantity k of the LSTM output vectors for the enrollment audio signal, wherein a predetermined length of each of the template fixed length representations is the same. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

#The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination. The method may include, for each of the multiple variable length enrollment audio signals: determining whether at least the quantity k of enrollment feature vectors were generated for the respective enrollment audio signal, and in response to determining that less than the quantity k of enrollment feature vectors were generated for the respective enrollment audio signal, adding leading zeros to a front of the respective template fixed length representation so that the respective template fixed length representation has the predetermined length. The method may include determining an average number of enrollment frames in all of the enrollment audio signals wherein the quantity k is the average number of enrollment frames. Each of the enrollment output vectors may have a predetermined size 1 that corresponds to a size of a last layer in the long short term memory neural network. Adding leading zeros to the front of the respective template fixed length representation may include adding leading zeros to the front of the respective template fixed length representation so that the respective template fixed length representation has a total of 1 times k values. The last layer in the long short term memory neural network may include a hidden layer during training of the long short term memory neural network.

#FIG. 1 is an example of a speech recognition system with a long short term memory neural network.

#FIG. 2 is an example system for determining whether an audio waveform contains a digital representation of a specific keyword or key phrase.

#FIG. 3 is a block diagram of an example system for a feature extraction process.

#Like reference numbers and designations in the various drawings indicate like elements.

#A device uses a long short term memory (LSTM) neural network to allow a user to specify whichever phrase they choose as a hotword key phrase. For instance, the device requests that the user speak the enrollment phrase a predetermined number of times, e.g., three, four, or five times, during an enrollment process and uses the LSTM neural network to generate enrollment vectors that represent the enrollment phrase. After enrollment, the device analyzes other phrases spoken by users to determine whether vectors that represent those phrases are within a predetermined distance of the enrollment vectors and those phrases are the same as the enrollment phrase and, optionally, spoken by the same user that performed the enrollment process. In some examples, the device may perform the analysis while in a sleep state and wake up upon determining that another phrase is the same as the enrollment phrase.

#In some examples, a device may use the neural network to analyze received audio waveforms and determine if a sequence of frames from an audio waveform include a digital representation of one of the specific keywords or key phrases provided by the user during the enrollment process. Upon determination that a sequence of frames contains a digital representation of one of the specific keywords, or has probability above a threshold probability of containing a digital representation of one of the specific keywords, the device may perform an action that corresponds with the one of the specific keywords. For instance, the device may exit a standby state, launch an application, or perform another action.