Deep learning internal state index-based search and classification

A non-transitory computer storage that stores executable program instructions that, when executed by one or more computing devices, configure the one or more computing devices to perform operations comprising:

• receiving an audio file and a text file;
• generating, from the received audio file, a sequence of audio phonemes;
• generating, from the received text file, a sequence of text phonemes;
• performing a first layer search comprising generating a mapping of the audio and text phonemes, the mapping comprising a first layer alignment of the audio and text phonemes;
• performing a second layer search comprising generating a plurality of candidates by adjusting the first layer alignment, each candidate comprising a candidate alignment of audio and text phonemes derived from the first layer alignment;
• selecting a set of candidates;
• iteratively reperforming the first layer and the second layer searches for each candidate in the selected set of candidates, wherein a candidate alignment is used to regenerate the first layer alignment;
• stop reperforming the first layer and the second layer searches when a stopping condition is satisfied;
• and
• generating an output alignment of audio and text phonemes when the stopping condition is satisfied;
• wherein the stopping condition comprises a number of matching phonemes between the audio phonemes and text phonemes not changing when reperforming the first layer and the second layer searches.