Method and apparatus for multi-lingual end-to-end speech recognition

A speech recognition system for performing multi-language speech recognition, comprising:

• an interface to receive voice sounds;
• one or more processors;
• and
• one or more storage devices storing an end-to-end speech recognition network module having been trained by using trained network parameters obtained by a method that comprises:
• providing utterance datasets corresponding to predetermined languages;
• inserting language identification (ID) labels into the utterance datasets, wherein each of the utterance datasets is labelled by each of the language ID labels;
• concatenating the labeled utterance datasets;
• generating initial network parameters from the utterance datasets;
• and
• training, iteratively, an end-to-end network with a series of the initial network parameters and the concatenated labeled utterance datasets until a training result reaches a threshold,
• wherein the end-to-end speech network recognition module includes instructions which, when executed, cause the one or more processors to perform operations comprising:
• extracting, using an acoustic feature extraction module, an acoustic feature sequence from audio waveform data converted from the voice sounds;
• encoding the acoustic feature sequence into a hidden vector sequence using an encoder network having encoder network parameters;
• predicting first output label sequence probabilities by feeding the hidden vector sequence to a decoder network having decoder network parameters;
• predicting second output label sequence probabilities by a connectionist temporal classification (CTC) module using CTC network parameters and the hidden vector sequence from the encoder network;
• and
• searching, using a label sequence search module, for an output label sequence having a highest sequence probability by combining the first and second output label sequence probabilities provided from the decoder network and the CTC module.

A multi-lingual speech recognition system for generating trained network parameters for a multi-language speech recognition, comprising:

• one or more processors;
• and
• one or more storage devices storing parameters and program modules including instructions executable by the one or more processors which, when executed, cause the one or more processors to perform operations comprising:
• providing utterance datasets corresponding to predetermined languages;
• inserting language identification (ID) labels into the utterance datasets, wherein each of the utterance datasets is labelled by each of the language ID labels;
• concatenating the labeled utterance datasets;
• generating initial network parameters from the utterance datasets;
• selecting the initial network parameters according to a predetermined sequence;
• and
• training, iteratively, an end-to-end network with a series of the selected initial network parameters and the concatenated labeled utterance datasets until a training result reaches a threshold,
• wherein the end-to-end network performs:
• extracting, using an acoustic feature extraction module, an acoustic feature sequence from audio waveform data converted from the voice sounds;
• encoding the acoustic feature sequence into a hidden vector sequence using an encoder network having encoder network parameters;
• predicting first output label sequence probabilities by feeding the hidden vector sequence to a decoder network having decoder network parameters;
• predicting second output label sequence probabilities by a connectionist temporal classification (CTC) module using CTC network parameters and the hidden vector sequence from the encoder network;
• and
• searching, using a label sequence search module, for an output label sequence having a highest sequence probability by combining the first and second output label sequence probabilities provided from the decoder network and the CTC module.