Conditional factorization for jointly modeling code-switched and monolingual asr

An apparatus for automatic speech recognition using conditional factorization for bilingual code-switched and monolingual speech, the apparatus comprising:

• at least one memory configured to store program code;
• and
• at least one processor configured to read the program code and operate a instructed by the program code, the program code including:
• first receiving code configured to cause the at least one processor to receive an audio observation sequence comprising a plurality of frames, the audio observation sequence comprising audio in a first language and intra-sentential code-switching audio in a second language;
• first mapping code configured to cause the at least one processor to map the audio observation sequence into a first sequence of hidden representations, the mapping being generated by a first encoder corresponding to the first language, wherein the first sequence of hidden representations comprises blank tokens associated with the intra-sentential code-switching audio in the second language;
• second mapping code configured to cause the at least one processor to map the audio observation sequence into a second sequence of hidden representations, the mapping being generated by a second encoder corresponding to the second language, wherein the second sequence of hidden representations comprises blank tokens associated with the audio in the first language;
• and
• first generating code configured to cause the at least one processor to generate a label-to-frame sequence based on the first sequence of hidden representations and the second sequence of hidden representations, using a joint neural network based model.

A non-transitory computer-readable medium storing instructions, the instructions comprising:

• one or more instructions that, when executed by one or more processors of a device for automatic speech recognition using conditional factorization for bilingual code-switched and monolingual speech, cause the one or more processors to:
• receive an audio observation sequence comprising a plurality of frames, the audio observation sequence comprising audio in a first language and intra-sentential code-switching audio in a second language;
• map the audio observation sequence into a first sequence of hidden representations, the mapping being generated by a first encoder corresponding to the first language, wherein the first sequence of hidden representations comprises blank tokens associated with the intra-sentential code-switching audio in the second language;
• map the audio observation sequence into a second sequence of hidden representations, the mapping being generated by a second encoder corresponding to the second language, wherein the second sequence of hidden representations comprises blank tokens associated with the audio in the first language;
• and
• generate a label-to-frame sequence based on the first sequence of hidden representations and the second sequence of hidden representations, using a joint neural network based model.