Back to results
US 12,725,608 B2Granted
Automatic speech recognition with multilingual scalability and low-resource adaptation
AssigneeInternational Business Machines Corporation
InventorsYang Zhang, Kaizhi Qian, Chuang Gan, Zhenfang Chen
Life of this patent
Estimating the term…
Abstract
Systems and techniques that facilitate multilingual ASR machine learning models are provided. For example, one or more embodiments described herein can comprise a system, which can comprise a memory that can store computer executable components. The system can also comprise a processor, operably coupled to the memory that can execute the computer executable components stored in memory. The computer executable components can comprise an automatic speech recognition machine learning model that learns shared weights across one or more languages in a set of target languages; and a training component that trains the automatic speech recognition model.
Claims20
A system comprising:
- a memory that stores computer executable components;
- a processor that executes the computer executable components stored in the memory,
- wherein the computer executable components comprise:
- an automatic speech recognition machine learning model that learns shared weights across one or more languages in a set of target languages, the automatic speech recognition machine learning model comprising a projection layer having pretrained weights inherited from a previously trained model;
- and
- a training component that trains the automatic speech recognition machine learning model, wherein the training comprises:
- learning, using the projection layer, a mapping matrix having a plurality of rows corresponding respectively to languages of the set of target languages and columns corresponding to individual pretrained weights of the projection layer, the mapping matrix comprising rows of specialist scores for the one or more languages in the set of target languages;
- generating a set of binary masks for the one or more languages in the set of target languages by thresholding the specialist scores of the mapping matrix to identify a subset of the pretrained weights associated with each language;
- and
- adapting the automatic speech recognition machine learning model to the one or more languages of the set of target languages by selectively activating the subset of pretrained weights of the projection layer according to the generated binary masks while retaining remaining pretrained weights as shared weights across the set of target languages.
7 more elements
A computer-implemented method comprising:
- training, by a system operatively coupled to a processor, an automatic speech recognition machine learning model to learn shared weights across one or more languages in a set of target languages, the automatic speech recognition machine learning model comprising a projection layer having pretrained weights inherited from a previously trained speech recognition model, wherein the training comprises:
- learning, by the system, using the projection layer of the automatic speech recognition machine learning model, a mapping matrix having a plurality of rows corresponding respectively to languages of the set of target languages and columns corresponding to projection weights of the projection layer, the mapping matrix comprising rows of specialist scores for the one or more languages in the set of target languages;
- generating, by the system, a set of binary masks for the one or more languages in the set of target languages based on the specialist scores of the mapping matrix, each binary mask identifying a subset of projection weights associated with a respective language;
- and
- adapting, by the system, the automatic speech recognition machine learning model to the one or more languages of the set of target languages by applying the binary mask corresponding to a selected language to modulate outputs of the projection layer using the subset of projection weights identified by the binary mask while maintaining remaining projection weights as shared weights across the set of target languages.
2 more elements
A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
- train, by the processor, an automatic speech recognition machine learning model to learn shared weights across one or more languages in a set of target languages, the automatic speech recognition machine learning model comprising a projection layer having pretrained weights inherited from a previously trained speech recognition model, wherein the training comprises:
- learning, by the processor, using the projection layer of the automatic speech recognition machine learning model, a mapping matrix having a plurality of rows corresponding respectively to languages of the set of target languages and columns corresponding to projection weights of the projection layer, the mapping matrix comprising rows of specialist scores for the one or more languages in the set of target languages;
- generating, by the processor, a set of binary masks for the one or more languages in the set of target languages based on the specialist scores of the mapping matrix, each binary mask representing a language-specific selection vector corresponding to the projection weights of the projection layer;
- and
- adapting, by the processor, the automatic speech recognition machine learning model to the one or more languages of the set of target languages by selectively applying, by the program instructions, the binary masks to the pretrained weights of the projection layer to enable language-specific specialization of the projection layer while retaining remaining pretrained weights as shared parameters across the set of target languages.
2 more elements
Dependencies are inferred from the claim text.