Method and system of detecting and improving real-time mispronunciation of words

A method for enhancing pronunciation during a speech, the method comprising:

• training an acoustic model to analyze digitized speech to identify linguistic units of the digitized speech and to output an indication of a first level of confidence that a pronunciation of one or more words included in the digitized speech was accurate by:
• generating a transcript of a plurality of audio recordings comprising example speech,
• analyzing the transcript and the plurality of audio recordings to create statistical representations of sounds representing each word of the example speech, and
• using the statistical representations to determine whether the pronunciation of the one or more words is correct;
• training a language model to distinguish between similar sounding words and phrases in example digitized speech and to output an indication of a second level of confidence that the pronunciation of the one or more words included in the example digitized speech was accurate;
• receiving, from a client device over a network connection, audio data, including a speech;
• digitizing the speech to generate digitized speech;
• performing acoustic scoring and language scoring on the digitized speech by providing the digitized speech as an input to the acoustic model to obtain as a first output the indication of a first level of confidence that a pronunciation of one or more words included in the digitized speech was accurate, and providing the digitized speech as an input to the language model to obtain as a second output the indication of the second level of confidence that the pronunciation of the one or more words included in the digitized speech was accurate;
• determining a pronunciation score of one or more words of the speech based on the acoustic scoring and the language scoring by combining the indication of the first level of confidence and the indication of the second level of confidence;
• determining that the pronunciation score for the one or more words does not satisfy a threshold score;
• responsive to determining that the pronunciation score does not satisfy the threshold score, identifying the one or more words as mispronounced;
• responsive to identifying the one or more words as mispronounced, outputting the one or more words and the pronunciation score thereof;
• generating a visual representation of a correct pronunciation of the one or more words that were mispronounced, the visual representation comprising a syllable or phoneme sequence of the correct pronunciation of the one or more words that were mispronounced;
• generating an audio representation of the correct pronunciation of the one or more words that were mispronounced;
• and
• sending, over the network connection, the visual representation and the audio representation of the correct pronunciation of the one or more words that were mispronounced to the client device.

A non-transitory computer readable medium on which are stored instructions that, when executed, cause a programmable device to enhance pronunciation of a speech by:

• training an acoustic model to analyze digitized speech to identify linguistic units of the digitized speech and to output an indication of a first level of confidence that a pronunciation of one or more words included in the digitized speech was accurate by:
• generating a transcript of a plurality of audio recordings comprising example speech,
• analyzing the transcript and the plurality of audio recordings to create statistical representations of sounds representing each word of the example speech, and
• using the statistical representations to determine whether the pronunciation of the one or more words is correct;
• training a language model to distinguish between similar sounding words and phrases in example digitized speech and to output an indication of a second level of confidence that the pronunciation of the one or more words included in the example digitized speech was accurate;
• receiving, from a client device over a network connection, audio data, the audio data including a speech;
• digitizing the speech to generate digitized speech;
• performing acoustic scoring and language scoring on the digitized speech by providing the digitized speech as an input to the acoustic model to obtain as a first output the indication of a first level of confidence that a pronunciation of one or more words included in the digitized speech was accurate, and providing the digitized speech as an input to the language model to obtain as a second output the indication of the second level of confidence that the pronunciation of the one or more words included in the digitized speech was accurate;
• determining a pronunciation score of one or more words of the speech based on the acoustic scoring and the language scoring by combining the indication of the first level of confidence and the indication of the second level of confidence;
• determining that the pronunciation score for the one or more words does not satisfy a threshold score;
• responsive to determining that the pronunciation score does not satisfy the threshold score, identifying the one or more words as mispronounced;
• responsive to identifying the one or more words as mispronounced, outputting the one or more words and the pronunciation score thereof;
• generating a visual representation of a correct pronunciation of the one or more words that were mispronounced, the visual representation comprising a syllable or phoneme sequence of the correct pronunciation of the one or more words that were mispronounced;
• generating an audio representation of the correct pronunciation of the one or more words that were mispronounced;
• and
• sending, over the network connection, the visual representation and the audio representation of the correct pronunciation of the one or more words that were mispronounced to the client device.