Visual speech recognition based communication training system

A computer-readable storage medium containing program instructions for a method being executed by an application, the application comprising code for one or more components that are called by the application during runtime, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to perform steps comprising:

• obtaining a user profile specific to a user;
• selecting, from a content database, a subset of training instances based on the user profile;
• generating a set of lesson content for the user that includes the subset of training instances and a set of evaluation prompts;
• providing, to a user device, the set of lesson content;
• receiving, from the user device, a set of responses to the set of evaluation prompts;
• deriving, via a VSR-based evaluation model, a score for each set of responses by comparing the responses provided by the user with the predictions of the VSR model to the same set of evaluation prompts;
• and
• updating any portion of the set of lesson content based on the derived score for each of the set of responses.

A computer-implemented method comprising:

• obtaining a user profile specific to a user, the user profile specifying any of:
• a user type, an age of the user, one or more interests of the user, and a learning goal of the user;
• generating at least one training instance by:
• obtaining a video source depicting a subject speaking;
• determining by a visual speech recognition (VSR) model, the speech content of the video source;
• and
• processing each word of the speech content to derive one or more attributes of each word in each training instance;
• storing the training instance to a content database that includes a set of training instances and corresponding audio output and/or text subtitles;
• selecting, from the content database, a subset of training instances based at least on the learning goal of the user as specified in the user profile;
• generating a set of evaluation prompts, wherein each of the set of evaluation prompts include any of:
• a string of text with a request for the user to record a video on a user device to use tailored or silent speech to reproduce the string of text;
• and
• a video of a sample subject speaking without audio, and a request for the user to respond by accurately providing corresponding speech content;
• providing, to a user device, a set of lesson content for the user that includes the subset of training instances and a set of evaluation prompts;
• receiving, from the user device, a set of responses to the set of evaluation prompts;
• deriving, via a VSR-based evaluation model, a score for each set of responses by comparing the responses provided by the user with the predictions of the VSR model to the same set of evaluation prompts;
• and
• updating any portion of the set of lesson content based on the derived score for each of the set of responses.