Computer-implemented systems and methods for evaluating speech dialog system engagement via video

A computer-implemented system for providing a spoken dialog system, comprising:

• one or more data processors;
• a computer-readable medium encoded with instructions for commanding the one or more data processors to execute steps comprising:
• providing an engagement engine comprising a spoken dialog system configured to have a conversation with a person;
• providing an output from the spoken dialog system to the person, wherein the output is intended to prompt a plurality of responses from the person;
• capturing audio and video data of the person's responses to the output;
• extracting audio and video features from the audio and video data, wherein the audio and video features are indicative of the person's level of engagement with the spoken dialog system;
• deriving a plurality of engagement metrics from the audio and video features, wherein the plurality of engagement metrics are indicative of the level of the person's engagement with the spoken dialog system;
• deriving additional engagement metrics based on a quality of the person's responses;
• and
• adjusting the spoken dialog system during the conversation, based a combination of the plurality of engagement metrics derived from the audio features, the video features, and the quality of the person's responses to improve the person's level of engagement with the spoken dialog system, the adjusting comprising displaying at least one of a picture or a video when the engagement metrics indicate that the level of the person's engagement with the spoken dialog system is below a pre-defined threshold.

A non-transitory computer-readable medium encoded with instructions for commanding one or more data processors to execute steps of a method for providing a spoken dialog system, the steps comprising:

• providing an engagement engine comprising a spoken dialog system configured to have a conversation with a person;
• providing an output from the spoken dialog system to the person, wherein the output is intended to prompt a plurality of responses from the person;
• capturing audio and video data of the person's responses to the output;
• extracting audio and video features from the audio and video data, wherein the audio and video features are indicative of the person's level of engagement with the spoken dialog system;
• deriving a plurality of engagement metrics from the audio and video features, wherein the plurality of engagement metrics are indicative of the level of the person's engagement with the spoken dialog system;
• deriving additional engagement metrics based on a quality of the person's responses;
• and
• adjusting the spoken dialog system during the conversation, based a combination of the plurality of engagement metrics derived from the audio features, the video features, and the quality of the person's responses to improve the person's level of engagement with the spoken dialog system, the adjusting comprising displaying at least one of a picture or a video when the engagement metrics indicate that the level of the person's engagement with the spoken dialog system is below a pre-defined threshold.