System and method for using gestures and expressions for controlling speech applications

A system comprising:

• a speech input device wearable on a user configured to measure electromyography (EMG) signals produced by the user;
• and
• at least one processor configured to:
• determine a first prompt from the user based at least in part on a first EMG signal measured by the speech input device
• provide the first prompt from the user to an interactive system to cause the interactive system to perform a first function of the interactive system;
• responsive to the interactive system performing the first function, determine a feedback signal by:
• receiving at least a second EMG signal indicative of a facial expression of the user responsive to the interactive system performing the first function;
• and
• generating, using a machine learning model, the feedback signal by providing at least the second EMG signal as input to the machine learning model;
• and
• provide the feedback signal to the interactive system to cause the interactive system to perform a second function based at least in part on the feedback signal and the first function.

A computer-implemented method used in a distributed computer system, the method comprising acts of:

• measuring, by a speech input device wearable on a user, a first EMG signal produced by the user;
• determining a first prompt from the user based at least in part on the first EMG signal;
• providing the first prompt from the user to an interactive system to cause the interactive system to perform a first function of the interactive system;
• responsive to the interactive system performing the first function, determining a feedback signal by:
• receiving at least a second EMG signal from the speech input device, the second EMG signal indicative of a facial expression of the user response to the interactive system performing the first function;
• and
• generating, using a machine learning model, the feedback signal by providing at least the second EMG signal as input to the machine learning model;
• and
• providing the feedback signal to the interactive system to cause the interactive system to perform a second function based at least in part on the feedback signal and the first function.

A non-transitory computer-readable medium containing instruction that, when executed, cause at least one computer hardware processor to perform a method comprising acts of:

• measuring, by a speech input device wearable on a user, a first EMG signal produced by the user;
• determining a first prompt from the user based at least in part on the first EMG signal;
• providing the first prompt from the user to an interactive system to cause the interactive system to perform a first function of the interactive system;
• responsive to the interactive system performing the first function, determining a feedback signal by:
• receiving at least a second EMG signal from the speech input device, the second EMG signal indicative of a facial expression of the user response to the interactive system performing the first function;
• and
• generating, using a machine learning model, the feedback signal by providing at least the second EMG signal as input to the machine learning model;
• and
• providing the feedback signal to the interactive system to cause the interactive system to perform a second function based at least in part on the feedback signal and the first function.