Sensor enhanced speech recognition

A method, comprising:

• obtaining visual content associated with a user and an environment of the user;
• obtaining, from the visual content, metadata associated with the user and the environment of the user;
• determining, based on the visual content or the metadata, if the user is speaking;
• determining, based on the visual content and the metadata, a type of a first device being utilized in the environment;
• determining, based on the visual content and the metadata, a distance between the first device and the user;
• identifying, based on the visual content and the metadata, an interferer and a location of the interferer in the environment;
• obtaining, if the user is determined to be speaking, audio content associated with the user and the environment;
• adapting an acoustic model corresponding to the user and the environment based on the visual content, the audio content, the metadata, and the type of the first device being utilized in the environment;
• enhancing, by utilizing the acoustic model and by utilizing instructions from memory that are executed by a processor, a speech recognition process utilized for processing speech of the user within the audio content;
• cancelling, after identifying the interferer and the location of the interferer in the environment and by utilizing an audio profile corresponding to the interferer, noise generated by the interferer that interferes with the speech of the user;
• adjusting, based on the distance between the first device and the user, a ratio of direct to indirect audio signals occurring in the environment and a level of reverberation associated with the audio content in the environment;
• adapting, based on a user profile of the user, a user interface of an application executing the speech recognition process utilized for processing the speech of the user, wherein the user interface is adapted based on the visual content, the audio content, and the metadata, wherein adapting the user interface of the application comprises enhancing a visual aspect of the user interface itself to reflect a characteristic of the user, a location of the user, and an interest of the user;
• and
• adjusting, based on the user profile and a location profile corresponding to the location of the user, a feature of the application executing the speech recognition process so as to tailor the application to the user.

A non-transitory computer-readable device comprising instructions, which when executed by a processor, cause the processor to perform operations comprising:

• obtaining visual content associated with a user and an environment of the user;
• obtaining, from the visual content, metadata associated with the user and the environment of the user;
• determining, based on the visual content or the metadata, if the user is speaking;
• determining, based on the visual content and the metadata, a type of a first device being utilized in the environment;
• determining, based on the visual content and the metadata, a distance between the first device and the user;
• identifying, based on the visual content and the metadata, an interferer and a location of the interferer in the environment;
• obtaining, if the user is determined to be speaking, audio content associated with the user and the environment;
• adapting an acoustic model corresponding to the user and the environment based on the visual content, the audio content, the metadata, and the type of the first device being utilized in the environment;
• enhancing, by utilizing the acoustic model, a speech recognition process utilized for processing speech of the user within the audio content;
• cancelling, after identifying the interferer and the location of the interferer in the environment and by utilizing an audio profile corresponding to the interferer, noise generated by the interferer that interferes with the speech of the user;
• adjusting, based on the distance between the first device and the user, a ratio of direct to indirect audio signals occurring in the environment and a level of reverberation associated with the audio content in the environment;
• adapting, based on a user profile of the user, a user interface of an application executing the speech recognition process utilized for processing the speech of the user, wherein the user interface is adapted based on the visual content, the audio content, and the metadata, wherein adapting the user interface of the application comprises enhancing a visual aspect of the user interface itself to reflect a characteristic of the user, a location of the user, and an interest of the user;
• and
• adjusting, based on the user profile and a location profile corresponding to the location of the user, a feature of the application executing the speech recognition process so as to tailor the application to the user.