Using multiple modality input to feedback context for natural language understanding

A system for providing input context to a speech recognizer, the system comprising:

• at least one processor;
• and
• a memory operatively connected to the at least one processor, the memory storing instructions that when executed by the at least one processor perform operations comprising:
• training a statistical dialog manager using training data including user data collected during a previous search session;
• receiving a spoken query for a first input field;
• receiving an input to a second input field of an interactive input form of a computer device;
• identifying, by a machine learning model, a category for the spoken query based on a context of at least a value of the received input in the second input field, wherein the first input field and the second input field are distinct, and the machine learning model is trained based on the input to the second input field and spoken words received while interactively outputting content, and the machine learning model predicts the context as the category for the spoken query based on the data input and content of a display;
• construct a statistical dialog manager according to the identified category, wherein the statistical dialogue language model comprises a weight of a word in the identified category;
• and
• converting the spoken query to generate a text according to a trained statistical dialog manager according to the category based on statistically weight terms belonging to the category.

A storage device, having computer-executable instructions that, when executed by at least one processor, executes operations comprising:

• determining, by a machine learning model, based on contextual data associated with information, a plurality of categories comprising a category, wherein the category includes a subcategory, wherein the information includes an element, the subcategory is associated with the element in the information, the machine learning model is trained based on previously received input while interactively outputting content, and the machine learning model predicts a context as at least the category of the contextual data;
• building a first and second statistical dialog managers, wherein a first statistical dialog manager is associated with the category and a second statistical dialog manager is associated with the subcategory of the category, wherein the first statistical dialogue manager comprises a weight of a word in the category;
• receiving a spoken query as user input associated with the information in a first input field;
• receiving an input in a second input field of an interactive input form of a computer device;
• identifying, by the machine learning model, the category for the spoken query based on the context of at least a value of the received input in the second input field, wherein the first input field and the second input field are distinct;
• determining, by the machine learning model, the subcategory for the spoken query as user input;
• and
• converting the spoken query to generate a response text according to the first statistical dialog manager according to the category and the second statistical dialog manager according to the subcategory of the category.