System and methods for providing a live query response session

A computing system, comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that store instructions wherein, when executed by the one or more processors, the instructions cause the one or more processors to perform operations, the operations comprising:
• initiating a live query response session with a computing device associated with a user;
• while conducting the live query response session:
• receiving a multimodal input query from the computing device, wherein the multimodal input query includes video content;
• determining, by an orchestration model, one or more information retrieval steps to acquire additional information associated with the multimodal input query, wherein the information retrieval steps include a clarifying question requesting additional information from the user;
• providing the clarifying question to the user computing device;
• receiving a user response to the clarifying question, wherein the user response is received in real-time;
• generating model input based on the multimodal input query and the user response;
• processing the model input with a query response model to generate a model output based on the model input;
• transmitting the model output for display at a user computing device;
• receiving, by the computing device, audio content from the user computing device;
• and
• determining, by the computing device, that the audio content contains user feedback with respect to the multimodal input query.

One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:

• initiating a live query response session with a computing device associated with a user;
• while conducting the live query response session:
• receiving a multimodal input query from the computing device, wherein the multimodal input query includes video content;
• determining, by an orchestration model, one or more information retrieval steps to acquire additional information associated with the multimodal input query, wherein the information retrieval steps include a clarifying question requesting additional information from the user;
• providing the clarifying question to the user computing device;
• receiving a user response to the clarifying question, wherein the user response is received in real-time;
• generating model input based on the multimodal input query and the user response;
• processing the model input with a query response model to generate a model output based on the model input;
• transmitting the model output for display at a user computing device;
• receiving, by the computing device, audio content from the user computing device;
• and
• determining, by the computing device, that the audio content contains user feedback with respect to the multimodal input query.