# US 12,118,321 B2 · Collaborative plan-based dialogue system and method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,321 B2 - Document ID: US12118321B2 - Status: Granted - Kind: B2 - Application number: 18/397,749 - Filed: 2023-12-27 - Published: 2024-07-18 - Granted: 2024-10-15 - Earliest claimed: 2022-12-28 - Assignees: Openstream Inc. - Applicants: Openstream Inc. - Inventors: Philip R. Cohen; Lucian Galescu; Rajasekhar Tumuluri - Web page: https://patentgrep.com/patent/US12118321B2 ## CPC - G06F 40/35: Discourse or dialogue representation - G06F 40/30: Semantic analysis - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods and systems for multimodal collaborative conversational dialogue are disclosed. The multimodal collaborative conversational dialogue system include a multimodal avatar interface and one or more sensors, which obtains one or more multimodal inputs. A multimodal semantic parser generates one or more logical form representations based on the one or more multimodal inputs. A collaborative dialogue manager infers a goal of the user from the one or more logical form representations, and develops a plan including communicative actions and non-communicative actions with regard to the goal. The multimodal avatar interface outputs one or more multimodal collaborative plan-based dialogue system-generated communications with respect to execution of at least one communicative action. The collaborative dialogue manager maintains a collaborative dialogue with the user until obtainment of the goal. ## Claims ### Claim 1 A multimodal collaborative plan-based dialogue method comprising: obtaining, by one or more sensors in cooperation with a multimodal avatar interface in a multimodal collaborative plan-based dialogue system from a user, one or more multimodal inputs including at least one of utterances, speech, text, touch-based input, gestures, facial expressions, audio, video, body language, visual, body postures, eye gaze, and lip reading; generating, by a semantic parser and multimodal fuser in the multimodal collaborative plan-based dialogue system, one or more logical form representations based on the one or more multimodal inputs; inferring, by a dialogue manager in the multimodal collaborative plan-based dialogue system, a goal of the user from the one or more logical form representations; developing, by the dialogue manager, a plan including communicative actions and non-communicative actions with regard to the goal; outputting, by the multimodal avatar interface, one or more multimodal collaborative plan-based dialogue system-generated communications with respect to execution of at least one communicative action; and maintaining, by the dialogue manager, a collaborative dialogue via the obtaining, the generating, the inferring, the developing, and the outputting until obtainment of the goal. ### Claim 2 (depends on 1) The method of claim 1, further comprising: generating, by the dialogue manager, linguistic and multimodal communications with respect to the at least one communicative action; and wherein the outputting further comprises: outputting, by the multimodal avatar interface, the linguistic and multimodal communications. ### Claim 3 (depends on 1) The method of claim 1, wherein the maintaining further comprises: generating, by the dialogue manager, knowledge acquisition goals by processing one or more constraints and by applying domain rules-based reasoning. ### Claim 4 (depends on 3) The method of claim 3, wherein the generating knowledge acquisition goals further comprises: identifying, by the dialogue manager, entities. ### Claim 5 (depends on 4) The method of claim 4, wherein the generating knowledge acquisition goals further comprises: determining, by the dialogue manager, questions to ask the entities. ### Claim 6 (depends on 1) The method of claim 1, further comprising: generating, by the dialogue manager, facial gestures and rapport-utterances to signal one or more emotional states; and wherein the outputting further comprises: outputting, by the multimodal avatar interface, the facial gestures and rapport-utterances. ### Claim 7 (depends on 1) The method of claim 1, further comprising: generating, by the dialogue manager in cooperation with a reasoner in the multimodal collaborative plan-based dialogue system, empathic expressions by analyzing emotional cues with respect to the one or more multimodal inputs and the one or more logical form representations; and wherein the outputting further comprises: outputting, by the multimodal avatar interface, the empathic expressions. ### Claim 8 (depends on 1) The method of claim 1, wherein the maintaining further comprises: engaging, by the dialogue manager, in goal-oriented interactions to form and maintain self-beliefs and beliefs regarding user goals and intentions. ### Claim 9 (depends on 1) The method of claim 1, wherein the developing further comprises: generating, by the dialogue manager, epistemic goals, wherein the epistemic goals in conjunction with a model of the user's mental states support slot-filling processing in the multimodal collaborative plan-based dialogue system. ### Claim 10 (depends on 1) The method of claim 1, further comprising: generating, by the dialogue manager, proactive information communications to inform the user about information deemed relevant to the goal; and wherein the outputting further comprises: outputting, by the multimodal avatar interface, the proactive information communications. ### Claim 11 (depends on 1) The method of claim 1, further comprising: generating, by the dialogue manager, explanation communications to enhance user understanding of the plan; and wherein the outputting further comprises: outputting, by the multimodal avatar interface, explanation communications. ### Claim 12 (depends on 1) The method of claim 1, wherein the developing further comprises: generating, by the dialogue manager, hierarchical descriptions of actions incorporating conditional, sequential, disjunctive, and simultaneous actions; and wherein the outputting further comprises: outputting, by the multimodal avatar interface, at least a portion of the hierarchical descriptions in response to user inquiry. ### Claim 13 A multimodal collaborative plan-based dialogue system comprising: a multimodal avatar interface; one or more sensors, the one or more sensors and the multimodal avatar interface configured to obtain one or more multimodal inputs including at least one of utterances, speech, text, touch-based input, gestures, facial expressions, audio, video, body language, visual, body postures, eye gaze, and lip reading; a multimodal semantic parser configured to generate one or more logical form representations based on the one or more multimodal inputs; a collaborative dialogue manager configured to: infer a goal of a user from the one or more logical form representations; develop a plan including communicative actions and non-communicative actions with regard to the goal; the multimodal avatar interface configured to output one or more multimodal collaborative plan-based dialogue system-generated communications with respect to execution of at least one communicative action; and the collaborative dialogue manager configured to maintain a collaborative dialogue with the user until obtainment of the goal. ### Claim 14 (depends on 13) The system of claim 13, wherein the collaborative dialogue manager is further configured to generate linguistic and multimodal communications with respect to the at least one communicative action and the multimodal avatar interface is further configured to output the linguistic and multimodal communications. ### Claim 15 (depends on 13) The system of claim 13, wherein the collaborative dialogue manager is further configured to generate knowledge acquisition goals by processing one or more constraints and by applying domain rules-based reasoning, identify entities, and determine questions to ask the entities. ### Claim 16 (depends on 13) The system of claim 13, wherein: the collaborative dialogue manager is further configured to generate facial gestures and rapport-utterances to signal one or more emotional states and generate, in cooperation with a reasoner in the multimodal collaborative plan-based dialogue system, empathic expressions by analyzing emotional cues with respect to the one or more multimodal inputs and the one or more logical form representations; and the multimodal avatar interface is further configured to output the facial gestures and rapport-utterances and the empathic expressions. ### Claim 17 (depends on 13) The system of claim 13, wherein the collaborative dialogue manager is further configured to: engage in goal-oriented interactions to form and maintain self-beliefs and beliefs regarding user goals and intentions; and generate epistemic goals, wherein the epistemic goals in conjunction with a model of the user's mental states support slot-filling processing in the multimodal collaborative plan-based dialogue system. ### Claim 18 (depends on 13) The system of claim 13, wherein the collaborative dialogue manager is further configured to generate proactive information communications to inform the user about information deemed relevant to the goal and the multimodal avatar interface is further configured to output the proactive information communications. ### Claim 19 (depends on 13) The system of claim 13, wherein the collaborative dialogue manager is further configured to generate explanation communications to enhance user understanding of the plan and the multimodal avatar interface is further configured to output the explanation communications. ### Claim 20 (depends on 13) The system of claim 13, wherein the collaborative dialogue manager is further configured to generate hierarchical descriptions of actions incorporating conditional, sequential, disjunctive, and simultaneous actions and the multimodal avatar interface is further configured to output at least a portion of the hierarchical descriptions in response to user inquiry.