US 2024/0242035 A1Published application

Collaborative plan-based dialogue system and method

AssigneeOpenstream Inc.

InventorsPhilip R. Cohen, Lucian Galescu, Rajasekhar Tumuluri

Life of this patent

Estimating the term…

Loading patent…

Abstract

Methods and systems for multimodal collaborative conversational dialogue are disclosed. The multimodal collaborative conversational dialogue system include a multimodal avatar interface and one or more sensors, which obtains one or more multimodal inputs. A multimodal semantic parser generates one or more logical form representations based on the one or more multimodal inputs. A collaborative dialogue manager infers a goal of the user from the one or more logical form representations, and develops a plan including communicative actions and non-communicative actions with regard to the goal. The multimodal avatar interface outputs one or more multimodal collaborative plan-based dialogue system-generated communications with respect to execution of at least one communicative action. The collaborative dialogue manager maintains a collaborative dialogue with the user until obtainment of the goal.

Claims20

Dependent claims

A multimodal collaborative plan-based dialogue method comprising:

obtaining, by one or more sensors in cooperation with a multimodal avatar interface in a multimodal collaborative plan-based dialogue system from a user, one or more multimodal inputs including at least one of utterances, speech, text, touch-based input, gestures, facial expressions, audio, video, body language, visual, body postures, eye gaze, and lip reading;
generating, by a semantic parser and multimodal fuser in the multimodal collaborative plan-based dialogue system, one or more logical form representations based on the one or more multimodal inputs;
inferring, by a dialogue manager in the multimodal collaborative plan-based dialogue system, a goal of the user from the one or more logical form representations;
developing, by the dialogue manager, a plan including communicative actions and non-communicative actions with regard to the goal;
outputting, by the multimodal avatar interface, one or more multimodal collaborative plan-based dialogue system-generated communications with respect to execution of at least one communicative action;
and
maintaining, by the dialogue manager, a collaborative dialogue via the obtaining, the generating, the inferring, the developing, and the outputting until obtainment of the goal.

4 more elements

A system multimodal collaborative plan-based dialogue comprising:

a multimodal avatar interface;
one or more sensors, the one or more sensors and the multimodal avatar interface configured to obtain one or more multimodal inputs including at least one of utterances, speech, text, touch-based input, gestures, facial expressions, audio, video, body language, visual, body postures, eye gaze, and lip reading;
a multimodal semantic parser configured to generate one or more logical form representations based on the one or more multimodal inputs;
a collaborative dialogue manager configured to:
infer a goal of the user from the one or more logical form representations;
develop a plan including communicative actions and non-communicative actions with regard to the goal;
the multimodal avatar interface configured to output one or more multimodal collaborative plan-based dialogue system-generated communications with respect to execution of at least one communicative action;
and
the collaborative dialogue manager configured to maintain a collaborative dialogue with the user until obtainment of the goal.

6 more elements

Dependencies are inferred from the claim text.