Multimodal collaborative plan-based dialogue
AssigneeOpenstream Inc.
InventorsPhilip R. Cohen, Rajasekhar Tumuluri
Life of this patent
Estimating the term…
Abstract
Methods and systems for multimodal collaborative plan-based dialogue. The multimodal collaborative plan-based dialogue system includes multiple sensors to detect multimodal inputs from a user. The multimodal collaborative plan-based dialogue system includes a multimodal sematic parser that generates logical forms based on the multimodal inputs. The multimodal collaborative plan-based dialogue system includes a dialogue manager that infers a goal of the user from the logical forms and develops a plan that includes communicative actions with regard to the goal.
Claims20
A multimodal collaborative plan-based dialogue method comprising:
- obtaining, by one or more sensors in cooperation with a presentation tool, multimodal inputs including at least one of utterances, speech, text, touch-based input, gestures, facial expressions, audio, video, body language, visual, body postures, eye gaze, and lip reading;
- generating, by a multimodal semantic parser in a multimodal collaborative plan-based dialogue system, one or more logical forms based on the one or more multimodal inputs;
- inferring, by a dialogue manager in the multimodal collaborative plan-based dialogue system, a goal of a user from the one or more logical forms;
- developing, by the dialogue manager, a plan including communicative actions with regard to the goal;
- generating, by the dialogue manager, hierarchical descriptions of simultaneous actions, conditional actions, sequential actions, and disjunctive actions;
- outputting, by the presentation tool, one or more multimodal collaborative plan-based dialogue system-generated responses with respect to execution of at least one communicative action;
- outputting, by the presentation tool, at least a portion of the hierarchical descriptions;
- and
- maintaining, by the dialogue manager, a collaborative dialogue via the obtaining, the generating, the inferring, the developing, and the outputting until obtainment of the goal.
A system for multimodal collaborative plan-based dialogue, the system comprising:
- a robot;
- one or more sensors, the one or more sensors and the robot configured to obtain one or more multimodal inputs including at least one of utterances, speech, text, touch-based input, gestures, facial expressions, audio, video, body language, visual, body postures, eye gaze, and lip reading;
- a multimodal semantic parser configured to generate one or more logical forms based on the one or more multimodal inputs;
- a dialogue manager configured to:
- infer a goal of a user from the one or more logical forms;
- develop a plan including communicative actions with regard to the goal;
- and
- generate hierarchical descriptions of actions incorporating simultaneous actions, conditional actions, sequential actions, and disjunctive actions;
- the robot configured to output:
- one or more multimodal collaborative plan-based dialogue system-generated responses with respect to execution of at least one communicative action;
- and
- at least a portion of the hierarchical descriptions in response to user inquiry;
- and
- the dialogue manager configured to maintain a collaborative dialogue with the user until obtainment of the goal.
Dependencies are inferred from the claim text.