# US 7,610,556 B2 · Dialog manager for interactive dialog with computer user

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 7,610,556 B2
- Document ID: US7610556B2
- Status: Granted
- Kind: B2
- Application number: 10/099,348
- Filed: 2002-03-14
- Published: 2003-07-24
- Granted: 2009-10-27
- Earliest claimed: 2001-12-28
- Assignees: Microsoft Corporation
- Inventors: Baining Guo; Bo Zhang; Heung-Yeung Shum
- Web page: https://patentgrep.com/patent/US7610556B2

## CPC

- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A dialog manager module accesses a dialog model, a discourse model and a domain model to render information to a computer user through the course of a dialog exchange with the computer user. Actions are performed according to a set of rules during the dialog. Each of the rules comprise an action to perform as a function of the current dialog state and a filled form of slots of semantic information of the user input.

## Claims

### Claim 1

A method of providing a dialog interface between a computer system and a user, the method comprising:
(a) rendering information to the user if a dialog does not exist between the computer system and the user to initiate a dialog with the user using the computer system by accessing a task model defining tasks in a hierarchical structure, wherein at least some of the tasks comprise sub-tasks related to actions for rendering the information to the user, and wherein the rendering of the information to the user includes sequentially rendering the sub-tasks to complete a larger task;
(b) receiving user input;
(c) ascertaining semantic information of the user input;
(d) ascertaining a current dialog state as a function of the semantic information;
(e) adjusting a user preference as a function of the dialog with the user; and
(f) repeating at least steps (b) and (c) and rendering information to the user according to both a verbal action and a non-verbal action related to the verbal action when a rule of a set of rules has been met, each rule of the set of rules comprising a verbal action and a non-verbal action to be performed as a function of the current dialog state, a filled form of slots of semantic information of the user input, and the user preference, wherein the information to be rendered in step (f) is prioritized, and wherein the user preference indicates a level of priority of information to be rendered.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1 wherein rendering information to the user if a dialog does not exist includes suspending rendering of information according to a task when a dialog begins with the user and resuming rendering according to the task when the dialog ends.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1 wherein adjusting the user preference includes monitoring a number of interruptions made by the user during rendering.

### Claim 4 (depends on 1)

The method of claim 1 wherein non-verbal actions include at least one of an image and a video.

### Claim 5 (depends on 4)

The method of claim 4 wherein rendering information according to both the verbal action and the non-verbal action includes rendering a representation of a talking head performing the verbal action and rendering the at least one of an image and a video in a content window.

### Claim 6

A computer readable storage medium having instructions, which when executed on a computer provide a user interface between a user and the computer, the instructions comprising:
a user input module adapted to receive input from the user and provide semantic information as a function of the input;
a dialog model defining a set of rules, each rule comprising an action to be performed as function of received input when a dialog exists with the user;
a discourse model defining if a dialog exists between the computer and the user as a function of received input;
a domain model defining information to be rendered to the user, wherein the information to be rendered is prioritized;
a task model defining some of the information to be rendered to the user as tasks, the task model being organized in a hierarchical manner wherein at least some tasks comprise a plurality of sub-tasks related to actions for rendering information and wherein the sub-tasks include low priority information that can be skipped from rendering as a function of user input;
a user model defining user preferences related to rendering information; and
a module receiving the user input and accessing the discourse model, the dialog model, the task model and the domain model to render information that includes audio and multi-media information related to the audio according to a rule if dialog exists with the user and according to a task by initiating dialog with the user using the computer if dialog does not exist between the computer and the user, wherein the module also accesses the user model for rendering the information as a function of the user preferences and for adjusting the user preferences as a function of the dialog with the user, and wherein the user preferences indicate a level of priority of information to be rendered.

### Claim 7 (depends on 6)

The computer readable storage medium of claim 6 and further comprising a parser receiving the user input and ascertaining semantic information pertaining to the user input, the module rendering information as a function of the semantic information.

### Claim 8 (depends on 6)

The computer readable storage medium of claim 6 wherein the module sequentially renders information defined by the sub-tasks in order to complete a larger task.

### Claim 9 (depends on 8)

The computer readable storage medium of claim 8 wherein the module suspends rendering information defined by a task in the task model when using a rule of the dialog model, and wherein the module resumes rendering information defined by the task when at least some of the rules of the dialog model do not apply.

### Claim 10 (depends on 9)

The computer readable storage medium of claim 9 wherein each sub-task comprises at least one of an action for rendering information and another sub-task.

### Claim 11 (depends on 6)

The computer readable storage medium of claim 6 wherein the module adjusts the user preference related to the level of priority as a function of interruptions made by the user.

### Claim 12 (depends on 6)

The computer readable storage medium of claim 6 wherein the information to be rendered includes stories, and wherein one of the user preferences relates to a frequency of stories to be rendered.

### Claim 13 (depends on 12)

The computer readable storage medium of claim 12 wherein the module adjusts the user preference related to the frequency of stories to be rendered as a function of at least one of responses received from the user as to whether a story should be rendered and interruptions made by the user during telling a story.

### Claim 14 (depends on 6)

The computer readable storage medium of claim 6 wherein at least some of the actions comprise rendering information to provoke a question from the user, and wherein one of the user preferences relates to a frequency for rendering information to provoke a question from the user.

### Claim 15 (depends on 6)

The computer readable storage medium of claim 6 wherein the rules of the dialog model are hierarchical.

### Claim 16 (depends on 15)

The computer readable storage medium of claim 15 wherein at least some of the rules pertain to sub-dialogs.

### Claim 17 (depends on 6)

The computer readable storage medium of claim 6 wherein information of the domain is hierarchically organized and classified into portions, wherein at least some of the portions include a plurality of properties about the information of the portions.

### Claim 18 (depends on 6)

The computer readable storage medium of claim 6 wherein the multi-media information includes at least one of images and video.

### Claim 19

A computer readable storage medium having instructions, which when executed on a computer provide a user interface between a user and the computer, the instructions comprising:
a user input module adapted to receive input from the user and provide semantic information as a function of the input;
a user model defining user preferences related to a level of priority for rendering information;
a dialog model defining a set of rules, each rule comprising an action to be performed as function of received input pertaining to semantic information and a dialog state with the user;
a discourse model defining a current dialog state with the user as a function of received input pertaining to semantic information;
a domain model defining information to be rendered to the user as audio responses and visual information related to the audio responses, each response having a parameter indicative of high priority information and low priority information;
a module receiving the semantic information and corresponding input and accessing the discourse model, the dialog model and the domain model to ascertain the current dialog state and perform an action to render selected audio responses to the user, wherein the module adjusts the user preferences as a function of the dialog with the user, and wherein the module renders high priority information and skipping low priority information from the audio responses as a function of the user preferences; and
a video rendering module rendering the visual information related to the selected audio responses.

### Claim 20 (depends on 19)

The computer readable storage medium of claim 19 wherein the user input module comprises a speech recognizer and a parser.

### Claim 21 (depends on 19)

The computer readable storage medium of claim 19 wherein the module adjusts the user preference related to the level of priority as a function of interruptions made by the user.

### Claim 22 (depends on 19)

The computer readable storage medium of claim 19 wherein the information to be rendered includes stories, and wherein one of the user preferences relates to a frequency of stories to be rendered.

### Claim 23 (depends on 22)

The computer readable storage medium of claim 22 wherein the module adjusts the user preference related to the frequency of stories to be rendered as a function of at least one of responses received from the user as to whether a story should be rendered and interruptions made by the user during telling a story.

### Claim 24 (depends on 19)

The computer readable storage medium of claim 19 wherein at least some of the actions comprise rendering information to provoke a question from the user, and wherein one of the user preferences relates to a frequency for rendering information to provoke a question from the user.

### Claim 25 (depends on 19)

The computer readable storage medium of claim 19 wherein the semantic information of at least some of the rules comprises a plurality of semantic types.

### Claim 26 (depends on 25)

The computer readable storage medium of claim 25 wherein the rules of the dialog model are hierarchical.

### Claim 27 (depends on 26)

The computer readable storage medium of claim 26 wherein at least some of the rules pertain to sub-dialogs.

### Claim 28 (depends on 19)

The computer readable storage medium of claim 19 wherein information of the domain is hierarchically organized and classified into portions, wherein at least some of the portions include a plurality of properties about the information of the portions.

### Claim 29 (depends on 28)

The computer readable storage medium of claim 28 and further comprising:
a store having data related to the properties of the portions;
a speech synthesizer; and
wherein the module accesses the store as a function of access of the domain model
to obtain data to be provided to the speech synthesizer for speech synthesis.

### Claim 30 (depends on 29)

The computer readable medium of claim 29 wherein the video rendering module renders a representation of a talking head having a talking state with mouth movements in accordance with the selected audio responses.

### Claim 31 (depends on 30)

The computer readable storage medium of claim 30 wherein the video rendering module renders visual information in a content window separate from the representation of the talking head.

### Claim 32 (depends on 30)

The computer readable storage medium of claim 30 wherein the video rendering module renders the representation of the talking head in a waiting state with movements in accordance with listening.

### Claim 33 (depends on 32)

The computer readable storage medium of claim 32 wherein the video rendering module renders a sequence of video frames having the talking head.

### Claim 34 (depends on 33)

The computer readable storage medium of claim 33 wherein the video rendering module continuously renders the video frames having the talking head with non-talking mouth movements during the waiting state and adds a talking mouth position to each of the frames during the talking state.

### Claim 35 (depends on 19)

The computer readable storage medium of claim 19 and further comprising:
a task model defining some of the information to be rendered to the user as a function of a topic, wherein the module accesses the task model to ascertain information to be rendered if a rule of the dialog model is not being used.

### Claim 36 (depends on 35)

The computer readable storage medium of claim 35 wherein the task model is organized in a hierarchical manner wherein at least some tasks comprise a plurality of sub-tasks related to actions for rendering information, and wherein the module sequentially renders information defined by the sub-tasks in order to complete a larger task.

### Claim 37 (depends on 36)

The computer readable storage medium of claim 36 wherein the module suspends rendering information defined by a task in the task model when using a rule of the dialog model, and wherein the module resumes rendering information defined by the task when at least some of the rules of the dialog model do not apply.

### Claim 38 (depends on 37)

The computer readable storage medium of claim 37 wherein each sub-task comprises at least one of an action for rendering information and another sub-task.