Method and system for assisting users in interacting with multi-modal dialog systems

#This application is related to U.S. patent application Ser. No. 11/004,339, filed on Dec. 3, 2004, entitled “Method and System for Generating Input Grammars for Multimodal Dialog Systems”.

#This invention is in the field of software, and more specifically in the field of software that assists users in interacting with multi-modal dialog systems.

#Dialog systems are systems that allow a user to interact with a computer system to perform tasks such as retrieving information, conducting transactions, and other such problem solving tasks. A dialog system can use several modalities for interaction. Examples of modalities include speech, gesture, touch, and handwriting. User-computer interactions in dialog systems are enhanced by employing multiple modalities in a coordinated manner. Dialog systems using multiple modalities for user-computer interaction are referred to as multi-modal dialog systems. The user interacts with a multi-modal dialog system by using multiple modalities that are coordinated as a dialog-based user interface. The user interface is referred to as a multi-modal dialog interface. A set of interactions of the user with the dialog system is referred to as a dialog. Each interaction is known as a turn of the dialog. The information provided by either the user or the dialog system is referred to as a context of the dialog.

#The present invention is illustrated by way of example, and not limitation, by the accompanying figures, in which like references indicate similar elements, and in which:

#FIG.1 is a block diagram of a multi-modal dialog system, in accordance with some embodiments of the present invention;

#FIG. 2 is a block diagram of an input processor in a multi-modal dialog system, in accordance with some embodiments of the present invention;

#Those skilled in the art will appreciate that the elements in the figures are illustrated for simplicity and clarity, and have not been necessarily drawn to scale. For example, the dimensions of some of the elements in the figures may be exaggerated, relative to other elements, for an improved perception of the embodiments of the present invention.

#Before describing in detail a method and system for assisting a user in interacting with a multi-modal dialog system, in accordance with the present invention, it should be observed that the present invention resides primarily in combinations of method steps and apparatus components related to multimodal dialog-based user interfaces. Apparatus components and method steps have been represented accordingly, where appropriate, by conventional symbols in the drawings. These drawings show only specific details that are pertinent for understanding the present invention, so as not to obscure the disclosure with details that will be apparent to those with ordinary skill in the art and the benefit of the description herein.

#Referring to FIG. 1, a block diagram shows a representative environment in which the present invention may be practiced, in accordance with some embodiments of the present invention. The representative environment includes an input-output module 102 and a multi-modal dialog system 104. The input-output module 102 receives user inputs and displays system outputs. In an embodiment of the invention, the input-output module 102 is a user interface such as a computer monitor, a touch screen, or a keyboard. A user interacts with the multi-modal dialog system 104 through the input-output module 102. The user provides an input by using various modalities through one or more input-output modules, of which one input-output module 102 is shown. Examples of various modalities include, but are not limited to, voice, gesture, and handwriting. In an embodiment of the invention, the user interacts with the multi-modal dialog system 104 by using one or more modalities in a coordinated manner. In other words, the multi-modal dialog system 104 uses a combined interpretation of user inputs provided through various modalities. For example, a user can say, “I want to find all the restaurants here”, touching an area on a map displayed on a touch screen. In this case, the multi-modal dialog system 104 combines the inputs provided through speech and gesture, to generate a combined interpretation. In another embodiment of the invention, in order to issue a correct query, the user may ask the multi-modal dialog system 104 how various modalities can be used in a coordinated manner. This type of question is referred to as a “What Can I Do?” (WCID) question.