Multimedia interface and method for computer system

An expert computer system, comprising:

• input means for entering user commands;
• text display means for displaying text messages;
• multimedia output means for presenting information to the user of said system using a plurality of output media in addition to said text display means;
• a knowledge base storing a specified set of information, including means for denoting text strings that can be displayed on the text display means and corresponding multimedia commands denoting additional information that can be presented by said multimedia output means;
• queue means for receiving and storing selected ones of said multimedia commands;
• inference software means, coupled to said input means, text display means, knowledge base and queue means, for accessing selected portions of the information in said knowledge base, responding to user commands from said input means, sending text strings corresponding to selected portions of said information to said text display means for display, and storing in said queue means selected multimedia commands corresponding to said selected portions of said information in said knowledge base;
• and
• dispatch means coupled to said queue means for delivering said multimedia commands stored in said queue means to said multimedia output means in response to a predefined dispatch command;
• wherein said inference software includes means for generating said predefined dispatch command at predefined times after accessing selected portions of the information in said knowledge base.

In an expert system having a knowledge base and inference software which accesses selected portions of the knowledge base, input means for user entry of commands, text display means for displaying text messages, the knowledge base including means for storing a multiplicity of text strings that can be displayed on the text display means, the inference software including means for responding to user commands from the input means and for displaying selected ones of said text strings stored in the knowledge base;

• the combination comprising:
• multimedia output means for presenting information to the user of said system using a plurality of output media in addition to said text display means;
• means for storing multimedia commands in the knowledge base along with corresponding ones of said text strings;
• said multimedia commands being embedded in said corresponding text strings;
• queue means for storing multimedia commands;
• output parsing means for removing and storing in said queue means said multimedia commands embedded in said text strings when said inference software selects corresponding portions of said knowledge base;
• and
• dispatch means coupled to said queue means for executing said multimedia commands stored in said queue means by delivering said multimedia commands to said multimedia output means and thereby presenting said information denoted by said multimedia commands via said multimedia output means;
• wherein said inference software includes means for generating a predefined dispatch command at predefined times after accessing a selected portion of the knowledge base.

A method of processing and presenting information in a computer system, said computer system including text display means for displaying text messages;

• the steps of the method comprising:
• providing multimedia output means for presenting information to the user of said system using a plurality of output media in addition to said text display means;
• running an application program on said computer system said application program generating text strings to be displayed on said text display means and corresponding multimedia commands denoting additional information that can be presented by said multimedia output means;
• displaying said text strings generated by said application program, and storing said multimedia commands generated by said application program in a queue;
• and
• delivering said stored multimedia commands to said multimedia output means after said text string displaying step.

A method of processing and presenting information in an expert computer system, said expert computer system including text display means for displaying text messages;

• the steps of the method comprising:
• providing multimedia output means for presenting information to the user of said system using a plurality of output media in addition to said text display means;
• providing a knowledge base storing a multiplicity of information records, each information record including means for denoting text strings that can be displayed on the text display means and corresponding multimedia commands denoting additional information that can be presented by said multimedia output means;
• receiving user commands;
• accessing selected information records in said knowledge base in response to said user commands;
• displaying said text strings denoted by said accessed information records on said text display means, and storing said multimedia commands denoted by said accessed information records in a queue;
• and
• delivering said stored multimedia commands to said multimedia output means after said text string displaying step.

A method of processing and presenting information in an expert computer system, said expert computer system including text display means for displaying text messages;

• the steps of the method comprising:
• providing multimedia output means, including video display means for displaying video images and voice generating means for generating spoken messages;
• providing a knowledge base storing a specified set of information including text strings that can be displayed on the text display means, and corresponding multimedia commands denoting video images and voice messages that can be presented by said video display means and said voice generating means;
• receiving user commands;
• accessing selected portions of the information in said knowledge base in response to said user commands;
• displaying on said display means text strings corresponding to said selected portions of the information in said knowledge base, and queuing corresponding ones of said multimedia commands denoting video images and voice messages stored in said knowledge base;
• said multimedia commands including distinct first and second sets of commands;
• delivering said first set of queued multimedia commands to said multimedia output means after displaying corresponding text strings;
• and
• delivering said second set of queued multimedia commands to said multimedia output means when a corresponding user command is received

A method of processing and presenting information in an expert computer system, said expert computer system including text display means for displaying test messages;

• the steps of the method comprising:
• providing multimedia output means, including video display means for displaying video images and voice generating means for generating spoken messages;
• providing a knowledge base storing a specified set of information related to a predefined set of tasks to be performed on a predefined set of equipment;
• said stored information including text strings, corresponding video images showing said set of equipment, and corresponding voice messages;
• receiving user commands;
• accessing selected portions of the information in said knowledge base in response to said user commands, including selected text strings, and corresponding video images and voice messages;
• and
• then displaying said selected text strings on said text display means, displaying corresponding video images on said video image means, and generating at least a portion of said corresponding selected voice messages with said voice generating means.

A method of processing and presenting information in an expert computer system, said expert computer system including text display means for displaying text messages;

• the steps of the method comprising:
• providing multimedia output means, including video display means for displaying video images and voice generating means for generating spoken messages;
• providing a knowledge base storing a specified set of information including text strings that can be displayed on the text display means, and corresponding multimedia commands denoting video images and voice messages that can be presented by said video display means and said voice generating means;
• said knowledge base storing information related to a predefined set of tasks to be performed on a predefined set of pieces of equipment;
• said video images denoted by multimedia commands including a multiplicity of images showing said set of pieces of equipment, including images showing said pieces of equipment in the context of said predefined set of tasks;
• said information stored by said knowledge base including text strings and voice messages related to said predefined set of tasks;
• receiving user commands;
• accessing selected portions of the information in said knowledge base in response to said user commands;
• displaying on said display means text strings corresponding to said selected portions of the information in said knowledge base, and queuing corresponding multimedia commands denoting video images and voice messages stored in said knowledge base;
• said multimedia commands including distinct first and second sets of commands;
• dispatching said first set of queued multimedia commands to said video display means and voice generating means at predefined times after displaying corresponding text strings;
• said dispatching step causing said expert system to simultaneously display text strings and video images related to selected ones of said predefined set of tasks;
• and dispatching said second set of queued multimedia commands to said video display means and voice generating means when a corresponding user command is received;
• said second dispatching step causing said expert system to provide additional explanatory material related to selected ones of said predefined set of tasks.

A method of processing and presenting information in an computer system, said computer system including text display means for displaying text messages;

• the steps of the method comprising:
• providing multimedia output means, including video display means for displaying video images and voice generating means for generating spoken messages;
• providing an application program which generates output strings, said output strings including text strings for display on said text display means, a multiplicity of said output strings including multimedia commands denoting video images and voice messages that can be presented by said video display means and said voice generating means;
• said multimedia commands including distinct first and second sets of commands;
• receiving user commands;
• displaying said text strings on said text display means;
• queuing said multimedia commands generated by said application program;
• and
• dispatching said first set of queued multimedia commands to said multimedia output means after displaying corresponding text strings;
• and
• dispatching said second set of queued multimedia commands to said multimedia output means when a corresponding user command is received.

In a computer system, the combination comprising:

• selecting means for selecting a piece of equipment from a predefined set of pieces of equipment;
• said selecting means including voice recognition means for receiving spoken user commands;
• video means for storing and displaying digitized video images corresponding to said selected piece of equipment, including video random access memory means for storing a multiplicity of digitized video images, each said stored image corresponding to one or more of said pieces of equipment and display means coupled to said video random access memory means for accessing and displaying at least one digitized video image stored in said video random access memory corresponding to said selected piece of equipment;
• and
• output means including text displaying means for displaying text corresponding to said displayed video images, and voice generating means for generating spoken messages corresponding to said display video images;
• whereby said computer system can be used to provide video and textual displays as well as spoken messages when working with a selected piece of equipment.

In a computer system, the combination comprising:

• means for defining a predefined set of pieces of equipment and a predefined set of maintenance tasks in conjunction with corresponding ones of said pieces of equipment;
• task specifying means for specifying one of said equipment maintenance tasks to be performed in conjunction with corresponding ones of said pieces of equipment;
• said task specifying means including voice recognition means for receiving spoken user commands;
• video means for storing and displaying one or more digitized video images corresponding to said selected piece of equipment, including video random access memory means for storing a multiplicity of digitized video images, each said stored image corresponding to one or more of said pieces of equipment, and display means for displaying a sequence of said stored digitized video images related to said specified equipment maintenance task;
• and
• text means for simultaneously displaying text corresponding to said video images, including means for displaying text related to said specified equipment maintenance task;
• whereby said computer system can be used to provide assistance in the form of video sequences and textual displays to a person performing a specified equipment maintenance task.

In a computer system, a method of obtaining information regarding various pieces of equipment, the steps of the method comprising:

• storing a multiplicity of digitized video images, each said stored image corresponding to one or more of said pieces of equipment from a predefined set of pieces of equipment;
• selecting a piece of equipment from said predefined set of pieces of equipment;
• said selecting step including the step of receiving spoken user commands;
• displaying at least one digitized video image stored in said video random access memory corresponding to said selected piece of equipment;
• displaying text corresponding to said displayed video images, and
• generating spoken messages corresponding to said displayed video images;
• whereby a person using said computer system receives information in the form of digitized video images, textual displays and spoken messages while working with a selected piece of equipment.