Intelligent multi modal communications apparatus utilizing predetermined rules to choose optimal combinations of input and output formats

A multimodal interaction apparatus for performing a predetermined process by interacting with a user and outputting a result of the process, said multimodal interaction apparatus comprising:

• an input device, having plural input means, for obtaining information from the user, interpreting the information input from at least one input means selected from said plural input means in accordance with an input-output media allocation and generating corresponding information in accordance with a result of interpretation;
• problem solving means for performing the predetermined process in accordance with the corresponding information and outputting a result of the process;
• an output device, having plural output means, for outputting a result of the process output from said problem solving means from at least one output means selected from the plural output means in accordance with the input-output media allocation;
• and
• media allocation means for performing the input-output media allocation by interacting with the user, said input-output media allocation including
• (i) at least one of a selection of a corresponding information to be supplied from said input means to said problem solving means from corresponding information generated for each of the input media and a combination of the corresponding information generated from said problem solving means for each of the input media, and
• (ii) to obtain an information output from said problem solving means to said output device, at least one of a selection of an information from said problem solving means and a combination of the selected information,
• said media allocation means including a media allocation history slot for storing a series of input-output media allocation information which have been selected, when the input-output media allocation has been changed, selected input-output media allocation information is added to said media allocation history slot, and when an instruction to change the input-output media allocation has been issued, a new media allocation is performed by using the input-output media allocation information recorded in said media allocation slot.

A multimodal interaction apparatus for performing a predetermined process by interacting with a user and outputting a result of the process, said multimodal interaction apparatus comprising:

• an input device, having plural input means, for obtaining information from the user, interpreting the information input from at least one input means selected from said plural input means in accordance with an input-output media allocation and generating corresponding information in accordance with a result of interpretation;
• problem solving means for performing the predetermined process in accordance with the corresponding information and outputting a result of the process;
• an output device, having plural output means, for outputting a result of the process output from said problem solving means from at least one output means selected from the plural output means in accordance with the input-output media allocation;
• interaction control means for changing the input-output media allocation in accordance with the result of the process performed by said problem solving means and controlling input and output to be performed by said input means and said output means;
• and
• media allocation means for performing the input-output media allocation by interacting with the user, said input-output media allocation
• (i) to obtain an information to be supplied to said input device from information input from the user, selecting a combination of said input means and a data input method, and
• (ii) to obtain an expression instructing the user from the output of said problem solving means, selecting a combination of said output means and a data output method,
• said interaction control means including means for monitoring an operating state of at least one of said input means and said output means to supply problem information indicating generation of a problem to said media allocation means when the problem inhibiting use of at least one of the input media allocation and the output media allocation which have been selected and used occurs, and
• said media allocation means including means for changing the input-output media allocation when the problem information is supplied to said media allocation means.

A multimodal interaction apparatus for performing a predetermined process by interacting with a user and outputting a result of the process, said multimodal interaction apparatus comprising:

• an input device, having plural input means, for obtaining information from the user, interpreting the information input from at least one input means selected from said plural input means in accordance with an input-output media allocation and generating corresponding information in accordance with a result of interpretation;
• problem solving means for performing the predetermined process in accordance with the corresponding information and outputting a result of the process;
• an output device, having plural output means, for outputting a result of the process output from said problem solving means from at least one output means selected from the plural output means in accordance with the input-output media allocation;
• interaction control means for changing the input-output media allocation in accordance with the result of the process performed by said problem solving means and controlling input and output to be performed by said input means and said output means;
• and
• media allocation means for performing the input-output media allocation by interacting with the user, said input-output media allocation
• (i) to obtain an information to be supplied to said input device from information input from the user, selecting a combination of said input means and a data input method, and
• (ii) to obtain an expression instructing the user from the output of said problem solving means, selecting a combination of said output means and a data output method,
• said problem solving means including means for supplying an input request to said interaction control means when information required to solve a problem of the system is needed to be acquired from the user so as to urge the user to input a required information, and
• said media allocation means including means for performing new media allocation by using said required information when an instruction for performing a process for changing the input-output media allocation has been supplied.

A multimodal interaction apparatus for performing a predetermined process by interacting with a user and outputting a result of the process, said multimodal interaction apparatus comprising:

• an input device, having plural input means for obtaining, information from the user, interpreting the information input from at least one input means selected from said plural input means in accordance with an input-output media allocation and generating corresponding information in accordance with a result of interpretation;
• problem solving means for performing the predetermined process in accordance with the corresponding information and outputting a result of the process;
• an output device, having plural output means, for outputting a result of the process output from said problem solving means from at least one output means selected from the plural output means in accordance with the input-output media allocation;
• interaction control means for changing the input-output media allocation in accordance with the result of the process performed by said problem solving means and controlling input and output to be performed by said input means and said output means;
• and
• media allocation means for performing the input-output media allocation by interacting with the user, said input-output media allocation
• (i) to obtain an information to be supplied to said input device from information input from the user, selecting a combination of said input means and a data input method, and
• (ii) to obtain an expression instructing the user from the output of said problem solving means, selecting a combination of said output means and a data output method,
• said interaction control means including means for supplying an output request for requiring said problem solving means to output information required to constitute an output to the user in accordance with the input-output media allocation in which said output device have been set and requiring said problem solving means to output a required information, and
• said media allocation means including means for performing new media allocation by using said requested information when an instruction for performing a process for changing the input-output media allocation have been supplied.

A multimodal interaction apparatus for performing a predetermined process by interacting with a user and outputting a result of the process, said multimodal interaction apparatus comprising:

• an input device, having plural input means, for obtaining information from the user, interpreting the information input from at least one input means selected from said plural input means in accordance with an input-output media allocation and generating corresponding information in accordance with a result of interpretation;
• problem solving means for performing the predetermined process in accordance with the corresponding information and outputting a result of the process;
• an output device, having plural output means, for outputting a result of the process output from said problem solving means from at least one output means selected from the plural output means in accordance with the input-output media allocation;
• interaction control means for changing the input-output media allocation in accordance with the result of the process performed by said problem solving means and controlling input and output to be performed by said input means and said output means;
• and
• media allocation means for performing the input-output media allocation by interacting with the user, said input-output media allocation
• (i) to obtain an information to be supplied to said input device from information input from the user, selecting a combination of said input means and a data input method, and
• (ii) to obtain an expression instructing the user from the output of said problem solving means, selecting a combination of said output means and a data output method,
• said media allocation means including a media allocation slot for recording an input-output media allocation information which are selected at present, and when an instruction to change the input-output media allocation has been issued, a new media allocation is performed by using the input-output media allocation information recorded in said media allocation slot.

A multimodal interaction apparatus for performing a predetermined process by interacting with a user and outputting a result of the process, said multimodal interaction apparatus comprising:

• an input device, having plural input means, for obtaining information from the user, interpreting the information input from at least one input means selected from said plural input means in accordance with an input-output media allocation and generating corresponding information in accordance with a result of interpretation;
• problem solving means for performing the predetermined process in accordance with the corresponding information and outputting a result of the process;
• an output device, having plural output means, for outputting a result of the process output from said problem solving means from at least one output means selected from the plural output means in accordance with the input-output media allocation;
• interaction control means for changing the input-output media allocation in accordance with the result of the process performed by said problem solving means and controlling input and output to be performed by said input means and said output means;
• and
• media allocation means for performing the input-output media allocation by interacting with the user, said input-output media allocation
• (i) to obtain an information to be supplied to said input device from information input from the user, selecting a combination of said input means and a data input method, and
• (ii) to obtain an expression instructing the user from the output of said problem solving means, selecting a combination of said output means and a data output method,
• said media allocation means including a media allocation history slot for storing a series of input-output media allocation information which have been selected, when the input-output media allocation has been changed, selected input-output media allocation information is added to said media allocation history slot, and when an instruction to change the input-output media allocation has been issued, a new media allocation is performed by using the input-output media allocation information recorded in said media allocation slot.

A multimodal interaction apparatus comprising:

• problem solving means for subjecting an input to a calculation process to satisfy an object of the system and outputting a result of process;
• an input device, having plural input means, for interpreting an information input from at least one input means selected from said plural input means in accordance with an input-output media allocation and supplying an information corresponding to a result of interpretation to said problem solving means;
• an output device, having plural output means, for outputting a result of the process output from said problem solving means from at least one output means selected from the plural output means in accordance with the input-output media allocation;
• interaction control means for changing the input-output media allocation in accordance with the result of the process output from said problem solving means and controlling input and output to be performed by said input means and said output means;
• and
• media allocation means for performing the input-output media allocation by interacting with the user, said input-output media allocation including
• (i) at least one of a selection of a corresponding information to be supplied from said input means to said problem solving means from corresponding information generated for each of the input media and a combination of the corresponding information generated from said problem solving means for each of the input media, and
• (ii) to obtain an expression instructing the user from the output of said problem solving means, at least one of a selection of an information from said problem solving means and a combination of the selected information, and said media allocation means performing new media allocation when an instruction to change the input-output media allocation has been issued,
• said interaction control means including means for monitoring an operating state of at least one of said input means or said output means so as to supply problem information indicating generation of a problem to said media allocation means when the problem inhibiting use of the input media allocation or the output media allocation which have been selected and used occurs, and
• said media allocation means including means for changing the input-output media allocation when the problem information is supplied to said media allocation means.

A multimodal interaction apparatus comprising:

• problem solving means for subjecting an input to a calculation process to satisfy an object of the system and outputting a result of process;
• an input device, having plural input means, for interpreting an information input from at least one input means selected from said plural input means in accordance with an input-output media allocation and supplying an information corresponding to a result of interpretation to said problem solving means;
• an output device, having plural output means, for outputting a result of the process output from said problem solving means from at least one output means selected from the plural output means in accordance with the input-output media allocation;
• interaction control means for changing the input-output media allocation in accordance with the result of the process output from said problem solving means and controlling input and output to be performed by said input means and said output means;
• and
• media allocation means for performing the input-output media allocation by interacting with the user, said input-output media allocation including
• (i) at least one of a selection of a corresponding information to be supplied from said input means to said problem solving means from corresponding information generated for each of the input media and a combination of the corresponding information generated from said problem solving means for each of the input media, and
• (ii) to obtain an expression instructing the user from the output of said problem solving means, at least one of a selection of an information from said problem solving means and a combination of the selected information, and said media allocation means performing new media allocation when an instruction to change the input-output media allocation has been issued,
• said problem solving means including means for supplying an input request to said interaction control means when information required to solve a problem of the system is needed to be acquired from the user so as to urge the user to input a required information, and
• said media allocation means including means for performing new media allocation by using said required information when an instruction for performing a process for changing the input-output media allocation has been supplied.

A multimodal interaction apparatus comprising:

• problem solving means for subjecting an input to a calculation process to satisfy an object of the system and outputting a result of process;
• an input device, having plural input means, for interpreting an information input from at least one input means selected from said plural input means in accordance with an input-output media allocation and supplying an information corresponding to a result of interpretation to said problem solving means;
• an output device, having plural output means, for outputting a result of the process output from said problem solving means from at least one output means selected from the plural output means in accordance with the input-output media allocation;
• interaction control means for changing the input-output media allocation in accordance with the result of the process output from said problem solving means and controlling input and output to be performed by said input means and said output means;
• and
• media allocation means for performing the input-output media allocation by interacting with the user, said input-output media allocation including
• (i) at least one of a selection of a corresponding information to be supplied from said input means to said problem solving means from corresponding information generated for each of the input media and a combination of the corresponding information generated from said problem solving means for each of the input media, and
• (ii) to obtain an expression instructing the user from the output of said problem solving means, at least one of a selection of an information from said problem solving means and a combination of the selected information, and said media allocation means performing new media allocation when an instruction to change the input-output media allocation has been issued,
• said interaction control means including means for supplying an output request for requiring said problem solving means to output information required to constitute an output to the user in accordance with the input-output media allocation in which said output device have been set and requiring said problem solving means to output a required information, and
• said media allocation means including means for performing new media allocation by using said requested information when an instruction for performing a process for changing the input-output media allocation have been supplied.

A multimodal interaction apparatus comprising:

• problem solving means for subjecting an input to a calculation process to satisfy an object of the system and outputting a result of process;
• an input device, having plural input means, for interpreting an information input from at least one input means selected from said plural input means in accordance with an input-output media allocation and supplying an information corresponding to a result of interpretation to said problem solving means;
• an output device, having plural output means, for outputting a result of the process output from said problem solving means from at least one output means selected from the plural output means in accordance with the input-output media allocation;
• interaction control means for changing the input-output media allocation in accordance with the result of the process output from said problem solving means and controlling input and output to be performed by said input means and said output means;
• and
• media allocation means for performing the input-output media allocation by interacting with the user, said input-output media allocation including
• (i) at least one of a selection of a corresponding information to be supplied from said input means to said problem solving means from corresponding information generated for each of the input media and a combination of the corresponding information generated from said problem solving means for each of the input media, and
• (ii) to obtain an expression instructing the user from the output of said problem solving means, at least one of a selection of an information from said problem solving means and a combination of the selected information, and said media allocation means performing new media allocation when an instruction to change the input-output media allocation has been issued,
• said media allocation means including a media allocation slot for recording an input-output media allocation information which are selected at present, and when an instruction to change the input-output media allocation has been issued, a new media allocation is performed by using the input-output media allocation information recorded in said media allocation slot.

A multimodal interaction apparatus comprising:

• problem solving means for subjecting an input to a calculation process to satisfy an object of the system and outputting a result of process;
• an input device, having plural input means, for interpreting an information input from at least one input means selected from said plural input means in accordance with an input-output media allocation and supplying an information corresponding to a result of interpretation to said problem solving means;
• an output device, having plural output means, for outputting a result of the process output from said problem solving means from at least one output means selected from the plural output means in accordance with the input-output media allocation;
• interaction control means for changing the input-output media allocation in accordance with the result of the process output from said problem solving means and controlling input and output to be performed by said input means and said output means;
• and
• media allocation means for performing the input-output media allocation by interacting with the user, said input-output media allocation including
• (i) at least one of a selection of a corresponding information to be supplied from said input means to said problem solving means from corresponding information generated for each of the input media and a combination of the corresponding information generated from said problem solving means for each of the input media, and
• (ii) to obtain an expression instructing the user from the output of said problem solving means, at least one of a selection of an information from said problem solving means and a combination of the selected information, and said media allocation means performing new media allocation when an instruction to change the input-output media allocation has been issued,
• said media allocation means including a media allocation history slot for storing a series of input-output media allocation information which have been selected, when the input-output media allocation has been changed, selected input-output media allocation information is added to said media allocation history slot, and when an instruction to change the input-output media allocation has been issued, a new media allocation is performed by using the input-output media allocation information recorded in said media allocation slot.

A multimodal interaction apparatus in which input correspondence information is generated by interpreting information obtained from a user by interaction and the input correspondence information is, through communication means, supplied to a problem solving device for processing the input correspondence information to satisfy predetermined purposes and outputting a result of process so as to output the result of the process of the input correspondence information by interaction, the apparatus comprising:

• an input device for interpreting information input from at least one input means among plural input means which has been selected in accordance with an input-output media allocation and generating a correspondence information in accordance with a result of interpretation;
• an output device for outputting said correspondence information obtained from said problem solving means through said communication means from at least one output means among plural output means which has been selected in accordance with the input-output media allocation;
• communication means for supplying, to said problem solving means, said input correspondence information generated by said input means and receiving, from said problem solving means, a result of a problem solving operation generated and output by said problem solving means;
• and
• media allocation means for performing the input-output media allocation by interacting with the user, said input-output media allocation including
• (i) at least one of a selection of a corresponding information to be supplied from said input means to said problem solving means from corresponding information generated for each of the input media and a combination of the corresponding information generated from said problem solving means for each of the input media, and
• (ii) to obtain an information output from said problem solving means to said output device, at least one of a selection of an information from said problem solving means and a combination of the selected information,
• said media allocation means including a media allocation slot for recording an input-output media allocation information which are selected at present, and when an instruction to change the input-output media allocation has been issued, a new media allocation is performed by using the input-output media allocation information recorded in said media allocation slot.

A multimodal interaction apparatus in which input correspondence information is generated by interpreting information obtained from a user by interaction and the input correspondence information is, through communication means, supplied to a problem solving device for processing the input correspondence information to satisfy predetermined purposes and outputting a result of process so as to output the result of the process of the input correspondence information by interaction, the apparatus comprising:

• an input device for interpreting information input from at least one input means among plural input means which has been selected in accordance with an input-output media allocation and generating a correspondence information in accordance with a result of interpretation;
• an output device for outputting said correspondence information obtained from said problem solving means through said communication means from at least one output means among plural output means which has been selected in accordance with the input-output media allocation;
• communication means for supplying, to said problem solving means, said input correspondence information generated by said input means and receiving, from said problem solving means, a result of a problem solving operation generated and output by said problem solving means;
• and
• media allocation means for performing the input-output media allocation by interacting with the user, said input-output media allocation including
• (i) at least one of a selection of a corresponding information to be supplied from said input means to said problem solving means from corresponding information generated for each of the input media and a combination of the corresponding information generated from said problem solving means for each of the input media, and
• (ii) to obtain an information output from said problem solving means to said output device, at least one of a selection of an information from said problem solving means and a combination of the selected information,
• said media allocation means including a media allocation history slot for storing a series of input-output media allocation information which have been selected, when the input-output media allocation has been changed, selected input-output media allocation information is added to said media allocation history slot, and when an instruction to change the input-output media allocation has been issued, a new media allocation is performed by using the input-output media allocation information recorded in said media allocation slot.

A multimodal interaction apparatus in which input correspondence information is generated by interpreting information obtained from a user by interaction and the input correspondence information is supplied to a problem solving device for processing the input correspondence information to satisfy predetermined purposes and outputting a result of process so as to output the result of the process of the input correspondence information by interaction, the apparatus comprising:

• an input device for interpreting information input from at least one input means among plural input means which has been selected in accordance with an input-output media allocation and generating a correspondence information in accordance with a result of interpretation;
• an output device for outputting said correspondence information obtained from said problem solving means through said communication means from at least one output means among plural output means which has been selected in accordance with the input-output media allocation;
• interaction control means for changing the input-output media allocation in accordance with the result of the process performed by said problem solving means and controlling input and output to be performed by said input means and said output means;
• communication means for supplying, to said problem solving means, said input correspondence information generated by said input means and receiving, from said problem solving means, a result of a problem solving operation generated and output by said problem solving means;
• and
• media allocation means for performing the input-output media allocation by interacting with the user, said input-output media allocation including
• (i) to obtain the correspondence information to be supplied to the problem solving means from the input of said at least one input means, at least one of a selection of a corresponding information to be supplied from said input means to said problem solving means from corresponding information generated for each of the input media and a combination of the corresponding information generated from said problem solving means for each of the input media, and
• (ii) to obtain an expression instructing the user from the output of the problem solving means, at least one of a selection of an information from said problem solving means and a combination of the selected information,
• said media allocation means including a media allocation slot for recording an input-output media allocation information which are selected at present, and when an instruction to change the input-output media allocation has been issued, a new media allocation is performed by using the input-output media allocation information recorded in said media allocation slot.

A multimodal interaction apparatus in which input correspondence information is generated by interpreting information obtained from a user by interaction and the input correspondence information is supplied to a problem solving device for processing the input correspondence information to satisfy predetermined purposes and outputting a result of process so as to output the result of the process of the input correspondence information by interaction, the apparatus comprising:

• an input device for interpreting information input from at least one input means among plural input means which has been selected in accordance with an input-output media allocation and generating a correspondence information in accordance with a result of interpretation;
• an output device for outputting said correspondence information obtained from said problem solving means through said communication means from at least one output means among plural output means which has been selected in accordance with the input-output media allocation;
• interaction control means for changing the input-output media allocation in accordance with the result of the process performed by said problem solving means and controlling input and output to be performed by said input means and said output means;
• communication means for supplying, to said problem solving means, said input correspondence information generated by said input means and receiving, from said problem solving means, a result of a problem solving operation generated and output by said problem solving means;
• and
• media allocation means for performing the input-output media allocation by interacting with the user, said input-output media allocation including
• (i) to obtain the correspondence information to be supplied to the problem solving means from the input of said at least one input means, at least one of a selection of a corresponding information to be supplied from said input means to said problem solving means from corresponding information generated for each of the input media and a combination of the corresponding information generated from said problem solving means for each of the input media, and
• (ii) to obtain an expression instructing the user from the output of the problem solving means, at least one of a selection of an information from said problem solving means and a combination of the selected information,
• said media allocation means including a media allocation history slot for storing a series of input-output media allocation information which have been selected, when the input-output media allocation has been changed, selected input-output media allocation information is added to said media allocation history slot, and when an instruction to change the input-output media allocation has been issued, a new media allocation is performed by using the input-output media allocation information recorded in said media allocation slot.

A multimodal interaction apparatus in which input correspondence information is generated by interpreting information obtained from a user by interaction and the input correspondence information is supplied to a problem solving device for processing the input correspondence information to satisfy predetermined purposes and outputting a result of process so as to output the result of the process of the input correspondence information by interaction, the apparatus comprising:

• an input device for interpreting information input from at least one input means among plural input means which has been selected in accordance with an input-output media allocation and generating a correspondence information in accordance with a result of interpretation;
• an output device for outputting said correspondence information obtained from said problem solving means through said communication means from at least one output means among plural output means which has been selected in accordance with the input-output media allocation;
• interaction control means for changing the input-output media allocation in accordance with the result of the process performed by said problem solving means and controlling input and output to be performed by said input means and said output means;
• communication means for supplying, to said problem solving means, said input correspondence information generated by said input means and receiving, from said problem solving means, a result of a problem solving operation generated and output by said problem solving means;
• and
• media allocation means for performing the input-output media allocation by interacting with the user, said input-output media allocation including
• (i) to obtain the correspondence information to be supplied to the problem solving means from the input of said at least one input means, at least one of a selection of a corresponding information to be supplied from said input means to said problem solving means from corresponding information generated for each of the input media and a combination of the corresponding information generated from said problem solving means for each of the input media, and
• (ii) to obtain an expression instructing the user from the output of the problem solving means, at least one of a selection of an information from said problem solving means and a combination of the selected information,
• said interaction control means including at least one of input request means for causing said input means to urge the user to input a required information in response to input request information generated by said problem solving means and supplied to said interaction control means through said communication means when said problem solving means requires an information needed to solve a problem of the system, and output request means for generating an output request for requesting said problem solving means to output information needed to constitute an output to the user in accordance with the input-output media allocation in which said output means has been set and supplying the output request to said problem solving means through said communication means to request said problem solving means to supply said output requirement through said communication means, and
• said media allocation means including means for performing new media allocation by using at least one of said input request and said output request when an instruction to change the input-output media allocation has been issued.