Electronic device for processing multi-modal input, method for processing multi-modal input and server for processing multi-modal input

An electronic device comprising:

• a housing;
• a touchscreen display exposed through a first portion of the housing;
• a microphone disposed at a second portion of the housing;
• a speaker disposed at a third portion of the housing;
• a memory disposed inside the housing;
• and
• at least one processor disposed inside the housing, and electrically connected to the display, the microphone, the speaker, and the memory,
• wherein the memory is configured to store a first plurality of application programs and a second plurality of application programs, wherein each of the application programs includes a graphic user interface (GUI), and
• wherein the memory is further configured to store instructions that, when executed, cause the at least one processor to:
• control the microphone to receive an utterance from a user;
• generate or receive text data based on the utterance;
• generate or receive a sequence of states in which the electronic device performs a task based on the text data, wherein the sequence of states includes at least two states of at least one of the application programs;
• select a first application program from the first plurality of application programs, based at least partly on the sequence of states;
• launch the selected first application program;
• control to display the GUI of the selected first application program on the display;
• cause the selected first application program to perform a first operation that utilizes at least one first parameter, based at least partly on the sequence of states, while displaying at least part of a sequence of performing the first operation on the display;
• control to display a result of the first operation on the display;
• after displaying the result, control to display one or more identifiers on the display, based at least partly on the sequence of states, wherein each of the identifiers represents a respective one of the second plurality of application programs;
• receive a selection of one of the identifiers from the user via the display or the microphone;
• select a second application program from the second plurality of application programs, based on the received selection;
• and
• cause the selected second application program to perform a second operation that utilizes at least one second parameter, using at least part of the result of the first operation.

An electronic device comprising:

• a housing;
• a touchscreen display exposed through a first portion of the housing;
• a microphone disposed at a second portion of the housing;
• a speaker disposed at a third portion of the housing;
• a memory disposed inside the housing;
• and
• at least one processor disposed inside the housing, and electrically connected to the display, the microphone, the speaker, and the memory,
• wherein the memory is configured to store a plurality of application programs, each of the application programs including a graphic user interface (GUI), and
• wherein the memory is further configured to store instructions that, when executed, cause the at least one processor to:
• control the microphone to receive a verbal request from a user;
• generate or receive text data based on the request;
• generate or receive a sequence of states in which the electronic device performs a task based on the text data, wherein the sequence of states includes at least two states of at least one of the application programs;
• select one of the applications programs, based at least partly on the sequence of states;
• determine a portion of an operation that can be performed automatically on the selected application program, using the text data;
• launch the selected application program;
• control to display the GUI of the selected application program on the display;
• control to display at least part of a sequence of automatically performing the portion of the operation, based at least partly on the sequence of states, without completing the operation;
• receive a user input via the display or microphone;
• and
• cause the application program to complete the operation, using the received user input.

A voice input processing method, the method comprising:

• receiving an utterance from a user via a microphone;
• generating or receiving text data based on the utterance;
• generate or receive a sequence of states in which the electronic device performs a task based on the text data, wherein the sequence of states includes at least two states of at least one of the application programs;
• selecting the application program based at least partly on the sequence of states;
• launching the selected application program;
• displaying a graphic user interface (GUI) of the selected application program on a display;
• displaying at least part of a sequence of automatically providing an input into the GUI of the selected application program on the display, based at least partly on the sequence of states, wherein the input is provided from outside the selected application program;
• causing the selected application program to perform an operation that utilizes at least one parameter, using the input as the at least one parameter, depending on the sequence of states;
• and
• displaying a result of the operation on the display.