Adding Voice or Chat User interface to graphical user interface (gui)-based virtualized applications and desktops using large language and large action models
AssigneeCitrix Systems, Inc.
InventorsHubert Divoux, Mukund Ingale
Life of this patent
Estimating the term…
Abstract
Methods and systems for enhanced remote desktop interfaces are described. A computing system may train, using historical or live information, a LAM to execute, within a remote desktop application, textual actions with their parameters if any. A user declarative request (voice or chat) may be interpreted by a LLM to match a specific action (and potentially ask for the corresponding parameters in a conversational way). Subsequently, from the textual action and its parameters, the LAM may execute the action within a remote desktop application and report the result to the user via voice or chat.
Claims20
A method comprising:
- training, using historical remote desktop interaction information indicating user inputs and corresponding actions executed within historical remote desktop application sessions, a large action model (LAM), wherein training the LAM configures the LAM to execute, for a given textual input, one or more actions to perform within a given remote desktop application to complete a task requested by the given textual input;
- deploying, to a remote desktop host server, a LAM agent, configured to access the LAM to identify the one or more actions;
- receiving, during a remote desktop session, a textual input indicating a first task to perform;
- identifying, based on the first task, a remote desktop application configured to perform the task and a list of actions that the remote desktop application is configured to perform;
- identifying, using a large language model (LLM), at least one action of the list of actions to execute to perform the task;
- executing, using the LAM, the at least one action to produce an action result;
- and
- displaying the action result, wherein the action result comprises an indication that the task has been executed.
A computing system comprising:
- one or more processors;
- memory storing computer executable instructions that, when executed by the one or more processors, cause the computing system to:
- train, using historical remote desktop interaction information indicating user inputs and corresponding actions executed within historical remote desktop application sessions, a large action model (LAM), wherein training the LAM configures the LAM to execute, for a given textual input, one or more actions to perform within a given remote desktop application to complete a task requested by the given textual input;
- deploy, to a remote desktop host server, a LAM agent, configured to access the LAM to identify the one or more actions;
- receive, during a remote desktop session, a textual input indicating a first task to perform;
- identify, based on the first task, a remote desktop application configured to perform the task and a list of actions that the remote desktop application is configured to perform;
- identify, using a large language model (LLM), at least one action of the list of actions to execute to perform the task;
- execute, using the LAM, the at least one action to produce an action result;
- and
- display the action result, wherein the action result comprises an indication that the task has been executed.
One or more non-transitory computer-readable media storing instructions that, when executed by a computing system comprising at least one processor, a communication interface, and memory, cause the computing system to:
- train, using historical remote desktop interaction information indicating user inputs and corresponding actions executed within historical remote desktop application sessions, a large action model (LAM), wherein training the LAM configures the LAM to execute, for a given textual input, one or more actions to perform within a given remote desktop application to complete a task requested by the given textual input;
- deploy, to a remote desktop host server, a LAM agent, configured to access the LAM to identify the one or more actions;
- receive, during a remote desktop session, a textual input indicating a first task to perform;
- identify, based on the first task, a remote desktop application configured to perform the task and a list of actions that the remote desktop application is configured to perform;
- identify, using a large language model (LLM), at least one action of the list of actions to execute to perform the task;
- execute, using the LAM, the at least one action to produce an action result;
- and
- display the action result, wherein the action result comprises an indication that the task has been executed.
Dependencies are inferred from the claim text.