Digital assistant with copilot support to enhance application usage

A system comprising:

• one or more processors;
• and
• one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the system to perform operations comprising:
• receiving a message payload comprising:
• (i) an utterance or indication of an interface action, and (ii) application context, associated with an interaction between a user and an application;
• invoking, using a thread, a flow based on the message payload;
• generating a context variable data structure associated with the thread, wherein generating the context variable data structure comprises storing one or more variables from the application context as at least a portion of data stored within the context variable data structure;
• responsive to invoking the flow, determining, using a machine learning model, an intent of the user based on (i) the utterance or the indication of the interface action, (ii) the data in the context variable data structure, or (iii) both;
• accessing, based on the intent, a prompt and an object schema;
• inserting:
• the utterance or the indication of the interface action, at least a portion of the data in the context variable data structure, and the object schema into the prompt;
• generating, using a generative artificial intelligence (GenAI) model, a list comprising one or more executable actions based on the prompt, wherein the generating the list comprises:
• deriving one or more parameters from (i) the utterance or the indication of the interface action, (ii) the data in the context variable data structure, or (iii) both, and
• filling one or more input argument slots of the one or more executable actions with a version of the one or more parameters that conforms to the object schema;
• executing the one or more executable actions based on the version of the one or more parameters to obtain an output;
• and
• sending the output or a communication derived from the output to the user.

One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:

• receiving a message payload comprising:
• (i) an utterance or indication of an interface action, and (ii) application context, associated with an interaction between a user and an application;
• invoking, using a thread, a flow based on the message payload;
• generating a context variable data structure associated with the thread, wherein generating the context variable data structure comprises storing one or more variables from the application context as at least a portion of data stored within the context variable data structure;
• responsive to invoking the flow, determining, using a machine learning model, an intent of the user based on (i) the utterance or the indication of the interface action, (ii) the data in the context variable data structure, or (iii) both;
• accessing, based on the intent, a prompt and an object schema;
• inserting:
• the utterance or the indication of the interface action, at least a portion of the data in the context variable data structure, and the object schema into the prompt;
• generating, using a generative artificial intelligence (GenAI) model, a list comprising one or more executable actions based on the prompt, wherein the generating the list comprises:
• deriving one or more parameters from (i) the utterance or the indication of the interface action, (ii) the data in the context variable data structure, or (iii) both, and
• filling one or more input argument slots of the one or more executable actions with a version of the one or more parameters that conforms to the object schema;
• executing the one or more executable actions based on the version of the one or more parameters to obtain an output;
• and
• sending the output or a communication derived from the output to the user.