Systems, methods, and computer-readable media for application aware sequence generation and execution

One or more non-transitory computer-readable media comprising instructions that, when executed by a computing system, cause the computing system to:

• receive a first user input;
• analyze, based at least in part on a user profile, the first user input to detect a command to be performed on a user device;
• determine an application environment of the user device based at least in part on traversing one or more accessibility trees corresponding to an operating system of the user device and one or more applications of the user device;
• analyze, based at least in part on the application environment of the user device, the command using a natural language understanding (NLU) model to identify one or more parameters of the command;
• retrieve an application profile based at least in part on the application environment of the user device and the one or more parameters of the command;
• generate, based at least in part on the application profile and one or more of the one or more parameters of the command, a first sequence corresponding to the command, the sequence comprising one or more steps to be executed in one or more applications corresponding to the application profile;
• execute a first step of the one or more steps of the sequence in an application of the one or more applications corresponding to the application profile;
• and
• determine, based at least in part on the application environment, whether a post-execution condition is satisfied.

A system comprising:

• one or more processors;
• and
• at least one memory operatively coupled to the processor and storing instructions that, when executed by at least one processor of the one or more processors, cause the system to:
• receive a first user input;
• analyze, based at least in part on a user profile, the first user input to detect a command to be performed on a user device;
• determine an application environment of the user device based at least in part on traversing one or more accessibility trees corresponding to an operating system of the user device and one or more applications of the user device;
• analyze, based at least in part on the application environment of the user device, the command using a natural language understanding (NLU) model to identify one or more parameters of the command;
• retrieve an application profile based at least in part on the application environment of the user device and the one or more parameters of the command;
• generate, based at least in part on the application profile and one or more of the one or more parameters of the command, a first sequence corresponding to the command, the sequence comprising one or more steps to be executed in one or more applications corresponding to the application profile;
• execute a first step of the one or more steps of the sequence in an application of the one or more applications corresponding to the application profile;
• and
• determine, based at least in part on the application environment, whether a post-execution condition is satisfied.