Crossmodal Interface Automation And Orchestration Without Api Integration

A computer-implemented method for automating tasks in a target software application without invoking a published application programming interface (API) of the target application at runtime, the method comprising:

• (a) capturing, from a user device executing the target application, synchronized streams including (i) images of a display on which the target application renders a user interface, (ii) input events received at the user device, (iii) audio of a task request or instruction, and (iv) text of at least one digital procedure or policy;
• (b) generating, by an aligner executed by one or more processors, time-aligned crossmodal tokens that associate visual features and layout, the input events, segments of an automatic speech recognition transcript of the audio, and text snippets from the at least one digital procedure or policy;
• (c) training a Large Action Model (LAM) to predict a user-interface action for the target application from the crossmodal tokens;
• (d) training a Large Orchestration Model (LOM) to select and order multiple actions output by the LAM to satisfy a goal expressed in the audio or the text;
• (e) executing the ordered actions by generating operating-system native input signals directed to the user interface of the target application without invoking a published API of the target application;
• and
• (f) verifying completion by analyzing subsequent images of the display and, based on a result, updating at least one of the LAM or the LOM.