# US 2026/0064249 A1 · Crossmodal Interface Automation And Orchestration Without Api Integration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0064249 A1
- Document ID: US20260064249A1
- Status: Published application
- Kind: A1
- Application number: 19/313,686
- Filed: 2025-08-28
- Published: 2026-03-05
- Earliest claimed: 2024-08-28
- Inventors: Ramin Bolouri
- Web page: https://patentgrep.com/patent/US20260064249A1

## CPC

- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A system automates tasks in software applications without invoking a published API of the target application at runtime. Synchronized visual display frames, input events, audio of a task request, and text from digital procedures are captured and aligned to form crossmodal tokens. A Large Action Model maps tokens to user-interface actions, and a Large Orchestration Model composes and orders the actions to satisfy a goal and policy constraints. An executor issues operating-system native input signals to the target application and verifies outcomes from subsequent display frames using optical character recognition and layout cues. A feedback loop updates the models. The approach provides no-integration automation across legacy and modern applications with semantic reanchoring for UI changes and privacy protections.

## Claims

### Claim 1

A computer-implemented method for automating tasks in a target software application without invoking a published application programming interface (API) of the target application at runtime, the method comprising:
(a) capturing, from a user device executing the target application, synchronized streams including (i) images of a display on which the target application renders a user interface, (ii) input events received at the user device, (iii) audio of a task request or instruction, and (iv) text of at least one digital procedure or policy;
(b) generating, by an aligner executed by one or more processors, time-aligned crossmodal tokens that associate visual features and layout, the input events, segments of an automatic speech recognition transcript of the audio, and text snippets from the at least one digital procedure or policy;
(c) training a Large Action Model (LAM) to predict a user-interface action for the target application from the crossmodal tokens;
(d) training a Large Orchestration Model (LOM) to select and order multiple actions output by the LAM to satisfy a goal expressed in the audio or the text;
(e) executing the ordered actions by generating operating-system native input signals directed to the user interface of the target application without invoking a published API of the target application; and
(f) verifying completion by analyzing subsequent images of the display and, based on a result, updating at least one of the LAM or the LOM.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating time-aligned crossmodal tokens includes forced temporal alignment of speech transcript segments to user-interface state transitions detected in the images of the display.

### Claim 3 (depends on 1)

The method of claim 1, wherein the LAM uses semantic user-interface tokenization that labels user-interface elements by function class obtained from combined computer-vision, optical character recognition, and layout graph features.

### Claim 4 (depends on 1)

The method of claim 1, wherein the LOM maintains a latent process graph whose nodes reference LAM task primitives and whose edges encode ordering, branching, retries, and recovery policies responsive to visual confidence.

### Claim 5 (depends on 1)

The method of claim 1, further comprising reanchoring a target user-interface element by matching a function-equivalent element when an expected element is absent or moved.

### Claim 6 (depends on 1)

The method of claim 1, wherein the at least one digital procedure includes a compliance policy, and the LOM constrains an action sequence to satisfy the compliance policy.

### Claim 7 (depends on 1)

The method of claim 1, wherein the audio comprises a customer call recording, and the goal is extracted from the call.

### Claim 8 (depends on 1)

The method of claim 1, further comprising masking personally identifiable information in captured images before storage.

### Claim 9 (depends on 1)

The method of claim 1, wherein verifying completion includes optical character recognition of a confirmation code and comparison to a regular expression specified in the digital procedure.

### Claim 10 (depends on 1)

The method of claim 1, wherein the LAM is trained using behavior cloning from observed user actions together with textual step labels derived from the digital procedure.

### Claim 11 (depends on 1)

The method of claim 1, wherein executing the ordered actions includes issuing keyboard, mouse, or touch events via operating-system calls while refraining from invoking a published API of the target application.

### Claim 12 (depends on 1)

The method of claim 1, wherein the target application is a legacy application lacking accessibility metadata and the method completes the task using only the images of the display and the operating-system native input signals.

### Claim 13 (depends on 1)

The method of claim 1, wherein the LOM selects among alternative action sequences using a confidence-weighted utility that incorporates a policy-violation cost.

### Claim 14 (depends on 1)

The method of claim 1, wherein the crossmodal tokens further include a speaker diarization tag distinguishing customer from agent speech.

### Claim 15 (depends on 1)

The method of claim 1, wherein the LAM and the LOM are updated by a feedback loop that incorporates manual overrides from an administrator dashboard.

### Claim 16 (depends on 1)

The method of claim 1, wherein the capturing and executing occur within a virtual desktop session or a remote application window.

### Claim 17 (depends on 1)

The method of claim 1, further comprising batch execution of multiple goals by queuing LOM plans and rate-limiting operating-system native input signals.

### Claim 18 (depends on 1)

The method of claim 1, wherein the text of the digital procedure includes an enterprise knowledge base and change logs, and the LOM adapts the latent process graph when the knowledge base changes.

### Claim 19 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18)

A system comprising one or more processors and memory storing instructions that, when executed by the one or more processors, cause the system to perform the method of any one of claims 1-18.

### Claim 20 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18)

A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause performance of the method of any one of claims 1-18.
