Teaching computer-use artificial intelligence (AI) agents from video demonstration

A system, comprising:

• a processor configured to:
• receive a task recording including (i) video information that captures a multi-step interaction with a computer user interface and (ii) device input information associated with the multi-step interaction;
• extract a sequence of anchor actions from the video information using the device input information;
• for each anchor action in at least a subset of the sequence of anchor actions, obtain a preceding frame and a following frame from the video information adjacent to the anchor action;
• generate, by a prompt generator, a prompt, wherein:
• the prompt includes a task prompt in natural language that describes the multi-step interaction based on the sequence of anchor actions and corresponding obtained preceding frames and following frames;
• and
• the task prompt includes instructions executable by a computer-use agent to replicate the multi-step interaction;
• execute, by a machine learning model, the prompt to obtain an execution trace including a sequence of predicted actions;
• compare the sequence of predicted actions with the sequence of anchor actions to identify whether there is at least one discrepancy between an action of the sequence of predicted actions and an action of the sequence of anchor actions;
• responsive to identifying that there is at least one discrepancy between the action of the sequence of predicted actions and the action of the sequence of anchor actions, obtain an enhanced prompt by iteratively enhancing the prompt until there is no discrepancy between the sequence of predicted actions and the sequence of anchor actions;
• and
• output at least one of:
• the generated prompt and the enhanced prompt;
• and
• a memory coupled to the processor and configured to provide the processor with instructions.

A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:

• receiving a task recording including (i) video information that captures a multi-step interaction with a computer user interface and (ii) device input information associated with the multi-step interaction;
• extracting a sequence of anchor actions from the video information using the device input information;
• for each anchor action in at least a subset of the sequence of anchor actions, obtaining a preceding frame and a following frame from the video information adjacent to the anchor action;
• generating, by a prompt generator, a prompt, wherein:
• the prompt includes a task prompt in natural language that describes the multi-step interaction based on the sequence of anchor actions and corresponding obtained preceding frames and following frames;
• and
• the task prompt includes instructions executable by a computer-use agent to replicate the multi-step interaction;
• executing, by a machine learning model, the prompt to obtain an execution trace including a sequence of predicted actions;
• comparing the sequence of predicted actions with the sequence of anchor actions to identify whether there is at least one discrepancy between an action of the sequence of predicted actions and an action of the sequence of anchor actions;
• responsive to identifying that there is at least one discrepancy between the action of the sequence of predicted actions and the action of the sequence of anchor actions, obtaining an enhanced prompt by iteratively enhancing the prompt until there is no discrepancy between the sequence of predicted actions and the sequence of anchor actions;
• and
• outputting at least one of:
• the generated prompt and the enhanced prompt.