US 12,769,932 B2 · Vision-based generation of navigation workflow for automatically filling application forms using large language models

Abstract

Robotic Process Automation (RPA) systems face challenges in handling complex processes and diverse screen layouts that require advanced human-like decision-making capabilities. These systems typically rely on pixel-level encoding through drag-and-drop or automation frameworks such as Selenium to create navigation workflows, rather than visual understanding of screen elements. Present disclosure provides systems and methods that implement large language models (LLMs) coupled with deep learning based image understanding which adapt to new scenarios, including changes in user interface and variations in input data, without the need for human intervention. System of the present disclosure uses computer vision and natural language processing to perceive visible elements on graphical user interface (GUI) and convert them into a textual representation. This information is then utilized by LLMs to generate one or more navigation workflows that include a sequence of actions that are executed by a scripting engine/code to complete an assigned task from a task-request.

Claims18

Dependent claims

A processor implemented method comprising:

receiving, via one or more hardware processors, an input metadata pertaining to an application form, wherein the input metadata comprises a layout mapping of the application form, and a location of the application form;
extracting, by using at least one of one or more vision-based techniques and one or more Large language Models (LLMs) via the one or more hardware processors, one or more field names, and one or more associated field types from the application form;
merging, by using the one or more LLMs via the one or more hardware processors, the one or more field names, and the one or more associated field types with the layout mapping to obtain a mapping list, wherein the mapping list comprises one or more coordinates associated with the one or more field names, and the one or more associated field types;
and
generating, by using the one or more LLMs via the one or more hardware processors, a navigation workflow using the mapping list and a task-request obtained from a user, wherein the navigation workflow comprises a sequence of actions for execution of one or more micro-level steps comprised therein and the task-request for handling of the one or more associated field types using one or more screenshots associated with the application form via the one or more vision-based techniques.

2 more elements

A system, comprising:

a memory storing instructions;
one or more communication interfaces;
and
one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
receive an input metadata pertaining to an application form, wherein the input metadata comprises a layout mapping of the application form, and a location of the application form;
extract, by using at least one of one or more vision-based techniques and one or more Large language Models (LLMs), one or more field names, and one or more associated field types from the application form;
merge, by using the one or more LLMs, the one or more field names, and the one or more associated field types with the layout mapping to obtain a mapping list, wherein the mapping list comprises one or more coordinates associated with the one or more field names, and the one or more associated field types;
and
generate, by using the one or more LLMs, a navigation workflow using the mapping list and a task-request obtained from a user, wherein the navigation workflow comprises a sequence of actions for execution of one or more micro-level steps comprised therein and the task-request for handling of the one or more associated field types using one or more screenshots associated with the application form via the one or more vision-based techniques.

6 more elements

One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

receiving an input metadata pertaining to an application form, wherein the input metadata comprises a layout mapping of the application form, and a location of the application form;
extracting, by using at least one of one or more vision-based techniques and one or more Large language Models (LLMs), one or more field names, and one or more associated field types from the application form;
merging, by using the one or more LLMs, the one or more field names, and the one or more associated field types with the layout mapping to obtain a mapping list, wherein the mapping list comprises one or more coordinates associated with the one or more field names, and the one or more associated field types;
and
generating, by using the one or more LLMs, a navigation workflow using the mapping list and a task-request obtained from a user, wherein the navigation workflow comprises a sequence of actions for execution of one or more micro-level steps comprised therein and the task-request for handling of the one or more associated field types using one or more screenshots associated with the application form via the one or more vision-based techniques.

2 more elements

Dependencies are inferred from the claim text.

Vision-based generation of navigation workflow for automatically filling application forms using large language models

Life of this patent

Vision-based generation of navigation workflow for automatically filling application forms using large language models

Life of this patent

Abstract

Claims18