# US 2023/0419161 A1 · Target-based schema identification and semantic mapping for robotic process automation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0419161 A1
- Document ID: US20230419161A1
- Status: Published application
- Kind: A1
- Application number: 17/747,448
- Filed: 2022-05-18
- Published: 2023-12-28
- Earliest claimed: 2022-05-17
- Assignees: UiPath, Inc.
- Applicants: UiPath, Inc.
- Inventors: Daniel DINES
- Web page: https://patentgrep.com/patent/US20230419161A1

## CPC

- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 20/00: Machine learning
- B25J 9/1653: {parameters identification, estimation, stiffness, accuracy, error analysis}
- G05B 19/4155: characterised by program execution, i.e. part program or machine function execution, e.g. selection of a program
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 40/174: Form filling; Merging
- G06F 40/18: of spreadsheets (form-filling G06F40/174)
- G06F 40/216: using statistical methods
- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G05B 2219/50391: Robot
- G06N 3/045: Combinations of networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Target-based schema identification and semantic mapping for robotic process automation (RPA) are disclosed. When looking at a source, such as a document, a web form, a user interface of a software application, a data file, etc., it is often difficult for software to determine which fields are labels and which are values associated with those labels. Since values have not yet been entered for various labels (e.g., first name, company, customer number, etc.), these labels are easier to detect than when the target also includes various values associated with the labels. A selection of an empty target may be received and target-based schema identification may be performed on the empty target, determining labels and a type of the target. Semantic matching may then be performed between a source and the target. These features may be performed at design time or runtime.

## Claims

### Claim 1

A computing system, comprising:
memory storing computer program instructions for an automation configured to be executed by a robotic process automation (RPA) robot at runtime; and
at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause the at least one processor to:
execute the automation, by the RPA robot;
receive a selection of an empty target, by the RPA robot; and
perform target-based schema identification on the empty target, wherein the target-based schema identification comprises determining labels and a type of the target, by the RPA robot.

### Claim 2 (depends on 1)

The computing system of claim 1, wherein the target comprises a web page, a graphical user interface (GUI) of an application, an image, a file explorer, a spreadsheet, a relational database, a flat file source, or any combination thereof.

### Claim 3 (depends on 1)

The computing system of claim 1, wherein the determination of the labels and the type of the target comprises calling a computer vision (CV) model, OCR model, a document processing model, or any combination thereof, etc.

### Claim 4 (depends on 1)

The computing system of claim 1, wherein the computer program instructions are further configured to cause the at least one processor to:
receive a selection of a source, by the RPA robot; and
perform semantic matching between the labels of the target and key-value pairs of the source using a semantic matching model based on the determined labels and type of the target, by the RPA robot.

### Claim 5 (depends on 4)

The computing system of claim 4, wherein the semantic matching model is trained by:
providing words and phrases with semantic associations between the words and phrases such that similar words and phrases for a given word or phrase based on a metric can be identified, and
providing contextual labels pertaining to a screen in which the words and phrases appear.

### Claim 6 (depends on 4)

The computing system of claim 4, wherein the target-based schema identification is performed using a natural language processing (NLP) model.

### Claim 7 (depends on 4)

The computing system of claim 4, wherein the computer program instructions are further configured to cause the at least one processor to:
receive indications of graphical elements associated with semantically matched labels in the target and respective confidence scores from the semantic matching model, by the RPA robot; and
display the semantically matched labels and the respective target confidence scores in a matching interface, by the RPA robot.

### Claim 8 (depends on 4)

The computing system of claim 4, wherein the computer program instructions are further configured to cause the at least one processor to:
receive one or more corrections in the source, the target, or both, where semantic matching was incorrect, by the RPA robot; and
directly or indirectly store the one or more corrections for retraining the semantic matching model, by the RPA robot.

### Claim 9

A non-transitory computer-readable medium storing a computer program executed by a robotic process automation (RPA) robot at runtime, the computer program configured to cause the at least one processor to:
perform target-based schema identification on an empty target, wherein the target-based schema identification comprises determining labels and a type of the target; and
perform semantic matching between the labels of the target and key-value pairs of the source using a semantic matching model based on the determined labels and type of the target.

### Claim 10 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the target comprises a web page, a graphical user interface (GUI) of an application, an image, a file explorer, a spreadsheet, a relational database, a flat file source, or any combination thereof.

### Claim 11 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the determination of the labels and the type of the target comprises calling a computer vision (CV) model, OCR model, a document processing model, or any combination thereof, etc.

### Claim 12 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the semantic matching model is trained by:
providing words and phrases with semantic associations between the words and phrases such that similar words and phrases for a given word or phrase based on a metric can be identified, and
providing contextual labels pertaining to a screen in which the words and phrases appear.

### Claim 13 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the target-based schema identification is performed using a natural language processing (NLP) model.

### Claim 14 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the computer program is further configured to cause the at least one processor to:
receive indications of graphical elements associated with semantically matched labels in the target and respective confidence scores from the semantic matching model; and
display the semantically matched labels and the respective target confidence scores in a matching interface.

### Claim 15 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the computer program is further configured to cause the at least one processor to:
receive one or more corrections in the source, the target, or both, where semantic matching was incorrect; and
directly or indirectly store the one or more corrections for retraining the semantic matching model.

### Claim 16

A computer-implemented method, comprising:
performing target-based schema identification on an empty target, by a robotic process automation (RPA) robot, wherein the target-based schema identification comprises determining labels and a type of the target;
performing semantic matching between the labels of the target and key-value pairs of a source using a semantic matching model based on the determined labels and type of the target, by the RPA robot;
receiving indications of graphical elements associated with semantically matched labels in the target and respective confidence scores from the semantic matching model, by the RPA robot; and
displaying the semantically matched labels and the respective target confidence scores in a matching interface, by the RPA robot.

### Claim 17 (depends on 16)

The computer-implemented method of claim 16, wherein the determination of the labels and the type of the target comprises calling a computer vision (CV) model, OCR model, a document processing model, or any combination thereof, etc.

### Claim 18 (depends on 16)

The computer-implemented method of claim 16, wherein the semantic matching model is trained by:
providing words and phrases with semantic associations between the words and phrases such that similar words and phrases for a given word or phrase based on a metric can be identified, and
providing contextual labels pertaining to a screen in which the words and phrases appear.

### Claim 19 (depends on 16)

The computer-implemented method of claim 16, wherein the target-based schema identification is performed using a natural language processing (NLP) model.

### Claim 20 (depends on 16)

The computer-implemented method of claim 16, further comprising:
receiving one or more corrections in the source, the target, or both, where semantic matching was incorrect, by the RPA robot; and
directly or indirectly storing the one or more corrections for retraining the semantic matching model, by the RPA robot.
