# US 2021/0333983 A1 · Learning user interface controls via incremental data synthesis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0333983 A1
- Document ID: US20210333983A1
- Status: Published application
- Kind: A1
- Application number: 16/859,488
- Filed: 2020-04-27
- Published: 2021-10-28
- Earliest claimed: 2020-04-27
- Assignees: Automation Anywhere, Inc.
- Applicants: Automation Anywhere, Inc.
- Inventors: Sudhir Kumar Singh; Jesse Truscott; Virinchipuram J Anand
- Web page: https://patentgrep.com/patent/US20210333983A1

## CPC

- G06F 8/38: for implementing user interfaces
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 9/451: Execution arrangements for user interfaces
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/225: {based on a marking or identifier characterising the area}
- G06V 10/235: {based on user input or interaction}
- G06V 20/62: Text, e.g. of license plates, overlay texts or captions on TV images
- G06V 30/19147: {Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting}
- G06V 30/19173: {Classification techniques}
- G06V 30/40: Document-oriented image-based pattern recognition
- G06T 2210/12: Bounding box
- G06V 30/43: {Editing text-bitmaps, e.g. alignment, spacing; Semantic analysis of bitmaps of text without OCR}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A User Interface (UI) interface object detection system employs an initial dataset comprising a set of images, that may include synthesized images, to train a Machine Learning (ML) engine to generate an initial trained model. A data point generator is employed to generate an updated synthesized image set which is used to further train the ML engine. The data point generator may employ images generated by an application program as a reference by which to generate the updated synthesized image set. The images generated by the application program may be tagged in advance. Alternatively, or in addition, the images generated by the application program may be captured dynamically by a user using the application program.

## Claims

### Claim 1

A computer-implemented method for detecting one or more user interface control objects contained in a screen image of a user interface generated by an application program, the method comprising:
training a machine learning model with a portion of an initial dataset comprising a plurality of images to cause the machine learning model to detect user interface control objects in each of a plurality of object classes, wherein each image represents a screen of information containing one or more user interface control objects, wherein each user interface control object is part of a corresponding user interface object class;
modifying the initial dataset by,
processing screen images generated by a first application program to detect user interface control objects in the screen images generated by the first application program,
creating a set of new application screen images as a function of the user interface control objects detected in the screen images generated by the first application program, and
generating an updated dataset by modifying the initial dataset with the set of new application screen images; and
further training the machine learning model with the updated dataset.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1 wherein the operation of processing the screen images generated by the first application program to detect user interface control objects in the screen images generated by the first application program comprises:
retrieving from the initial dataset a set of tagged screen images where each screen image of the set of tagged screen images comprises, for one or more user interface control objects in the screen image, a set of tagged bounding boxes, each tagged bounding box surrounding a corresponding user interface control object, and a tagged object class identifier that classifies the corresponding user interface control object into one of a plurality of object classes.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2 wherein the operation of processing the screen images generated by the first application program to detect user interface control objects in the screen images generated by the first application program further comprises:
processing the set of tagged screen images with the machine learning model, as trained by the initial dataset, to generate for each tagged screen image of the set of tagged screen images, a set of detected bounding boxes each of the bounding boxes having associated therewith an object class label that identifies an object class associated with a user interface control object within the associated bounding box and a confidence score indicative of a level of confidence of identification of the user interface control object within the associated bounding box;
comparing for each tagged screen image, the set of detected bounding boxes to the set of tagged bounding boxes to generate a set of failure cases comprising missed or incorrectly identified bounding boxes;
collecting invariance parameters associated with each user interface control object contained in the set of failure cases; and
generating a set of updated invariance parameters by updating invariance parameters employed by the machine learning engine as a function of the invariance parameters associated with each user interface control object contained in the set of failure cases.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3 wherein the operation of creating a set of new application screen images as a function of the user interface control objects detected in the screen images generated by the first application program comprises:
generating for each new application screen image of the set of new application screen images, one or more user interface control objects as a function of the updated invariance parameters.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1 wherein the operation of processing the screen images generated by the first application program to detect user interface control objects in the screen images generated by the first application program comprises:
capturing application screen images generated by the first application program as the first application program is used by a user of the first application program and capturing user inputs entered by the user;
processing each captured application screen image with the machine learning model, as trained by the initial dataset, to detect user interface control objects in each captured screen image;
validating for each captured screen image, detected user interface control objects in each captured screen image, as a function of user inputs entered into the application screen of the first application program that corresponds to each captured screen image;
employing validated user interface control objects for a captured screen image to extract object level characteristics for each user interface control object in the application screen image;
computing for each user interface control object in the application screen image one or more invariance parameters;
computing for each application screen image one or more invariance parameters; and
synthesizing one or more new application screen images, each of which contain one or more user interface control objects, from the invariance parameters corresponding to the user interface control objects and to the application screen images.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5 wherein the processing each captured application screen image with the machine learning model, as trained by the initial dataset, to detect user interface control objects in each captured screen image, comprises:
generating for each captured application screen image, a set of detected bounding boxes, each of the bounding boxes having associated therewith an object class label that identifies an object class associated with a user interface control object within the associated bounding box and a confidence score indicative of a level of confidence of identification of the user interface control object within the associated bounding box.

### Claim 7 (depends on 5)

The computer-implemented method of claim 5 further comprising, receiving from the user a designation of a bounding box surrounding an undetected or incorrectly detected user interface control object, and further receiving from the user an object class label corresponding to the user interface control object within the bounding box designated by the user.

### Claim 8 (depends on 5)

The computer-implemented method of claim 5 further comprising applying a privacy constraint to one or more of detected user interface control objects to remove information in the one or more of detected user interface control objects.

### Claim 9 (depends on 5)

The computer-implemented method of claim 5 wherein synthesizing one or more new application screen images, each of which contain one or more user interface control objects, from the invariance parameters corresponding to the user interface control objects and to the application screen images comprises:
for a selected one of the captured screen images, generating for a new application screen image a monochromatic background color as a function of a dominant background color in the selected one of the captured screen images; and
generating for placement on the new application screen image user interface control objects at the same locations as user interface control objects in the selected one of the captured screen images and applying a background color to each generated user interface control object to cause textual input in the generated user interface control objects to be removed.

### Claim 10 (depends on 5)

The computer-implemented method of claim 5 wherein synthesizing one or more new application screen images, each of which contain one or more user interface control objects, from the invariance parameters corresponding to the user interface control objects and to the application screen images comprises:
for a selected one of the captured screen images, generating a new application screen image by generating for each text string in the selected one of the captured screen images a random text string and placing each generated random text strings at a location occupied by a text string in the selected one of the captured screen images.

### Claim 11 (depends on 5)

The computer-implemented method of claim 5 wherein synthesizing one or more new application screen images, each of which contain one or more user interface control objects, from the invariance parameters corresponding to the user interface control objects and to the application screen images comprises:
for a selected one of the captured screen images, generating a new application screen image by
selecting a subset of user interface control objects detected in the selected one of the captured screen images;
selecting a subset of text strings detected in the selected one of the captured screen images;
generating a uniform background color for the new application screen image; and
placing the selected subset of user interface control objects and the selected subset of text strings onto the new application screen image.

### Claim 12

A computer system comprising:
data storage having stored thereupon an initial dataset comprising a plurality of images, wherein each image represents a screen of information containing one or more user interface control objects, wherein each user interface control object is part of a corresponding user interface object class; and
a processor, programmed with instructions that cause the processor to train a machine learning model to detect one or more user interface objects contained in a screen image of a user interface generated by an application program by:
training the machine learning model with a portion of the initial dataset to cause the machine learning model to detect user interface control objects in each of a plurality of object classes;
modifying the initial dataset by,
processing screen images generated by a first application program to detect user interface control objects in the screen images generated by the first application program,
creating a set of new application screen images as a function of characteristics corresponding to the user interface control objects detected in the screen images generated by the first application program, and
generating an updated dataset by modifying the initial dataset with the set of new application screen images; and
further training the machine learning model with the updated dataset.

### Claim 13 (depends on 12)

The computer system of claim 12 wherein processing the screen images generated by the first application program to detect user interface control objects in the screen images generated by the first application program comprises:
retrieving from the initial dataset a set of tagged screen images where each screen image of the set of tagged screen images comprises, for one or more user interface control objects in the screen image, a set of tagged bounding boxes, each tagged bounding box surrounding a corresponding user interface control object, and a tagged object class identifier that classifies the corresponding user interface control object into one of a plurality of object classes.

### Claim 14 (depends on 13)

The computer system of claim 13 wherein processing the screen images generated by the first application program to detect user interface control objects in the screen images generated by the first application program further comprises:
processing the set of tagged screen images with the machine learning model, as trained by the initial dataset, to generate for each tagged screen image of the set of tagged screen images, a set of detected bounding boxes each of the bounding boxes having associated therewith an object class label that identifies an object class associated with a user interface control object within the associated bounding box and a confidence score indicative of a level of confidence of identification of the user interface control object within the associated bounding box;
comparing for each tagged screen image, the set of detected bounding boxes to the set of tagged bounding boxes to generate a set of failure cases comprising missed or incorrectly identified bounding boxes;
collecting invariance parameters associated with each user interface control object contained in the set of failure cases; and
generating a set of updated invariance parameters by updating invariance parameters employed by the machine learning engine as a function of the invariance parameters associated with each user interface control object contained in the set of failure cases.

### Claim 15 (depends on 14)

The computer system of claim 14 wherein creating a set of new application screen images as a function of characteristics corresponding to the user interface control objects detected in the screen images generated by the first application program comprises:
generating for each new application screen image of the set of new application screen images, one or more user interface control objects as a function of the updated invariance parameters.

### Claim 16 (depends on 12)

The computer system of claim 12 processing the screen images generated by the first application program to detect user interface control objects in the screen images generated by the first application program comprises:
capturing application screen images generated by the first application program as the first application program is used by a user of the first application program and capturing user inputs entered by the user;
processing each captured application screen image with the machine learning model, as trained by the initial dataset, to detect user interface control objects in each captured screen image;
validating for each captured screen image, detected user interface control objects in each captured screen image, as a function of user inputs entered into the application screen of the first application program that corresponds to each captured screen image;
employing validated user interface control objects for a captured screen image to extract object level characteristics for each user interface control object in the application screen image;
computing for each user interface control object in the application screen image one or more invariance parameters;
computing for each application screen image one or more invariance parameters; and
synthesizing one or more new application screen images, each of which contain one or more user interface control objects, from the invariance parameters corresponding to the user interface control objects and to the application screen images.

### Claim 17

A non-transitory storage medium, having stored thereupon one or more program modules comprising computer-executable instructions for execution on a computer system, the computer-executable instructions executing on a server processor to cause the computer system to perform a computer-implemented method for detecting one or more user interface control objects contained in a screen image of a user interface generated by an application program, the computer-implemented method comprising:
training a machine learning model with a portion of an initial dataset comprising a plurality of images to cause the machine learning model to detect user interface control objects in each of a plurality of object classes, wherein each image represents a screen of information containing one or more user interface control objects, wherein each user interface control object is part of a corresponding user interface object class;
modifying the initial dataset by,
processing screen images generated by a first application program to detect user interface control objects in the screen images generated by the first application program,
creating a set of new application screen images as a function of the user interface control objects detected in the screen images generated by the first application program, and
generating an updated dataset by modifying the initial dataset with the set of new application screen images; and
further training the machine learning model with the updated dataset.

### Claim 18 (depends on 17)

The non-transitory storage medium of claim 17 wherein the operation of processing the screen images generated by the first application program to detect user interface control objects in the screen images generated by the first application program comprises:
retrieving from the initial dataset a set of tagged screen images where each screen image of the set of tagged screen images comprises, for one or more user interface control objects in the screen image, a set of tagged bounding boxes, each tagged bounding box surrounding a corresponding user interface control object, and a tagged object class identifier that classifies the corresponding user interface control object into one of a plurality of object classes.

### Claim 19 (depends on 18)

The non-transitory storage medium of claim 18 wherein the operation of processing the screen images generated by the first application program to detect user interface control objects in the screen images generated by the first application program further comprises:
processing the set of tagged screen images with the machine learning model, as trained by the initial dataset, to generate for each tagged screen image of the set of tagged screen images, a set of detected bounding boxes each of the bounding boxes having associated therewith an object class label that identifies an object class associated with a user interface control object within the associated bounding box and a confidence score indicative of a level of confidence of identification of the user interface control object within the associated bounding box;
comparing for each tagged screen image, the set of detected bounding boxes to the set of tagged bounding boxes to generate a set of failure cases comprising missed or incorrectly identified bounding boxes;
collecting invariance parameters associated with each user interface control object contained in the set of failure cases; and
generating a set of updated invariance parameters by updating invariance parameters employed by the machine learning engine as a function of the invariance parameters associated with each user interface control object contained in the set of failure cases.

### Claim 20 (depends on 19)

The non-transitory storage medium of claim 19 wherein the operation of processing the screen images generated by the first application program to detect user interface control objects in the screen images generated by the first application program comprises:
capturing application screen images generated by the first application program as the first application program is used by a user of the first application program and capturing user inputs entered by the user;
processing each captured application screen image with the machine learning model, as trained by the initial dataset, to detect user interface control objects in each captured screen image;
validating for each captured screen image, detected user interface control objects in each captured screen image, as a function of user inputs entered into the application screen of the first application program that corresponds to each captured screen image;
employing validated user interface control objects for a captured screen image to extract object level characteristics for each user interface control object in the application screen image;
computing for each user interface control object in the application screen image one or more invariance parameters;
computing for each application screen image one or more invariance parameters; and
synthesizing one or more new application screen images, each of which contain one or more user interface control objects, from the invariance parameters corresponding to the user interface control objects and to the application screen images.

### Claim 21 (depends on 17)

The non-transitory storage medium of claim 17 wherein creating a set of new application screen images as a function of the user interface control objects makes use of invariance characteristics corresponding to the user interface control objects, and wherein the invariance characteristics corresponding to the user interface control objects comprise one or more of height, width, aspect ratio, color and text.

### Claim 22 (depends on 3)

The computer-implemented method of claim 3 wherein the invariance parameters associated with each user interface control object comprise one or more of height, width, aspect ratio, color and text associated with a user interface control object.

### Claim 23 (depends on 5)

The computer-implemented method of claim 5 wherein the invariance parameters corresponding to the application screen images comprise arrangement of user interface control objects in a corresponding screen image, and resolution and noise in the corresponding screen image.

### Claim 24 (depends on 12)

The computer-implemented method of claim 12 wherein the invariance characteristics corresponding to the user interface control objects comprise one or more of height, width, aspect ratio, color and text.
