Learning user interface controls via incremental data synthesis
AssigneeAutomation Anywhere, Inc.
InventorsSudhir Kumar Singh, Jesse Truscott, Virinchipuram J Anand
Life of this patent
Estimating the term…
Loading patent…
AssigneeAutomation Anywhere, Inc.
InventorsSudhir Kumar Singh, Jesse Truscott, Virinchipuram J Anand
Estimating the term…
A User Interface (UI) interface object detection system employs an initial dataset comprising a set of images, that may include synthesized images, to train a Machine Learning (ML) engine to generate an initial trained model. A data point generator is employed to generate an updated synthesized image set which is used to further train the ML engine. The data point generator may employ images generated by an application program as a reference by which to generate the updated synthesized image set. The images generated by the application program may be tagged in advance. Alternatively, or in addition, the images generated by the application program may be captured dynamically by a user using the application program.
A computer-implemented method for detecting one or more user interface control objects contained in a screen image of a user interface generated by an application program, the method comprising:
A computer system comprising:
A non-transitory storage medium, having stored thereupon one or more program modules comprising computer-executable instructions for execution on a computer system, the computer-executable instructions executing on a server processor to cause the computer system to perform a computer-implemented method for detecting one or more user interface control objects contained in a screen image of a user interface generated by an application program, the computer-implemented method comprising:
Dependencies are inferred from the claim text.