Detection and definition of virtual objects in remote screens

A robotic process automation system comprising:

• data storage that stores a screen signature database;
• and
• a processor operatively coupled to the data storage and configured to perform operations that when executed cause the processor to automate usage of an application program by detecting objects and text from a first screen of information generated by the application program for display on a computer monitor by:
• capturing a first image of the screen of information;
• generating a first signature to identify the first image;
• analyzing the first image to identify a set of actionable objects within the first image;
• performing optical character recognition to detect text fields in the first image;
• linking each identified actionable object to each detected text field positioned proximate to the identified actionable object in the first image;
• in the screen signature database, storing for the first image, the first signature, which is associated witha smart screen, wherein the smart screen stores the links between each actionable object and its correspondingly linked text field;
• and
• subsequently interacting with the application, and for a second screen of information displayed by the application,
• capturing a second image of a second screen of information displayed by the application program,
• generating a second signature to identify the second image;
• in the screen signature database, searching for a previously stored signature, which comprises at least the first signature, that matches the second signature,
• if the second signature matches a previously stored signature, then using the stored matching signature to retrieve and utilize a previously stored smart screen, that is associated with the stored matching signature, to process the second image.

A tangible storage medium, having stored thereupon one or more program modules comprising computer-executable instructions for execution on a computer system, the computer-executable instructions executing on a server processor to cause the computer system to implement a computer-implemented method for automating usage of an application program comprising:

• detecting objects and text from a first screen of information generated by the application program for display on a computer monitor by,
• capturing a first image of the screen of information;
• generating a first signature to identify the first image;
• analyzing the first image to identify a set of actionable objects within the first image;
• performing optical character recognition to detect text fields in the first image;
• linking each identified actionable object to each detected text field positioned proximate to the identified actionable object in the first image;
• in a screen signature database, storing for the first image, the first signature, which is associated with a smart screen, wherein the smart screen stores the links between each actionable object and its correspondingly linked text field;
• and
• subsequently interacting with the application, and for a second screen of information displayed by the application,
• capturing a second image of a second screen of information displayed by the application program,
• generating a second signature to identify the second image;
• in the screen signature database, searching for a previously stored signature, which comprises at least the first signature, that matches the second signature, if the second signature matches a previously stored signature, then using the stored matching signature to retrieve and utilize a previously stored smart screen, that is associated with the stored matching signature, to process the second image.