Automated detection of controls in computer applications with region based detectors

A computerized method for automated detection of application level controls displayed by a computer application, comprising:

• retrieving a first dataset comprising images containing a first type of application control, wherein each first type of application control of a plurality of first type of application controls in each image in the first dataset is characterized by dimensions where width of the first type of application control is greater than height of the first type of application control;
• retrieving a second dataset comprising images containing a second type of application control, wherein each second type of application control of a plurality of second type of application controls in each image in the second dataset is characterized by dimensions where width of the second type of application control is substantially equal to height of the second type of application control;
• retrieving a third dataset comprising images containing a third type of application control, wherein each third type of application control of a plurality of third type of application controls in each image in the third dataset is characterized by dimensions where height of the third type of application control is greater than width of the third type of application control;
• processing each of the first, second and third datasets with a region-based faster R-CNN engine to generate:
• a first trained region-based faster R-CNN engine that is trained to recognize application controls characterized by dimensions where width is greater than height,
• a second trained region-based faster R-CNN engine that is trained to recognize application controls characterized by dimensions where width is substantially equal to height, and
• a third trained region-based faster R-CNN engine that is trained to recognize application controls characterized by dimensions where height is greater than width, and
• combining the first, second and third trained region-based faster R-CNN engines to generate a software robot configured to recognize, from an input image generated from a screen of an application program, application controls characterized by dimensions where width is greater than height, where width is substantially equal to height, or where height is greater than width.

A robotic process automation system comprising:

• data storage for storing a plurality of images of application user interfaces, wherein each application user interface of the application user interfaces comprises one or more application controls usable by a human user to interact with an application that generates the application user interface;
• a server processor operatively coupled to the data storage and configured to execute instructions that when executed cause the server processor to recognize one or more of the application controls, by:
• retrieving a first dataset comprising images containing a first type of application control, wherein each first type of application control of a plurality of first type of application controls in each image in the first dataset is characterized by dimensions where width of the first type of application control is greater than height of the first type of application control;
• retrieving a second dataset comprising images containing a second type of application control, wherein each second type of application control of a plurality of second type of application controls in each image in the second dataset is characterized by dimensions where width of the second type of application control is substantially equal to height of the second type of application control;
• retrieving a third dataset comprising images containing a third type of application control, wherein each third type of application control of a plurality of third type of application controls in each image in the third dataset is characterized by dimensions where height of the third type of application control is greater than width of the third type of application control;
• processing each of the first, second and third datasets with a region-based R-FCN engine to generate:
• a first trained region-based R-FCN engine that is trained to recognize application controls characterized by dimensions where width is greater than height,
• a second trained region-based R-FCN engine that is trained to recognize application controls characterized by dimensions where width is substantially equal to height,
• a third trained region-based R-FCN engine that is trained to recognize application controls characterized by dimensions where height is greater than width;
• and
• combining the first, second and third trained region-based R-FCN engines to generate a first software robot configured to recognize, from an input image, application controls wherein the application controls are characterized by dimensions where width is greater than height, where width is substantially equal to height, or where height is greater than width.