Detecting user interface elements in robotic process automation using convolutional neural networks

A computer-implemented method, comprising:

• augmenting, by a computing system, a raw dataset using channel randomization, hue shift, color inversion, random cropping, random scaling, blurring of images, or any combination thereof, to produce an augmented dataset;
• training a convolutional neural network (CNN), by the computing system, using the augmented dataset;
• and
• detecting graphical elements in a user interface (UI), by the computing system, using the trained CNN.

A system, comprising:

• memory storing computer program instructions;
• and
• at least one processor configured to execute the computer program instructions, the at least one processor configured to:
• detect graphical elements in a user interface (UI) using a Faster Region-based Convolutional Neural Network (R-CNN) architecture with a region proposal network (RPN) backbone, the detection comprising overlaying rectangles over an image as a grid and providing a predetermined number of proposals for each rectangle in the grid, wherein
• each time two rectangles are compared, an intersection over a union or an intersection over a minimum is used with a given threshold, and
• the threshold is an adaptive threshold that depends on an area of a given rectangle.