Determining functional and descriptive elements of application images for intelligent screen automation

Abstract

The presently disclosed inventive concepts are directed to systems, computer program products, and methods for intelligent screen automation. The inventive techniques include: identifying first and second sets of elements within one or more images of a user interface, where each of the first set of elements is independently selected from: vertical lines, horizontal lines, and rectangular elements, and each of the second set of elements is independently selected from: radio buttons, icons, and textual elements. The methods also include determining one or more logical relationships between the textual elements and non-textual elements of the image; building a hierarchy comprising some or all of the first set of elements and some or all of the second set of elements in order to form a tree-based data structure representing functionality of the user interface; and outputting the tree-based data structure to a memory.

Dependencies are inferred from the claim text.