Automatic identification of types of user interface components

A non-transitory computer-readable medium having instructions stored thereon that are capable of execution by a computing device to perform operations comprising:

• determining, based on user interface code, a plurality of visible elements of a graphical user interface;
• determining, based on the user interface code, coordinates for bounding boxes for ones of the plurality of visible elements;
• grouping the visible elements into at least first and second groups;
• determining types of elements within the first and second groups;
• in response to detecting a match between the types of elements within the first and second groups, determining a similarity metric for the first and second groups based on the coordinates of determined bounding boxes within the first and second groups, wherein detecting the match includes:
• determining whether a number of elements of each element type matches to within a threshold similarity for the first and second groups of elements;
• and
• determining whether an element displayed at a position within the first group of elements is the same element type as an element displayed at a similar position within the second group of elements;
• and
• storing information defining a component type that corresponds to the first and second groups in response to the similarity metric meeting a threshold value.

An apparatus comprising:

• one or more processing elements configured to:
• determine, based on user interface code, a plurality of visible elements of a graphical user interface;
• determine, based on the user interface code, coordinates for bounding boxes for ones of the plurality of visible elements;
• group the visible elements into at least first and second groups;
• determine types of elements within the first and second groups;
• in response to detecting a match between the types of elements within the first and second groups, determine a similarity metric for the first and second groups based on the coordinates of determined bounding boxes within the first and second groups, wherein detecting the match includes:
• determining whether a number of elements of each element type matches to within a threshold similarity for the first and second groups of elements;
• and
• determining whether an element displayed at a position within the first group of elements is the same element type as an element displayed at a similar position within the second group of elements;
• and
• store information defining a component type that corresponds to the first and second groups in response to the similarity metric meeting a threshold value.