Systems and methods for automatic evaluation of rendered user interface using machine learning

A computing method for automated visual testing of graphical user interface designs using image transformations for scoring and evaluation, the method comprising:

• receiving a candidate data object and a reference data object from a user, each data object representing a graphical user interface design;
• transforming the candidate and reference data objects into a candidate and reference machine-encoded text objects using optical character recognition for comparison of textual and layout features to generate a first similarity score;
• segmenting the candidate and reference machine-encoded text objects into a candidate and reference sets of pixel clusters using object masks;
• generating a candidate and reference image histograms based on the distribution of pixels of the candidate and reference sets of pixel clusters for comparison of color distribution characteristics as between the candidate and reference image histograms to generate a second similarity score;
• generating a candidate and reference embedding vectors through image embedding based on the candidate and reference sets of pixel clusters for calculation of a third similarity score, wherein the third similarity score is a cosine similarity score between the candidate and reference embedding vectors;
• localizing sub-clusters of the candidate and reference sets of pixel clusters to perform template matching between the candidate and reference data objects to generate a fourth similarity score;
• computing and extracting a candidate and reference sets of descriptor objects based on local intensity extrema of the candidate and reference sets of pixel clusters for comparison to generate a fifth similarity score, each descriptor object representing a feature point in the data objects, the comparison involving each descriptor object of the candidate set of descriptor objects being compared to each descriptor object of the reference set of descriptor objects;
• transforming the candidate and reference sets of pixel clusters into greyscale format for comparison of luminance and contrast between each pixel of the candidate set of pixel clusters and each pixel of the reference set of pixel clusters to generate a sixth similarity score;
• providing the first, second, third, fourth, fifth and sixth similarity scores to a trained machine learning model data architecture for scanning and comparison against a set of passing threshold values;
• and
• generating an output structured data object based on the comparison of the first, second, third, fourth, fifth, and sixth similarity scores against the set of passing threshold values, the output structured data object containing a list of ordered pairs, wherein each ordered pair comprises a candidate pixel cluster of the candidate set of pixel clusters and a corresponding reference pixel cluster of the reference set of pixel clusters, wherein the candidate pixel cluster and the corresponding reference pixel cluster are detected by the trained machine learning model data architecture as being a difference between the candidate and reference data objects, wherein the output structured data object can be used by the user to correct the difference between the candidate and reference data objects for each ordered pair of the list of ordered pairs.

A non-transitory computer readable medium storing computer interpretable instructions, which when executed by a computer processor, cause the computer processor to perform a method for automated visual testing of graphical user interface designs using image transformations for scoring and evaluation, the method comprising:

• receiving a candidate data object and a reference data object from a user, each data object representing a graphical user interface design;
• transforming the candidate and reference data objects into a candidate and reference machine-encoded text objects using optical character recognition for comparison of textual and layout features to generate a first similarity score;
• segmenting the candidate and reference machine-encoded text objects into a candidate and reference sets of pixel clusters using object masks;
• generating a candidate and reference image histograms based on the distribution of pixels of the candidate and reference sets of pixel clusters for comparison of color distribution characteristics as between the candidate and reference image histograms to generate a second similarity score;
• generating a candidate and reference embedding vectors through image embedding based on the candidate and reference sets of pixel clusters for calculation of a third similarity score, wherein the third similarity score is a cosine similarity score between the candidate and reference embedding vectors;
• localizing sub-clusters of the candidate and reference sets of pixel clusters to perform template matching between the candidate and reference data objects to generate a fourth similarity score;
• computing and extracting a candidate and reference sets of descriptor objects based on local intensity extrema of the candidate and reference sets of pixel clusters for comparison to generate a fifth similarity score, each descriptor object representing a feature point in the data objects, the comparison involving each descriptor object of the candidate set of descriptor objects being compared to each descriptor object of the reference set of descriptor objects;
• transforming the candidate and reference sets of pixel clusters into greyscale format for comparison of luminance and contrast between each pixel of the candidate set of pixel clusters and each pixel of the reference set of pixel clusters to generate a sixth similarity score;
• providing the first, second, third, fourth, fifth and sixth similarity scores to a trained machine learning model data architecture for scanning and comparison against a set of passing threshold values;
• and
• generating an output structured data object based on the comparison of the first, second, third, fourth, fifth, and sixth similarity scores against the set of passing threshold values, the output structured data object containing a list of ordered pairs, wherein each ordered pair comprises a candidate pixel cluster of the candidate set of pixel clusters and a corresponding reference pixel cluster of the reference set of pixel clusters, wherein the candidate pixel cluster and the corresponding reference pixel cluster are detected by the trained machine learning model data architecture as being a difference between the candidate and reference data objects, wherein the output structured data object can be used by the user to correct the difference between the candidate and reference data objects for each ordered pair of the list of ordered pairs.