Machine learned retraining for detection of user interface controls via variance parameters

A tangible storage medium, having stored thereupon computer program code for execution on a computer system, the computer program code executing on a server processor to cause the computer system to perform a computer-implemented method for detecting one or more user interface objects contained in a screen image of a user interface generated by an application program, the computer-implemented method comprising:

• selecting a dataset that comprises a plurality of object classes, each object class corresponding to a type of user interface object and comprising at least one variance parameter;
• training a machine learning model with a portion of the dataset designated as a training portion of the dataset to cause the machine learning model to detect, in the training portion of the dataset, objects in each of the plurality of object classes;
• testing the machine learning model with a portion of the dataset designated as a testing portion of the dataset to determine whether the machine learning model can detect objects in the plurality of object classes with an accuracy above a predetermined set of test metrics;
• wherein based on the machine learning model exhibiting an accuracy that does not meet the predetermined set of test metrics, the method further comprises the operations of:
• processing the testing portion of the dataset to generate, for each object class in the testing portion of the dataset, a numerical value that represents failure of the machine learning model with respect to each variance parameter of a corresponding object class,
• generating a first set of variance parameters by selecting from among the numerical values, variance parameters that fall within a predetermined variance range,
• generating an updated dataset comprising updated object classes by, for each variance parameter within the first set of variance parameters, and
• retraining the machine learning model with a portion of the updated dataset designated as an updated training portion of the dataset to cause the machine learning model to detect, in the updated training portion of the dataset, objects in each of the plurality of object classes.

A computer system, comprising:

• data storage having stored thereupon a plurality of image files, each image file containing one or more user interface objects of a user interface generated by an application program;
• and
• a processor, programmed with instructions that cause the processor to train, test and retrain a machine learning model, the machine learning model being used to detect one or more user interface objects contained in a screen image of a user interface generated by an application program by way of invariance guided learning, the instructions when executed by the processor operate to at least:
• selecting a dataset that comprises a plurality of object classes, each object class corresponding to a type of user interface object and comprising at least one variance parameter;
• training a machine learning model with a portion of the dataset designated as a training portion of the dataset to cause the machine learning model to detect, in the training portion of the dataset, objects in each of the plurality of object classes;
• testing the machine learning model with a portion of the dataset designated as a testing portion of the dataset to determine whether the machine learning model can detect objects in the plurality of object classes with an accuracy threshold;
• when the machine learning model exhibits an accuracy that does not meet the accuracy threshold, the method further comprises the operations of:
• processing the testing portion of the dataset to generate, for each object class in the testing portion of the dataset, a numerical value that represents failure of the machine learning model with respect to each variance parameter of a corresponding object class;
• generating an updated dataset based on the numerical values for the variance parameters;
• and
• retraining the machine learning model with a portion of the updated dataset designated as an updated training portion of the dataset to cause the machine learning model to detect, in the updated training portion of the dataset, objects in each of the plurality of object classes.