Method of and system for online machine learning with dynamic model evaluation and selection

A method for generating a labelled training dataset for training a set of machine learning models for performing a prediction for a given problem, the method being executed by a processing device, the processing device being connected to a non-transitory storage medium comprising:

• a plurality of unlabelled feature vectors for the given problem, each of the plurality of unlabelled feature vectors having respective values for a set of features,
• the method comprising:
• initializing, by the set of supervised machine learning algorithms, using the set of features associated with the given problem, the set of machine learning models;
• obtaining, from the non-transitory storage medium comprising the plurality of unlabelled feature vectors, a first unlabelled feature vector for the given problem;
• obtaining, for the first unlabelled feature vector, a first prediction label to thereby obtain a first labelled feature vector;
• adding the first labelled feature vector associated with the first prediction label to the labelled training dataset;
• obtaining, from the non-transitory storage medium comprising the plurality of unlabelled feature vectors, a set of unlabelled feature vectors;
• determining, for each of the set of unlabelled feature vectors, using the set of machine learning models, a respective set of predictions;
• determining, for each of the set of unlabelled feature vectors, using the respective set of predictions, a respective disagreement score indicative of a difference in predictions of the set of machine learning models;
• selecting, based on the respective disagreement score, from the set of unlabelled feature vectors, a second unlabelled feature vector;
• obtaining, for the second unlabelled feature vector, a second prediction label to thereby obtain a second labelled feature vector;
• and
• adding, the second labelled feature vector associated with the second prediction label to the labelled training dataset.

A system for providing a recommendation for a given problem by using a set of machine learning models online, the given problem being associated with a set of features, the system comprising:

• a processing device;
• a non-transitory computer-readable storage medium operatively connected to the processing device, the non-transitory computer-readable storage medium comprising instructions;
• the processing device, upon executing the instructions, being configured for:
• initializing, using a set of features associated with a given problem, a set of supervised machine learning algorithms;
• upon detection of a suitable initialization condition:
• providing, by the set of supervised machine learning algorithms, a set of machine learning models for online inference;
• obtaining a feature vector for the given problem;
• generating, by the set of machine learning models, using the feature vector, a set of predictions, each respective prediction being associated with a respective accuracy metric;
• providing, based on at least one of the set of predictions, the recommendation associated with a first prediction having been generated by at least one of the set of machine learning models;
• obtaining an actual decision associated with the feature vector for the given problem;
• and
• training, by the set of supervised machine learning algorithms, the set of machine learning models using the feature vector and the actual decision as a label.