Allocating computing resources during continuous retraining

On a computing device comprising a machine learning model configured to analyze video data, a method for allocating computing resources and selecting hyperparameter configurations during continuous retraining and operation of the machine learning model, the continuous retraining and operation comprising a plurality of jobs including, for each video stream of one or more video streams, an inference job and a retraining job, the method comprising:

• storing an initial resource allocation for the plurality of jobs as a stored resource allocation, storing an initial hyperparameter configuration as a stored hyperparameter configuration, and storing an inference accuracy for the initial hyperparameter configuration and the initial resource allocation as a stored inference accuracy;
• for the plurality of jobs, iteratively:
• designating a selected job of the plurality of jobs as a thief, and designating one or more other jobs each as a victim;
• for the thief and the one or more victim:
• reallocating an amount of computing resources from each victim to the thief to form an updated allocation,
• testing a plurality of hyperparameter configurations for the machine learning model using the updated allocation to determine an updated inference accuracy, and
• when the updated inference accuracy is more accurate than the stored inference accuracy, then
• storing a selected hyperparameter configuration that produced the updated inference accuracy as the stored hyperparameter configuration,
• storing the updated inference accuracy as the stored inference accuracy, and
• storing the updated allocation as the stored resource allocation;
• and
• after iterating through each job of the plurality of jobs as the thief, during a retraining window, retraining the machine learning model and making inferences with the machine learning model using the stored resource allocation and the stored hyperparameter configuration.

A computing device configured to be located at a network edge between a local network and a cloud service, the computing device comprising:

• a processor;
• and
• a memory storing instructions executable by the processor to perform continuous retraining and operation of a machine learning model configured to analyze video, the continuous retraining and operation comprising a plurality of jobs including, for each video stream of one or more video streams, a retraining job and an inference job, wherein the instructions are executable to:
• receive a video stream;
• during a retraining window, select a portion of the video stream for labeling;
• obtain a labeled retraining data set for the portion of the video stream that was selected for labeling comprising labels for one or more objects identified in the portion of the video stream;
• select one or more of a hyperparameter configuration and a computing resource allocation to use for the continuous retraining and operation of the machine learning model by testing one or more of a plurality of hyperparameter configurations and a plurality of computing resource allocations using an average inference accuracy over the retraining window as a testing metric;
• and
• retrain and operate the machine learning model using the one or more of the hyperparameter configuration and the computing resource allocation selected.