Incremental training of machine learning tools

A method comprising:

• producing operational parameters of a machine learning tool based on a primary set of training data;
• applying input data to the machine learning tool being used in an inference mode to generate an output of the machine learning tool;
• in response to determining a measure of prediction quality of the output of the machine learning tool is below a threshold, initiating incremental training of the operational parameters using the input data as training data for the machine learning tool;
• and
• storing updated operational parameters of the machine learning tool, the updated operational parameters being based on the incremental training.

A method of training operational parameters of a machine learning tool, the method comprising:

• training the operational parameters of the machine learning tool based on an initial set of training data;
• transmitting the operational parameters of the machine learning tool to an edge device via an interconnection network;
• receiving additional training data from the edge device, the additional training data selected based on a measure of quality applied to an output of the machine learning tool executing at the edge device;
• performing incremental training of the operational parameters using the additional training data received from the edge device to generate updated operational parameters;
• and
• transmitting the updated operational parameters to the edge device.