Multi-task machine learning architectures and training procedures

A system comprising:

• a hardware processing unit;
• and
• a storage resource storing computer-readable instructions which, when executed by the hardware processing unit, cause the system to:
• access a multi-task student instance, the multi-task student instance being a trained model that has been trained using first outputs of one or more first multi-task teacher instances to train a first task-specific layer of the multi-task student instance and using second outputs of one or more second multi-task teacher instances to train a second task-specific layer of the multi-task student instance;
• receive input data;
• provide the input data to the multi-task student instance;
• obtain a first task-specific result produced by the first task-specific layer of the multi-task student instance;
• and
• output the first task-specific result,
• wherein the multi-task student instance, the one or more first multi-task teacher instances, and the one or more second multi-task teacher instances are instances of a multi-task machine learning model having one or more shared layers, the first task-specific layer, and the second task-specific layer.

One or more computer-readable storage media storing instructions which, when executed by one or more hardware processing units, cause the one or more hardware processing units to perform acts comprising:

• evaluating a plurality of candidate multi-task teacher instances of a multi-task machine learning model, each candidate multi-task teacher instance having one or more shared layers, a first task-specific layer that performs a first task, and a second task-specific layer that performs a second task;
• based at least on the evaluating, selecting one or more first multi-task teacher instances of the plurality of candidate multi-task teacher instances for the first task and one or more second multi-task teacher instances of the plurality of candidate multi-task teacher instances for the second task;
• and
• training a multi-task student instance of the multi-task machine learning model, the multi-task student instance being a trained model having the one or more shared layers, the first task-specific layer, and the second task-specific layer, the training comprising using first outputs of the one or more first multi-task teacher instances to train the first task-specific layer of the multi-task student instance and using second outputs of the one or more second multi-task teacher instances to train the second task-specific layer of the multi-task student instance.