System and method for generating intermediate predictions in trained machine learning models

A non-transitory computer-readable medium on which are stored data representing instructions for classifying input data using a trained machine learning model, wherein the instructions are executed using a processing subsystem of a system, wherein the processing subsystem has a computational budget for classifying the input data in real-time or near-real-time, wherein the computational budget is a dynamic computational budget which changes over time, wherein the trained machine learning model includes one or more input layers, a plurality of intermediate layers, and one or more output layers to generate an output of the trained machine learning model, the instructions, when executed by the processing subsystem, causing the processing subsystem to perform the following steps:

• feeding input data into the trained machine learning model;
• partially performing a forward pass by partially propagating the input data through a part of the trained machine learning model, wherein an extent of the forward pass and thereby the part of the trained machine learning model through which the input data is propagated are determined by a currently available computational budget;
• obtaining a plurality of intermediate predictions from respective outputs of a plurality of layers in the part of the trained machine learning model;
• ensembling the plurality of intermediate predictions to obtain an ensemble prediction, wherein the ensembling includes determining the ensemble prediction as a product of weighted versions of the plurality of intermediate predictions and having a normalized probability density;
• and
• using the ensemble prediction as a substitute for the output of the trained machine learning model in the classifying of the input data.

A system, comprising:

• a processing subsystem classify input data using a trained machine learning model, wherein the processing subsystem has a computational budget for classifying the input data in real-time or near-real-time, wherein the computational budget is a dynamic computational budget which changes over time, wherein the trained machine learning model includes one or more input layers, a plurality of intermediate layers, and one or more output layers to generate an output of the trained machine learning model, the processing system configured to:
• feed input data into the trained machine learning model;
• partially perform a forward pass by partially propagating the input data through a part of the trained machine learning model, wherein an extent of the forward pass and thereby the part of the trained machine learning model through which the input data is propagated are determined by a currently available computational budget;
• obtain a plurality of intermediate predictions from respective outputs of a plurality of layers in the part of the trained machine learning model;
• ensemble the plurality of intermediate predictions to obtain an ensemble prediction, wherein the ensembling includes determining the ensemble prediction as a product of weighted versions of the plurality of intermediate predictions and having a normalized probability density;
• and
• use the ensemble prediction as a substitute for the output of the trained machine learning model in the classifying of the input data.