Optimizing mixture of experts, moe, integration into neural network architectures

A non-transitory machine-readable storage medium on which is stored a computer program including machine-readable instructions for determining, in a neural network that includes N layers and is configured for classification and/or regression of sensor data, one or more optimal layers for integration of Mixture of Experts (MoE) functionality, the MoE functionality including a plurality of M distinct processing blocks and a router block that routes each input to one or more processing blocks for processing, the instructions, when executed by one or more computers and/or compute instances, causing the one or more computers and/or compute instances to perform the following steps:

• constructing candidate versions of the neural network, wherein, in each candidate version of the candidate versions, one or more layers are replaced with surrogate layers, wherein:
• each surrogate layer is computationally cheaper to train than a respective MoE layer with full MoE functionality, while
• performance of the candidate neural network with the surrogate layer is commensurate with performance that the neural network would have with the MoE layer in place of the surrogate layer;
• training, using training examples of sensor data, each candidate version of the neural network;
• determining, using test and/or validation samples of sensor data for which respective ground truth outputs of the neural network are known, an accuracy with which the trained candidate version of the neural network reproduces the ground truth outputs;
• and
• determining the layers that are replaced with surrogate layers in a candidate version of the neural network with a best accuracy as optimal layers for the integration of MoE functionality.

One or more computers and/or compute instances with q non-transitory machine-readable storage medium on which is stored a computer program including machine-readable instructions for determining, in a neural network that includes N layers and is configured for classification and/or regression of sensor data, one or more optimal layers for integration of Mixture of Experts (MoE) functionality, the MoE functionality including a plurality of M distinct processing blocks and a router block that routes each input to one or more processing blocks for processing, the instructions, when executed by the one or more computers and/or compute instances, causing the one or more computers and/or compute instances to perform the following steps:

• constructing candidate versions of the neural network, wherein, in each candidate version of the candidate versions, one or more layers are replaced with surrogate layers, wherein:
• each surrogate layer is computationally cheaper to train than a respective MoE layer with full MoE functionality, while
• performance of the candidate neural network with the surrogate layer is commensurate with performance that the neural network would have with the MoE layer in place of the surrogate layer;
• training, using training examples of sensor data, each candidate version of the neural network;
• determining, using test and/or validation samples of sensor data for which respective ground truth outputs of the neural network are known, an accuracy with which the trained candidate version of the neural network reproduces the ground truth outputs;
• and
• determining the layers that are replaced with surrogate layers in a candidate version of the neural network with a best accuracy as optimal layers for the integration of MoE functionality.