Sparse and differentiable mixture of experts neural networks

A system for performing one or more machine learning tasks on a network input to generate one or more network outputs, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement:

• a main neural network configured to receive the network input and to process the network input to generate the one or more network outputs, the main neural network comprising a Mixture of Experts (MoE) subnetwork, wherein the MoE subnetwork comprises:
• a plurality of expert neural networks, wherein each expert neural network is configured to process a respective expert input in accordance with a respective set of expert parameters of the expert neural network to generate a respective expert output, and
• a gating subsystem configured to:
• apply a softmax function to a set of gating parameters having learned values to generate a respective softmax score for each of one or more of the plurality of expert neural networks;
• determine, from at least the softmax scores, a respective weight for each of the one or more of the plurality of expert neural networks;
• select, based on the respective weights, a proper subset of the plurality of expert neural networks;
• and
• combine the respective expert outputs generated by the one or more expert neural networks in the proper subset in accordance with the respective weights for the one or more of the plurality of expert neural networks to generate one or more MoE outputs.