Mixture of experts neural networks

A method comprising:

• receiving a network input;
• and
• processing the network input using a main neural network to generate a network output for the network input, wherein the main neural network is implemented by one or more computers and comprises a Mixture of Experts (MoE) subnetwork between a first neural network layer and a second neural network layer in the main neural network, wherein the network input is a first layer output generated by the first neural network layer preceding the MOE subnetwork, wherein the MoE subnetwork comprises:
• a plurality of expert neural networks, wherein each expert neural network is configured to process a first layer output generated by the first neural network layer in accordance with a respective set of expert parameters of the expert neural network to generate a respective expert output, and
• wherein processing the network input using the main neural network comprises:
• selecting, based on the first layer output, one or more of the expert neural networks and determining a respective weight for each selected expert neural network,
• providing the first layer output as input to each of the selected expert neural networks,
• combining the expert outputs generated by the selected expert neural networks in accordance with the weights for the selected expert neural networks to generate an MoE output, and
• providing the MoE output as input to the second neural network layer.