Parameter-efficient mixture of experts neural networks
OwnerVahab Seyed Mirrokni, Kaiyuan Wang +4inventors
InventorsVahab Seyed Mirrokni, Kaiyuan Wang, MohammadHossein Bateni, Lin Chen, Gang Fu, Kyriakos Axiotis
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task using a neural network that includes a Mixture of Experts (MoE) subnetwork configured to process an input tensor to generate an output tensor. The MoE Subnetwork includes a set of multiple expert blocks, each configured to process a corresponding partition of the input tensor to generate a corresponding partition of the output tensor.
Claims20
A method for using a neural network to process a network input to generate a network output, wherein the neural network comprises a Mixture of Experts (MoE) subnetwork, the method comprising:
- receiving, as an input to the MoE subnetwork, an input tensor generated from the network input;
- processing the input tensor using the MoE subnetwork to generate, as an output of the MoE subnetwork, an output tensor, wherein the MoE subnetwork comprises a set of multiple expert blocks, the processing comprising:
- partitioning the input tensor into a set of multiple input component tensors;
- for each respective expert block in the set of multiple expert blocks, processing a corresponding one of the input component tensors using a respective input block of the respective expert block to generate a respective intermediate tensor;
- processing using a shared mixing layer that is shared between the multiple expert blocks, a combined intermediate tensor that combines the intermediate tensors generated by the input blocks of the set of expert blocks to generate a mixed tensor;
- partitioning the mixed tensor into a set of multiple component mixed tensors;
- for each respective expert block in the set of multiple expert blocks, processing a respective corresponding one of the component mixed tensors using a respective output block of the respective expert block to generate a respective output component tensor;
- and
- concatenating the output component tensors generated by the output blocks of the set of expert blocks to form the output tensor;
- and
- generating the network output based on the output tensor.
A system comprising:
- one or more computers;
- and
- one or more storage devices storing instructions that when executed by the one or more computers, cause the one or more computers to perform operations processing a network input to generate a network output, wherein the neural network comprises a Mixture of Experts (MoE) subnetwork, the operations comprising:
- receiving, as an input to the MoE subnetwork, an input tensor generated from the network input;
- processing the input tensor using the MoE subnetwork to generate, as an output of the MoE subnetwork, an output tensor, wherein the MoE subnetwork comprises a set of multiple expert blocks, the processing comprising:
- partitioning the input tensor into a set of multiple input component tensors;
- for each respective expert block in the set of multiple expert blocks, processing a corresponding one of the input component tensors using a respective input block of the respective expert block to generate a respective intermediate tensor;
- processing using a shared mixing layer that is shared between the multiple expert blocks, a combined intermediate tensor that combines the intermediate tensors generated by the input blocks of the set of expert blocks to generate a mixed tensor;
- partitioning the mixed tensor into a set of multiple component mixed tensors;
- for each respective expert block in the set of multiple expert blocks, processing a respective corresponding one of the component mixed tensors using a respective output block of the respective expert block to generate a respective output component tensor;
- and
- concatenating the output component tensors generated by the output blocks of the set of expert blocks to form the output tensor;
- and
- generating the network output based on the output tensor.
One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations processing a network input to generate a network output, wherein the neural network comprises a Mixture of Experts (MoE) subnetwork, the operations comprising:
- receiving, as an input to the MoE subnetwork, an input tensor generated from the network input;
- processing the input tensor using the MoE subnetwork to generate, as an output of the MoE subnetwork, an output tensor, wherein the MoE subnetwork comprises a set of multiple expert blocks, the processing comprising:
- partitioning the input tensor into a set of multiple input component tensors;
- for each respective expert block in the set of multiple expert blocks, processing a corresponding one of the input component tensors using a respective input block of the respective expert block to generate a respective intermediate tensor;
- processing using a shared mixing layer that is shared between the multiple expert blocks, a combined intermediate tensor that combines the intermediate tensors generated by the input blocks of the set of expert blocks to generate a mixed tensor;
- partitioning the mixed tensor into a set of multiple component mixed tensors;
- for each respective expert block in the set of multiple expert blocks, processing a respective corresponding one of the component mixed tensors using a respective output block of the respective expert block to generate a respective output component tensor;
- and
- concatenating the output component tensors generated by the output blocks of the set of expert blocks to form the output tensor;
- and
- generating the network output based on the output tensor.
Dependencies are inferred from the claim text.