Systems and methods for distributing layers of special mixture-of-experts machine learning models
AssigneeMicrosoft Technology Licensing, LLC
InventorsDevangkumar Rameshbhai PATEL, Wei ZUO, Yuan YU
Life of this patent
Estimating the term…
Abstract
Some disclosed embodiments are directed to computing systems having different accelerators such that a first set of accelerators has a greater memory capability than a second set accelerators, while the second set of accelerators has a greater processing capability than the first set of accelerators. A machine learning model having different dense layers and sparse layers is distributed on the different accelerators such that the dense layers are distributed on one or more accelerators selected from the first set of accelerators and the sparse layers are distributed on one or more accelerators in the second set of accelerators.
Claims20
A computing system comprising:
- one or more hardware processors;
- one or more hardware storage devices configured to store computer-executable instructions which are executable by the one or more hardware processors for managing a distribution of sparse and dense layers of a mixture-of-experts (MOE) model on a plurality of accelerators;
- a plurality of accelerators comprising a first set of accelerators and a second set of accelerators, the accelerators in the second set of accelerators having a greater memory capacity than accelerators in the first set of accelerators, and accelerators in the first set of accelerators having a greater processing capability than accelerators in the second set of accelerators;
- and
- a machine learning model comprising a plurality of dense layers and a plurality of sparse layers, the machine learning model being distributed on the plurality of accelerators such that (i) the plurality of dense layers is distributed on one or more accelerators selected from the first set of accelerators, (ii) the plurality of sparse layers is distributed on one or more accelerators selected from the second set of accelerators, and (iii) at least one accelerator storing one or more sparse layers of the machine learning model is configured to receive multiple inputs from one or more accelerators storing dense layers of the machine learning model.
A method for managing a distribution of machine learning model comprising a mixture-of-experts (MOE) on a computing system, the method comprising:
- accessing a machine learning model comprising a plurality of dense layers and a plurality of sparse layers;
- identifying a number of dense layers and a number of experts included in the machine learning model;
- accessing a computing system comprising a plurality of accelerators, wherein at least some accelerators have a greater memory capacity than other accelerators and at least some accelerators have a greater processing capability than other accelerators;
- identifying which accelerators are available;
- determining which accelerators of the available accelerators are configured to support dense layers and which accelerators are configured to support experts;
- distributing the plurality of dense layers to one or more available accelerators configured to support dense layers;
- and
- distributing the number of experts to one or more available accelerators configured to support experts, such that the one or more accelerators configured to support experts is configured to receive multiple inputs from the one or more accelerators configured to support dense layers.
A method for determining a re-distribution of machine learning model comprising a mixture-of-experts (MOE) on a computing system, the method comprising:
- accessing a computing system comprising a plurality of accelerators, wherein at least some accelerators have a greater memory capacity than other accelerators and at least some accelerators have a greater processing capability than other accelerators;
- accessing a machine learning model comprising a plurality of dense layers and a plurality of sparse layers, each sparse layer further comprising one or more experts, the machine learning model being distributed on one or more accelerators included in the plurality of accelerators such that the plurality of dense layers of the machine learning model are distributed on one or more accelerators having the greater processing capability and the plurality of sparse layers of the machine learning model are distributed on one or more accelerators having the greater memory capability;
- executing one or more computer-executable instructions configured to cause the computing system to apply the machine learning model to input data;
- after executing the one or more computer-executable instructions, identifying a number of available accelerators included in the plurality of accelerators;
- determining a processing efficiency of the computing system according to a first distribution of the machine learning model on the one or more accelerators;
- and
- identifying a new distribution of the machine learning model on the plurality of accelerators, the new distribution configured to increase the processing efficiency of the computing system.
Dependencies are inferred from the claim text.