US 11,893,502 B2Granted

Dynamic hardware selection for experts in mixture-of-experts model

AssigneeAdvanced Micro Devices, Inc.

InventorsNicholas Malaya, Nuwan Jayasena

Life of this patent

Estimating the term…

Loading patent…

Abstract

A system assigns experts of a mixture-of-experts artificial intelligence model to processing devices in an automated manner. The system includes an orchestrator component that maintains priority data that stores, for each of a set of experts, and for each of a set of execution parameters, ranking information that ranks different processing devices for the particular execution parameter. In one example, for the execution parameter of execution speed, and for a first expert, the priority data indicates that a central processing unit (“CPU”) executes the first expert faster than a graphics processing unit (“GPU”). In this example, for the execution parameter of power consumption, and for the first expert, the priority data indicates that a GPU uses less power than a CPU. The priority data stores such information for one or more processing devices, one or more experts, and one or more execution characteristics.

Claims15

Dependent claims

A method for distributing an expert of a mixture-of-experts system for execution in a set of hardware devices, the method comprising:

executing experts of the mixture-of-experts system on the set of hardware devices using one or more items of parameter data associated with a plurality of execution parameters to obtain a set of priority data for different combinations of the execution parameters and the experts, the set of priority data ranking hardware devices of the set of hardware devices for executing each expert of the mixture-of-experts system for each execution parameter of the plurality of execution parameters, wherein the plurality of execution parameters includes execution speed, throughput, latency, and power consumption;
identifying one execution parameter of the plurality of execution parameters to act as a basis for selecting a hardware device, of the set of hardware devices, on which to execute the expert, the expert being a component of a mixture-of-experts machine learning model;
retrieving, from the set of priority data, priority data for a combination of the expert and the one execution parameter;
identifying a hardware device of the set of hardware devices based on the priority data, the identifying including identifying a highest ranked hardware device, for the combination of the expert and the one execution parameter, that is available for processing the expert;
and
dispatching the expert to the identified hardware device for execution.

3 more elements

A system for distributing an expert of a mixture-of-experts system for execution in a set of hardware devices, the system comprising:

a data store configured to store a set of priority data;
and
an orchestrator configured to:
execute experts of the mixture-of-experts system on the set of hardware devices using one or more items of parameter data associated with a plurality of execution parameters to obtain the set of priority data for different combinations of the execution parameters and the experts, the set of priority data ranking hardware devices of the set of hardware devices for executing each expert of the mixture-of-experts system for each execution parameter of the plurality of execution parameters, wherein the plurality of execution parameters includes of execution speed, throughput, latency, and power consumption;
identify one execution parameter of the plurality of execution parameters to act as a basis for selecting a hardware device, of the set of hardware devices, on which to execute the expert, the expert being a component of a mixture-of-experts machine learning model;
retrieve, from the set of priority data stored in the data store, priority data for a combination of the expert and the one execution parameter;
identify a hardware device of the set of hardware devices based on the priority data, the identifying including identifying a highest ranked hardware device, for the combination of the expert and the one execution parameter, that is available for processing the expert;
and
dispatch the expert to the identified hardware device for execution.

6 more elements

A system, comprising:

a set of hardware devices;
and
a system for distributing an expert of a mixture-of-experts system for execution in the set of hardware devices, the system comprising:
a data store configured to store a set of priority data;
and
an orchestrator configured to:
execute experts of the mixture-of-experts system on the set of hardware devices using one or more items of parameter data associated with a plurality of execution parameters to obtain the priority data for different combinations of the execution parameters and the experts, the set of priority data ranking hardware devices of the set of hardware devices for executing each expert of the mixture-of-experts system for each execution parameter of the plurality of execution parameters, wherein the plurality of execution parameters includes of execution speed, throughput, latency, and power consumption;
identify one execution parameter of the plurality of execution parameters to act as a basis for selecting a hardware device, of the set of hardware devices, on which to execute the expert, the expert being a component of a mixture-of-experts machine learning model;
retrieve, from the set of priority data stored in the data store, priority for a combination of the expert and the one execution parameter;
identify a hardware device of the set of hardware devices based on the priority data, the identifying including identifying a highest ranked hardware device, for the combination of the expert and the one execution parameter, that is available for processing the expert;
and
dispatch the expert to the identified hardware device for execution.

9 more elements

Dependencies are inferred from the claim text.