Machine Learning Inference Service Disaggregation
AssigneeGoogle Inc.
InventorsSalem Elie Haykal, Arvind Krishnamurthy, Chang Lan, Soroush Radpour
Life of this patent
Estimating the term…
Abstract
Aspects of the disclosure are directed to performing disaggregation-aware model graph partitioning, which can include provisioning and load balancing disaggregated resource pools, such as general purpose processors, accelerators, general purpose memory, and high bandwidth memory. Across these disaggregated resource pools, machine learning model operations can be packed and/or batched. The partitioning can further include automatically tuning runtime parameters.
Claims20
A method for performing machine learning model dis aggregation, the method comprising:
- profiling, with one or more processors, one or more machine learning models with a sample dataset;
- determining, with the one or more processors, a resource threshold to perform a machine learning application based on the machine learning model profiling;
- determining, with the one or more processors, a partition for the machine learning model based on the machine learning model profiling and a data transfer threshold;
- and
- partitioning, with the one or more processors, the machine learning model into a plurality of host nodes and at least one accelerator node based on the determined partition and resource threshold.
A system comprising:
- one or more processors;
- and
- one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, causes the one or more processors to perform operations for machine learning model disaggregation, the operations comprising:
- profiling one or more machine learning models with a sample dataset;
- determining a resource threshold to perform a machine learning application based on the machine learning model profiling;
- determining a partition for the machine learning model based on the machine learning model profiling and a data transfer threshold;
- and
- partitioning the machine learning model into a plurality of host nodes and at least one accelerator node based on the determined partition and resource threshold.
A non-transitory computer readable medium for storing instructions that, when executed by one or more processors, causes the one or more processors to perform operations for machine learning model disaggregation, the operations comprising:
- profiling one or more machine learning models with a sample dataset;
- determining a resource threshold to perform a machine learning application based on the machine learning model profiling;
- determining a partition for the machine learning model based on the machine learning model profiling and a data transfer threshold;
- and
- partitioning the machine learning model into a plurality of host nodes and at least one accelerator node based on the determined partition and resource threshold.
Dependencies are inferred from the claim text.