Intelligent data partitioning for distributed machine learning systems
AssigneeVMware, Inc.
InventorsYaniv Ben-Itzhak, Shay Vargaftik
Life of this patent
Estimating the term…
Abstract
Techniques for implementing intelligent data partitioning for a distributed machine learning (ML) system are provided. In one set of embodiments, a computer system implementing a data partition module can receive a training data instance for a ML task and identify, using a clustering algorithm, a cluster to which the training data instance belongs, the cluster being one of a plurality of clusters determined via the clustering algorithm that partition a data space of the ML task. The computer system can then transmit the training data instance to a ML worker of the distributed ML system that is assigned to the cluster, where the ML worker is configured to build or update a ML model using the training data instance.
Claims21
A method for implementing intelligent data partitioning in a distributed machine learning (ML) system, the method comprising:
- receiving, by a computer system implementing a data partition module, a training data instance for a ML task;
- identifying, by the computer system using a clustering algorithm, a cluster in a plurality of data clusters to which the training data instance belongs, wherein each cluster in the plurality of data clusters corresponds to a portion of an input data space of the ML task and is assigned to an ML worker in a plurality of ML workers, and wherein the identifying comprises determining that the training data instance is more strongly associated with the cluster than other clusters in the plurality of data clusters;
- and
- transmitting, by the computer system, the training data instance to a ML worker in the plurality of ML workers that is assigned to the cluster,
- wherein the ML worker is configured to build or update a ML model using the training data instance.
A non-transitory computer readable storage medium having stored thereon program code executable by a computer system implementing a data partition module for a distributed machine learning (ML) system, the program code embodying a method comprising:
- receiving a training data instance for a ML task;
- identifying, using a clustering algorithm, a cluster in a plurality of data clusters to which the training data instance belongs, wherein each cluster in the plurality of data clusters corresponds to a portion of an input data space of the ML task and is assigned to an ML worker in a plurality of ML workers, and wherein the identifying comprises determining that the training data instance is more strongly associated with the cluster than other clusters in the plurality of data clusters;
- and
- transmitting the training data instance to a ML worker in the plurality of ML workers that is assigned to the cluster,
- wherein the ML worker is configured to build or update a ML model using the training data instance.
A computer system implementing a data partition module for a distributed machine learning (ML) system, the computer system comprising:
- a processor;
- and
- a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to:
- receive a training data instance for a ML task;
- identify, using a clustering algorithm, a cluster in a plurality of data clusters to which the training data instance belongs, wherein each cluster in the plurality of data clusters corresponds to a portion of an input data space of the ML task and is assigned to an ML worker in a plurality of ML workers, and wherein the identifying comprises determining that the training data instance is more strongly associated with the cluster than other clusters in the plurality of data clusters;
- and
- transmit the training data instance to a ML worker in the plurality of ML workers that is assigned to the cluster,
- wherein the ML worker is configured to build or update a ML model using the training data instance.
Dependencies are inferred from the claim text.