Limiting fine tuning data in artificial intelligence model training for artificial intelligence workloads
AssigneeDell Products L.P.
InventorDHARMESH M. PATEL
Life of this patent
Estimating the term…
Abstract
Methods, systems, and devices are provided for managing operation of a system. To manage the system, components of an artificial intelligence workflow pipeline may be evaluated for migration, replication, or tuning. If positively evaluated for tuning, then a model tuning process may be performed. Locations for the tuning process may be evaluated based on resource consumption estimates to select a tuning location and deployment location. Resource use limits for portions of the process may be enforced during location selection.
Claims20
A method for managing operation of a distributed system hosting an artificial intelligence workflow pipeline, the method comprising:
- obtaining a tuning goal for a trained machine learning model of the artificial intelligence workflow pipeline;
- identifying, based on the tuning goal, a first set of potential locations in the distributed system for performing a tuning process;
- identifying sources of portions of tuning training data;
- identifying a tuning training data resource cost limit for the tuning training data;
- performing a lossy data reduction for the portions of the tuning training data to identify refined portions of the tuning training data that meet the tuning training data resource cost limit;
- selecting, using an objective function, at least:
- a support location from the first set of potential locations, and
- a deployment location;
- performing the tuning process at the support location to obtain a tuned trained machine learning model;
- deploying the tuned trained machine learning model to the deployment location to update operation of the artificial intelligence workflow pipeline;
- and
- provisioning desired computer implemented services using the updated artificial intelligence workflow pipeline.
A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause operations for managing a distributed system hosting an artificial intelligence workflow pipeline to be performed, the operations comprising:
- obtaining a tuning goal for a trained machine learning model of the artificial intelligence workflow pipeline;
- identifying, based on the tuning goal, a first set of potential locations in the distributed system for performing a tuning process;
- identifying sources of portions of tuning training data;
- identifying a tuning training data resource cost limit for the tuning training data;
- performing a lossy data reduction for the portions of the tuning training data to identify refined portions of the tuning training data that meet the tuning training data resource cost limit;
- selecting, using an objective function, at least:
- a support location from the first set of potential locations, and
- a deployment location;
- performing the tuning process at the support location to obtain a tuned trained machine learning model;
- deploying the tuned trained machine learning model to the deployment location to update operation of the artificial intelligence workflow pipeline;
- and
- provisioning desired computer implemented services using the updated artificial intelligence workflow pipeline.
A system, comprising:
- a processor;
- and
- a memory coupled to the processor to store instructions, which when executed by the processor, cause operations for managing a distributed system hosting an artificial intelligence workflow pipeline to be performed, the operations comprising:
- obtaining a tuning goal for a trained machine learning model of the artificial intelligence workflow pipeline;
- identifying, based on the tuning goal, a first set of potential locations in the distributed system for performing a tuning process;
- identifying sources of portions of tuning training data;
- identifying a tuning training data resource cost limit for the tuning training data;
- performing a lossy data reduction for the portions of the tuning training data to identify refined portions of the tuning training data that meet the tuning training data resource cost limit;
- selecting, using an objective function, at least:
- a support location from the first set of potential locations, and
- a deployment location;
- performing the tuning process at the support location to obtain a tuned trained machine learning model;
- deploying the tuned trained machine learning model to the deployment location to update operation of the artificial intelligence workflow pipeline;
- and
- provisioning desired computer implemented services using the updated artificial intelligence workflow pipeline.
Dependencies are inferred from the claim text.