Back to results
US 11,379,140 B2Granted
System and method for model training orchestration
AssigneeGrid.ai, Inc.
InventorsLuis Capelo, Williams Falcon, Karolis Rusenas, Luca Antiga, Neven Miculinic
Life of this patent
Estimating the term…
Abstract
A system for large-scale machine learning experiment execution, including: a platform configured to determine an experiment set from a run specification and schedule a run to one or more clusters; and a set of agents configured to receive the experiment set from the platform and facilitate individual experiment execution through a cluster orchestrator.
Claims20
A system, comprising:
- a plurality of agents, each installed on a different computing cluster, wherein each computing cluster is controlled by a different cluster orchestrator;
- and
- a processing system, configured to:
- receive cluster telemetry, comprising metrics for experiments running on a cluster, from the agent for the respective cluster;
- and
- send control instructions determined based on the cluster telemetry to the agent, wherein the agent controls the respective cluster orchestrator according to the control instructions, wherein the control instructions are in a platform-standard protocol, wherein the agent controls the respective cluster orchestrator using a cluster orchestrator-specific protocol.
3 more elements
A system for multi-cluster machine learning experiment orchestration, comprising:
- a plurality of agents, each installed on a different computing cluster, wherein each agent is configured to control a cluster orchestrator of the computing cluster, and wherein each cluster orchestrator controls a plurality of nodes within the respective computing cluster;
- and
- a processing system, comprising a multitenant platform, configured to:
- determine a set of runs, each run comprising a set of machine learning experiments;
- and
- schedule runs to clusters, wherein the agent associated with the cluster controls run execution on the cluster;
- wherein the processing system concurrently controls execution of different sets of runs on different sets of clusters for different users.
4 more elements
A system, comprising:
- a plurality of agents, each installed on a different computing cluster, wherein each computing cluster is controlled by a different cluster orchestrator;
- and
- a processing system, configured to:
- receive cluster telemetry, comprising metrics for experiments running on a cluster, from the agent for the respective cluster, wherein the experiments running on the cluster are part of a shared run, wherein the processing system determines the experiments for each run and schedules runs to clusters, wherein different experiments are scheduled to different sets of nodes within the cluster;
- and
- send control instructions determined based on the cluster telemetry to the agent, wherein the agent controls the respective cluster orchestrator according to the control instructions.
3 more elements
A system, comprising:
- a plurality of agents, each installed on a different computing cluster, wherein each computing cluster is controlled by a different cluster orchestrator;
- a processing system, configured to:
- receive cluster telemetry, comprising metrics for experiments running on a cluster, from the agent for the respective cluster;
- send control instructions determined based on the cluster telemetry to the agent, wherein the agent controls the respective cluster orchestrator according to the control instructions;
- and
- reconcile experiments across different clusters.
3 more elements
Dependencies are inferred from the claim text.