Large-Scale Distributed Training Framework For Holistically Optimizing Training Goodput
AssigneeGoogle Inc.
InventorsGerson Cheon Kroiz, Guillermo Nicolas Grande, Christopher John Pirillo, Daniel Crankshaw, Parin B. Dalal, Allen Chenjim Wang
Life of this patent
Estimating the term…
Abstract
The present disclosure provides a customizable, large-scale distributed training framework for holistically optimizing training goodput. A holistic controller is responsible for monitoring and controlling state over an entire training cluster. This controller supports a suite of resiliency mechanisms for fault tolerance at various layers of the machine learning stack, intelligently selecting the best available fault tolerance mechanism to maximize goodput for a given machine learning workload.
Claims20
A method for optimizing training goodput, the method comprising:
- monitoring, by a supervisor node, heartbeat signals from all processing units participating in training;
- detecting, by the supervisor node, an anomaly based on the heartbeat signals;
- identifying, by the supervisor node, affected processing units affected by the anomaly;
- and
- determining, by the supervisor node, a response strategy for addressing the affected processing units at a worker level without pausing unaffected processing units.
A system for optimizing training, comprising:
- a supervisor executing a plurality of independent processes, the independent processes including a sensor process, a controller process, and an actuator process, wherein the supervisor is configured to:
- monitor heartbeat signals from all processing units participating in training;
- detect an anomaly based on the heartbeat signals;
- identify affected processing units affected by the anomaly;
- and
- determine a response strategy for addressing the affected processing units at a worker level without pausing unaffected processing units.
Dependencies are inferred from the claim text.