Stash balancing in model parallelism
AssigneeMicrosoft Technology Licensing, LLC
InventorsBharadwaj PUDIPEDDI, Juliana Patrícia Vicente FRANCO
Life of this patent
Estimating the term…
Abstract
Embodiments of the present disclosure include systems and methods for training neural networks. In one embodiment, data for an artificial intelligence model is processed in a first plurality of stages and in a second plurality of stages. The first and second pluralities of stages form a pipeline. One or more of the first plurality of stages uses at least one memory associated with a corresponding one or more of the second plurality of stages to balance memory across the pipeline.
Claims20
A computer system comprising:
- one or more processors;
- and
- a non-transitory computer readable storage medium coupled to the one or more processors and having stored thereon program code executable by the one or more processors to:
- process data for an artificial intelligence model in a first plurality of stages;
- and
- process the data in a second plurality of stages, wherein the first and second pluralities of stages form a pipeline;
- wherein one or more of the first plurality of stages use at least one memory associated with a corresponding one or more of the second plurality of stages to balance memory across the pipeline.
A method of processing an artificial intelligence model comprising:
- processing data for the artificial intelligence model in a first plurality of stages;
- and
- processing the data in a second plurality of stages, wherein the first and second pluralities of stages form a pipeline,
- wherein one or more of the first plurality of stages use at least one memory associated with a corresponding one or more of the second plurality of stages.
A non-transitory computer readable storage medium having stored thereon program code executable by a computer system, the program code causing the computer system to:
- process data for an artificial intelligence model in a first plurality of stages;
- and
- process the data in a second plurality of stages, wherein the first and second pluralities of stages form a pipeline;
- wherein one or more of the first plurality of stages use at least one memory associated with a corresponding one or more of the second plurality of stages.
Dependencies are inferred from the claim text.