Preemption in a machine learning hardware accelerator
AssigneeGoogle Inc.
InventorsTemitayo Fadelu, Ravi Narayanaswami, JiHong Min, Dongdong Li, Suyog Gupta, Jason Jong Kyu Park
Life of this patent
Estimating the term…
Abstract
The present disclosure describes a system and method for preempting a long-running process with a higher priority process in a machine learning system, such as a hardware accelerator. The machine learning hardware accelerator can be a multi-chip system including semiconductor chips that can be application-specific integrated circuits (ASIC) designed to perform machine learning operations. An ASIC is an integrated circuit (IC) that is customized for a particular use.
Claims20
A method of operating a machine learning accelerator, comprising:
- executing, by a scalar core directing a plurality of compute units of the machine learning accelerator, a first process in a first context, wherein the first process is a long-running process;
- identifying, by a job scheduler of the machine learning accelerator, that a second process is queued, wherein the second process has a higher priority than a priority of the long-running process, and upon reaching a preemption checkpoint:
- determining, by the scalar core, an amount of available resources and in response to the amount of available resources being less than a required amount for the second process:
- saving, by the scalar core, in-process values of the first process;
- switching, by the scalar core, to a second context;
- executing, by the scalar core, the second process;
- upon completion of the second process, switching, by the scalar core, to the first context;
- restoring, by the scalar core, the in-process values of the first process;
- and
- resuming, by the scalar core, execution of the first process.
A system for preempting operations in a machine learning accelerator, comprising:
- a scalar core comprising one or more processors and configured to direct a plurality of compute units of the machine learning accelerator;
- a job scheduler;
- one or more tangible, non-transitory media operably connectable to the one or more processors and storing instructions that, when executed, cause the one or more processors to perform operations comprising:
- executing, by the scalar core, a first process in a first context, wherein the first process is a long-running process;
- identifying, by the job scheduler, that a second process is queued, wherein the second process has a higher priority than a priority of the long-running process, and upon reaching a preemption checkpoint:
- determining, by the scalar core, an amount of available resources and in response to the amount of available resources being less than a required amount for the second process:
- saving, by the scalar core, in-process values of the first process;
- switching, by the scalar core, to a second context;
- executing, by the scalar core, the second process;
- upon completion of the second process, switching, by the scalar core, to the first context;
- restoring, by the scalar core, the in-process values of the first process;
- and
- resuming, by the scalar core, execution of the first process.
A non-transitory computer readable storage medium storing instructions that, when executed by at least one processor, cause at least one processor of a machine learning accelerator to perform operations comprising:
- executing, by a scalar core directing a plurality of compute units of the machine learning accelerator, a first process in a first context, wherein the first process is a long-running process;
- identifying, by a job scheduler, that a second process is queued, wherein the second process has a higher priority than a priority of the long-running process, and upon reaching a preemption checkpoint:
- determining, by the scalar core, an amount of available resources and in response to the amount of available resources being less than a required amount for the second process:
- saving, by the scalar core, in-process values of the first process;
- switching, by the scalar core, to a second context;
- executing, by the scalar core, the second process;
- upon completion of the second process, switching, by the scalar core, to the first context;
- restoring, by the scalar core, the in-process values of the first process;
- and
- resuming, by the scalar core, execution of the first process.
Dependencies are inferred from the claim text.