Compiler for neural accelerator
AssigneeGoogle Inc.
InventorsArun Chauhan, Raksit Ashok, Dong Hyuk Woo
Life of this patent
Estimating the term…
Abstract
A compiler of a computing device is described that identifies a sequence of neural network models frequently invoked by an application of the computing device, compiles the models in that sequence, and loads a static random access memory (SRAM) of a hardware accelerator with the compiled models only when the same compiled models—from another, but same, sequence that was previously invoked—are not already present in the SRAM. This prevents unnecessary reloading of compiled models into the SRAM, thereby increasing runtime speed and conserving computational energy.
Claims20
A computer-implemented method comprising:
- identifying a first set of neural network models that have been executed on a hardware accelerator of a computing device more than a threshold number of times in a preset amount of time in the past;
- identifying a sequence in which the first set of models were executed on the hardware accelerator;
- compiling each neural network model of the first set of neural network models in the sequence for execution by the hardware accelerator and assigning a first hash to the identified sequence of models;
- outputting, for each neural network model of the first set of neural network models in the sequence, the compiled model to the hardware accelerator for storage in one or more memories of the hardware accelerator,
- compiling a second sequence of neural network models and computing a second hash for the second sequence of neural network models;
- determining that the first hash matches the second hash;
- and
- in response, bypassing loading compiled models in the second sequence of neural network models into storage of the hardware accelerator.
A system comprising:
- a compiler configured to:
- identify a first set of neural network models that have been executed on a hardware accelerator of a computing device more than a threshold number of times in a preset amount of time in the past;
- identify a sequence in which the first set of models are executed on the hardware accelerator;
- compile each neural network model of the first set of neural network models in the sequence for execution by the hardware accelerator and assigning a first hash to the identified sequence of models;
- output, for each neural network model of the first set of neural network models in the sequence, the compiled model to the hardware accelerator for storage in one or more memories of the hardware accelerator;
- compiling a second sequence of neural network models and computing a second hash for the second sequence of neural network models;
- determining that the first hash matches the second hash;
- and
- in response, bypassing loading compiled models in the second sequence of neural network models into storage of the hardware accelerator;
- the hardware accelerator comprising the one or more memories to store the compiled model for each neural network model of the first set of neural network models according to the sequence, the storage according to the sequence of the compiled model for each neural network model of the first set of neural network models in the one or more memories preventing a need for reloading a compiled result recompilation of the sequence of the first set of neural network models into the one or more memories when the first set of neural network models are to be executed again on the hardware accelerator.
Dependencies are inferred from the claim text.