Layer-based spilling
AssigneeAmazon Technologies, Inc.
InventorsParivallal Kannan, Hongbin Zheng
Life of this patent
Estimating the term…
Abstract
Techniques for reducing compilation time of a neural network model can include identifying boundary markers in a representation of the neural network model to determine proxy layers in the neural network model. A tensor that is used in a first proxy layer and is used next in a second proxy layer separated from the first proxy layer by at least a threshold distance can be identified, and a spilling operation can be inserted for the tensor to remove the tensor from a memory allocation process. A memory allocator can then be invoked to perform the memory allocation process without the tensor associated with the inserted spilling operation.
Claims20
A computer-implemented method performed by a compiler, the method comprising:
- receiving a description of a neural network model;
- generating an intermediate representation of the neural network model representing a data flow graph, the data flow graph comprising a plurality of operators;
- identifying collective compute operations to identify proxy layers in the data flow graph;
- determining long-distance tensors that are used sequentially in proxy layers separated by a threshold number of at least two proxy layers;
- inserting spilling operations for the long-distance tensors in the intermediate representation of the neural network model to generate a modified intermediate representation;
- providing the modified intermediate representation to a memory allocator to assign memory locations for tensors in the modified intermediate representation that have not been spilled;
- and
- generating machine code for the neural network model based on the modified intermediate representation and the assigned memory locations.
A computer-implemented method, comprising:
- identifying boundary markers in a representation of a neural network model to determine proxy layers in the neural network model;
- determining a tensor that is used in a first proxy layer and is used next in a second proxy layer separated from the first proxy layer by a threshold distance of at least two proxy layers;
- inserting a spilling operation for the tensor to remove the tensor from a memory allocation process;
- and
- invoking a memory allocator to perform the memory allocation process without the tensor associated with the inserted spilling operation.
A non-transitory computer readable medium having stored therein instructions that, when executed by one or more processors, cause the one or more processors to execute a compiler, the compiler performing operations including:
- identifying boundary markers to determine proxy layers in a representation of a neural network model to determine proxy layers in the neural network model;
- determining tensors that are used sequentially in proxy layers separated by a threshold distance of at least two proxy layers;
- inserting spilling operations for the tensors to remove the tensors from a memory allocation process;
- and
- invoking a memory allocator to perform the memory allocation process without the tensors associated with the inserted spilling operations.
Dependencies are inferred from the claim text.