Matrix multiplication packing with instruction reordering
AssigneeAmazon Technologies, Inc.
InventorsJiading Gai, Tobias Joseph Kastulus Edler von Koch, Robert Geva, Paul Gilbert Meyer, Donald John Kretsch, Ron Diamant
Life of this patent
Estimating the term…
Abstract
A technique for packing matrix multiplications for concurrent execution in an integrated circuit device may include obtaining a description of a neural network model, and generating an intermediate representation of the neural network model. Matrix multiplication instructions in the intermediate representation of the neural network model can then be vectorized for concurrent execution on an integrated circuit device, and machine instructions can be generated for the integrated circuit device based on the vectorized matrix multiplication instructions.
Claims22
A computer-implemented method for compiling a neural network model, the method comprising:
- obtaining a description of the neural network model;
- generating a representation of a data dependency graph of the neural network model;
- identifying accumulation groups (AGs) having one or more matrix multiplication instructions in the data dependency graph;
- initializing an AG pack from each AG, the AG pack being a data structure representing AGs to be reordered into a consecutive sequence of matrix multiplication instructions;
- identifying AG pack pairs that are within a search window distance in the data dependency graph;
- forming a set of packing candidates from AG pack pairs that fits in a tile arrangement of a processing engine array, each packing candidate containing an AG pack pair;
- selecting packing candidates based on a similarity metric;
- combining the AG packs in each of the selected packing candidates to form new AG packs;
- reordering the data dependency graph to place the AGs in each new AG pack together for concurrent execution on an integrated circuit device;
- and
- generating machine instructions based on the reordered data dependency graph for execution on the integrated circuit device to implement the neural network model.
A computer-implemented method comprising:
- obtaining a representation of a data dependency graph of a neural network model;
- identifying accumulation groups (AGs) having one or more matrix multiplication instructions in the data dependency graph;
- initializing AG packs each including one of the AGs;
- combining the AG packs in the data dependency graph to form new AG packs;
- reordering the data dependency graph to place the AGs in each new AG pack together for concurrent execution on an integrated circuit device;
- and
- generating machine instructions based on the reordered data dependency graph for execution on the integrated circuit device.
A non-transitory computer-readable medium having stored therein instructions that, when executed by one or more processors, cause the one or more processors to execute a compiler, the compiler performing operations including:
- obtaining a representation of a data dependency graph of a neural network model;
- identifying accumulation groups (AGs) having one or more matrix multiplication instructions in the data dependency graph;
- initializing AG packs each including one of the AGs;
- combining the AG packs in the data dependency graph to form new AG packs;
- reordering the data dependency graph according to the new AG packs to place the AGs in each new AG pack together for concurrent execution on an integrated circuit device;
- and
- generating machine instructions based on the reordered data dependency graph for execution on the integrated circuit device.
Dependencies are inferred from the claim text.