Lossless tiling in convolution networks—weight gradient calculation

A data processing system, comprising:

• storage medium storing instructions associated with compile time logic and runtime logic;
• and
• one or more processors coupled to the storage medium and executing the compile time logic and runtime logic,
• wherein the compile time logic, when executed by any of the one or more processors, processes a processing graph to generate a modified processing graph comprising a plurality of forward processing nodes of a forward pass and a plurality of backward processing nodes of a backward pass of a neural network model;
• and
• wherein the runtime logic, when executed by any of the one or more processors, is configured by the compile time logic to execute the modified processing graph to:
• generate, at a backward processing node of the plurality of backward processing nodes, a plurality of partial weight gradients, based on processing a corresponding plurality of gradient tiles of a gradient tensor, such that (i) a first partial weight gradient of the plurality of partial weight gradients is generated based on processing a corresponding first gradient tile of the plurality of gradient tiles of the gradient tensor, and (ii) a second partial weight gradient of the plurality of partial weight gradients is generated based on processing a corresponding second gradient tile of the plurality of gradient tiles of the gradient tensor,
• generate, based on the plurality of partial weight gradients, a final weight gradient corresponding to the gradient tensor, and
• train weights of a forward processing node of the plurality of forward processing nodes of the forward pass, based on the final weight gradient.

A computer implemented method, comprising:

• generating a plurality of partial weight gradients associated with weights of a convolution kernel of a processing node of a forward pass of a neural network, based on processing a corresponding plurality of gradient tiles of a gradient tensor, such that (i) a first partial weight gradient of the plurality of partial weight gradients is generated based on processing a corresponding first gradient tile of the plurality of gradient tiles of the gradient tensor, and (ii) a second partial weight gradient of the plurality of partial weight gradients is generated based on processing a corresponding second gradient tile of the plurality of gradient tiles of the gradient tensor;
• generating, based on the plurality of partial weight gradients, a final weight gradient corresponding to the gradient tensor;
• and
• training weights of the convolution kernel of the processing node of the forward pass of the neural network, based on the final weight gradient.