Multi-layer neural network system and method

A computer-implemented method for training a neural network comprising a plurality of layers, the method comprising:

• performing a forward pass using the neural network for each training sample of one or more training samples, each training sample comprising an input, performing each forward pass comprising:
• calculating, using a first one or more layers of the plurality of layers, a first intermediate output from the input, wherein the first intermediate output is of a given size and is stored in memory;
• reducing a size of one or more dimensions of the first intermediate output to generate a reduced first intermediate output that is smaller in size than the first intermediate output;
• calculating, from the first intermediate output using a second one or more layers of the plurality of layers, a second intermediate output, the second one or more layers comprising one or more ultra-low precision layers;
• reducing a size of one or more dimensions of the second intermediate output to generate a reduced second intermediate output that is smaller in size than the second intermediate output;
• deleting, after execution of at least one of the second one or more layers, the first intermediate output from the memory;
• calculating, after reducing the size of the one or more dimensions of the first intermediate output to generate the reduced first intermediate output and from the second intermediate output using a third one or more layers of the plurality of layers, a third intermediate output, wherein the first intermediate output is deleted from the memory prior to execution of at least one of the third one or more layers;
• combining a plurality of reduced intermediate outputs to derive a combined intermediate output, wherein the plurality of reduced intermediate outputs combined comprise the reduced first intermediate output and the reduced second intermediate output;
• and
• calculating, using one or more higher-precision layers of the plurality of layers, a neural network output using the combined intermediate output;
• and
• updating the parameters of the neural network.

A system comprising one or more processors, the one or more processors configured to:

• process an input using a neural network comprising a plurality of layers, comprising:
• calculating, using a first one or more layers of the plurality of layers, a first intermediate output from the input, wherein the first intermediate output is of a given size and is stored in memory;
• reducing a size of one or more dimensions of the first intermediate output to generate a reduced first intermediate output that is smaller in size than the first intermediate output;
• calculating, from the first intermediate output using a second one or more layers of the plurality of layers, a second intermediate output, the second one or more layers comprising one or more ultra-low precision layers;
• reducing a size of one or more dimensions of the second intermediate output to generate a reduced second intermediate output that is smaller in size than the second intermediate output;
• deleting, after execution of at least one of the second one or more layers, the first intermediate output from the memory;
• calculating, after reducing the size of the one or more dimensions of the first intermediate output to generate the reduced first intermediate output and from the second intermediate output using a third one or more layers of the plurality of layers, a third intermediate output, wherein the first intermediate output is deleted from the memory prior to execution of at least one of the third one or more layers;
• combining a plurality of reduced intermediate outputs to derive a combined intermediate output, wherein the plurality of reduced intermediate outputs combined comprise the reduced first intermediate output and the reduced second intermediate output;
• and
• calculating, using one or more higher-precision layers of the plurality of layers, a neural network output using the combined intermediate output.