Implementing neural networks in fixed point arithmetic computing systems

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• receiving a request to process a neural network using a processing system that performs neural network computations using fixed point arithmetic,
• the neural network comprising a plurality of layers including an input layer,
• each layer having a plurality of nodes, and
• each node of each layer having a set of floating point weight values;
• for each node of each layer, determining a respective scaling value for the node from the respective set of floating point weight values for the node;
• receiving a set of floating point input values for each of the plurality of nodes of the input layer of the neural network;
• for each of the plurality of nodes of the input layer, converting each floating point input value of the set of floating point input values for the node to a corresponding fixed point value using the respective scaling value for the node to generate a set of fixed point input values for the node;
• and
• providing each fixed point floating point input value to the processing system for use in processing inputs using the neural network.

A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• receiving a request to process a neural network using a processing system that performs neural network computations using fixed point arithmetic,
• the neural network comprising a plurality of layers including an input layer,
• each layer having a plurality of nodes, and
• each node of each layer having a set of floating point weight values;
• for each node of each layer, determining a respective scaling value for the node from the respective set of floating point weight values for the node;
• receiving a set of floating point input values for each of the plurality of nodes of the input layer of the neural network;
• for each of the plurality of nodes of the input layer, converting each floating point input value of the set of floating point input values for the node to a corresponding fixed point value using the respective scaling value for the node to generate a set of fixed point input values for the node;
• and
• providing each fixed point floating point input value to the processing system for use in processing inputs using the neural network.