Fixed-point training method for deep neural networks based on static fixed-point conversion scheme

A fixed-point training method for a floating point neural network, comprising

• an initial training step, for training said floating point neural network into an intermediate neural network with a predetermined accuracy;
• a fixed-point conversion scheme determining step, for using a verification set to conduct a floating point forward calculation of said intermediate neural network, so as to determine a fixed-point conversion scheme used for a forward calculation process of said intermediate neural network, wherein said determining the fixed-point conversion scheme is based on said intermediate neural network such that the fixed-point conversion scheme reduces a number of overflow instances in network forward calculation (A) according to the fixed-point conversion scheme and (B) related to said intermediate neural network,
• wherein said fixed-point conversion scheme comprises a total bit length and a decimal bit length of a fixed-point number used for representing a corresponding floating point number, and
• wherein forward calculation for a neural network comprises calculation and storage of intermediate results and outputs for the neural network in an order from an input layer of the neural network to an output layer of the neural network;
• a fixed-point neural network obtaining step, for converting weights of said intermediate neural network from floating point numbers to fixed-point numbers, so as to obtain a corresponding fixed-point neural network;
• a fixed-point forward calculation step, for conducting a fixed-point forward calculation of said fixed-point neural network using a training dataset, on the basis of said fixed-point conversion scheme;
• and
• an intermediate neural network updating step, for updating weights of said intermediate neural network based on the calculation result of said fixed-point forward calculation,
• wherein said fixed-point conversion scheme determining step comprises:
• using a verification set to conduct a floating point forward calculation of said floating point neural network;
• determining the value range of intermediate calculation results of said floating point forward calculation;
• and
• determining said fixed-point conversion scheme based on said value range.

A fixed-point training apparatus for a floating point neural network, comprising

• a fixed-point conversion scheme determining module, for determining a fixed-point conversion scheme used for a forward calculation process of said floating point neural network, wherein said determining the fixed-point conversion scheme is based on said floating point neural network such that the fixed-point conversion scheme reduces a number of overflow instances in network forward calculation (A) according to the fixed-point conversion scheme and (B) related to said floating point neural network,
• wherein said fixed-point conversion scheme comprises a total bit length and a decimal bit length of a fixed-point number used for representing a corresponding floating point number, and
• wherein forward calculation for a neural network comprises calculation and storage of intermediate results and outputs for the neural network in an order from an input layer of the neural network to an output layer of the neural network;
• a fixed-point neural network obtaining module, for converting weights of said floating point neural network from floating point numbers to fixed-point numbers, so as to obtain a corresponding fixed-point neural network;
• a fixed-point forward calculation module, for conducting a fixed-point forward calculation of said fixed-point neural network using a training dataset, on the basis of said fixed-point conversion scheme;
• and
• a neural network updating module, for updating weights of said floating point neural network based on the calculation result of said fixed-point forward calculation,
• wherein said fixed-point conversion scheme determining module comprises:
• a value range determining unit, for using a verification set to conduct a floating point forward calculation of said floating point neural network, so as to determine the value range of intermediate calculation results of said floating point forward calculation;
• and
• a fixed-point conversion scheme determining unit, for determining said fixed-point conversion scheme based on said value range.