# US 11,308,392 B2 · Fixed-point training method for deep neural networks based on static fixed-point conversion scheme > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,392 B2 - Document ID: US11308392B2 - Status: Granted - Kind: B2 - Application number: 15/693,491 - Filed: 2017-09-01 - Published: 2019-01-31 - Granted: 2022-04-19 - Earliest claimed: 2017-07-28 - Assignees: Xilinx Technology Beijing Limited - Applicants: Xilinx Technology Beijing Limited - Inventors: Xin Li; Tong Meng; Song Han - Web page: https://patentgrep.com/patent/US11308392B2 ## CPC - G06N 3/08: Learning methods - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/063: using electronic means - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 3/09: Supervised learning - G06F 7/485: Adding; Subtracting {(G06F7/4833, G06F7/4836 take precedence)} - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The present disclosure proposes a fixed-point training method and apparatus based on static fixed-point conversion scheme. More specifically, the present disclosure proposes a fixed-point training method for LSTM neural network. According to this method, during the fine-tuning process of the neural network, it uses fixed-point numbers to conduct forward calculation. Accordingly, within several training cycles, the network accuracy may returned to the desired accuracy level under floating point calculation. ## Claims ### Claim 1 A fixed-point training method for a floating point neural network, comprising a fixed-point conversion scheme determining step, for determining a fixed-point conversion scheme used for a forward calculation process of said floating point neural network, wherein said determining the fixed-point conversion scheme is based on said floating point neural network such that the fixed-point conversion scheme reduces a number of overflow instances in network forward calculation (A) according to the fixed-point conversion scheme and (B) related to said floating point neural network, wherein said fixed-point conversion scheme comprises a total bit length and a decimal bit length of a fixed-point number used for representing a corresponding floating point number, and wherein forward calculation for a neural network comprises calculation and storage of intermediate results and outputs for the neural network in an order from an input layer of the neural network to an output layer of the neural network; a fixed-point neural network obtaining step, for converting weights of said floating point neural network from floating point numbers to fixed-point numbers, so as to obtain a corresponding fixed-point neural network; a fixed-point forward calculation step, for conducting a fixed-point forward calculation of said fixed-point neural network using a training dataset according to said fixed-point conversion scheme; and a neural network updating step, for updating weights of said floating point neural network based on the calculation result of said fixed-point forward calculation, wherein said fixed-point conversion scheme determining step comprises: using a verification set to conduct a floating point forward calculation of said floating point neural network; determining the value range of the intermediate calculation results of said floating point forward calculation; and determining said fixed-point conversion scheme based on said value range. ### Claim 2 (depends on 1) The method according to claim 1, further comprises: executing iteratively said fixed-point conversion scheme determining step, said fixed-point neural network obtaining step, said fixed-point forward calculation step and said neural network updating step, until the floating point neural network achieves a desired accuracy. ### Claim 3 (depends on 1) The method according to claim 1, wherein said fixed-point neural network obtaining step further comprises: converting weights of said floating point neural network to fixed-point numbers by a rule of fixed-point conversion of floating point number. ### Claim 4 (depends on 1) The method according to claim 1, wherein said fixed-point forward calculation step further comprises: during said fixed-point forward calculation, determining the total bit number and the decimal bit length of said training dataset as well as the total bit number and the decimal bit length of intermediate calculation results of the fixed-point forward calculation on the basis of said fixed-point conversion scheme. ### Claim 5 (depends on 1) The method according to claim 1, wherein said neural network updating step further comprises: comparing the calculation result of said fixed-point forward calculation with a standard value to obtain an accuracy loss; and calculating a floating point gradient based on said accuracy loss, and updating said floating point neural network based on said floating point gradient. ### Claim 6 (depends on 1) The method according to claim 1, wherein said floating point neural network is an LSTM neural network, and said training dataset is a voice dataset. ### Claim 7 A fixed-point training method for a floating point neural network, comprising an initial training step, for training said floating point neural network into an intermediate neural network with a predetermined accuracy; a fixed-point conversion scheme determining step, for using a verification set to conduct a floating point forward calculation of said intermediate neural network, so as to determine a fixed-point conversion scheme used for a forward calculation process of said intermediate neural network, wherein said determining the fixed-point conversion scheme is based on said intermediate neural network such that the fixed-point conversion scheme reduces a number of overflow instances in network forward calculation (A) according to the fixed-point conversion scheme and (B) related to said intermediate neural network, wherein said fixed-point conversion scheme comprises a total bit length and a decimal bit length of a fixed-point number used for representing a corresponding floating point number, and wherein forward calculation for a neural network comprises calculation and storage of intermediate results and outputs for the neural network in an order from an input layer of the neural network to an output layer of the neural network; a fixed-point neural network obtaining step, for converting weights of said intermediate neural network from floating point numbers to fixed-point numbers, so as to obtain a corresponding fixed-point neural network; a fixed-point forward calculation step, for conducting a fixed-point forward calculation of said fixed-point neural network using a training dataset, on the basis of said fixed-point conversion scheme; and an intermediate neural network updating step, for updating weights of said intermediate neural network based on the calculation result of said fixed-point forward calculation, wherein said fixed-point conversion scheme determining step comprises: using a verification set to conduct a floating point forward calculation of said floating point neural network; determining the value range of intermediate calculation results of said floating point forward calculation; and determining said fixed-point conversion scheme based on said value range. ### Claim 8 (depends on 7) The method according to claim 7, further comprises: executing iteratively said fixed-point conversion scheme determining step, said fixed-point neural network obtaining step, said fixed-point forward calculation step, and said intermediate neural network updating step, until said intermediate neural network reaches a desired accuracy. ### Claim 9 (depends on 7) The method according to claim 7, wherein said fixed-point conversion scheme includes: the total bit length and the decimal bit length of a fixed-point number used for representing a corresponding floating point number. ### Claim 10 (depends on 7) The method according to claim 7, wherein said fixed-point neural network obtaining step further comprises: converting weights of said floating point neural network to fixed-point numbers by a rule of fixed-point conversion of floating point number. ### Claim 11 (depends on 7) The method according to claim 7, wherein said fixed-point forward calculation step further comprises: during said fixed-point forward calculation, determining the total bit number and the decimal bit length of said training dataset as well as the total bit number and the decimal bit length of intermediate calculation results of said fixed-point forward calculation, on the basis of said fixed-point conversion scheme. ### Claim 12 (depends on 7) The method according to claim 7, wherein said intermediate neural network updating step further comprises: comparing the calculation result of said fixed-point forward calculation with a standard value to obtain an accuracy loss; and calculating a floating point gradient based on said accuracy loss, and updating said intermediate neural network based on said floating point gradient. ### Claim 13 A fixed-point training apparatus for a floating point neural network, comprising a fixed-point conversion scheme determining module, for determining a fixed-point conversion scheme used for a forward calculation process of said floating point neural network, wherein said determining the fixed-point conversion scheme is based on said floating point neural network such that the fixed-point conversion scheme reduces a number of overflow instances in network forward calculation (A) according to the fixed-point conversion scheme and (B) related to said floating point neural network, wherein said fixed-point conversion scheme comprises a total bit length and a decimal bit length of a fixed-point number used for representing a corresponding floating point number, and wherein forward calculation for a neural network comprises calculation and storage of intermediate results and outputs for the neural network in an order from an input layer of the neural network to an output layer of the neural network; a fixed-point neural network obtaining module, for converting weights of said floating point neural network from floating point numbers to fixed-point numbers, so as to obtain a corresponding fixed-point neural network; a fixed-point forward calculation module, for conducting a fixed-point forward calculation of said fixed-point neural network using a training dataset, on the basis of said fixed-point conversion scheme; and a neural network updating module, for updating weights of said floating point neural network based on the calculation result of said fixed-point forward calculation, wherein said fixed-point conversion scheme determining module comprises: a value range determining unit, for using a verification set to conduct a floating point forward calculation of said floating point neural network, so as to determine the value range of intermediate calculation results of said floating point forward calculation; and a fixed-point conversion scheme determining unit, for determining said fixed-point conversion scheme based on said value range. ### Claim 14 (depends on 13) The apparatus according to claim 13, wherein said fixed-point conversion scheme includes: the total bit length and the decimal bit length of a fixed-point number used for representing a corresponding floating point number. ### Claim 15 (depends on 13) The apparatus according to claim 13, wherein said fixed-point neural network obtaining module is configured to convert weights of said floating point neural network to fixed-point numbers by a rule of fixed-point conversion of floating point number. ### Claim 16 (depends on 13) The apparatus according to claim 13, wherein said fixed-point forward calculation module is configured to determine, during said fixed-point forward calculation, the total bit number and the decimal bit length of said training dataset as well as the total bit number and the decimal bit length of intermediate calculation results of said fixed-point forward calculation, on the basis of said fixed-point conversion scheme. ### Claim 17 (depends on 13) The apparatus according to claim 13, wherein said neural network updating module further comprises: a comparing unit, for comparing the calculation result of said fixed-point forward calculation with a standard value to obtain an accuracy loss; and an updating unit, for calculating a floating point gradient based on said accuracy loss, and updating weights of said floating point neural network based on said floating point gradient.