Loading patent…
AssigneeHong Kong Applied Science and Technology Research Institute Company Limited
InventorsChao Shi, Luhong Liang, Kwok Wai Hung, King Hung Chiu
Estimating the term…
A bit-depth optimization engine reduces the hardware cost of a neural network. When training data is applied to a neural network during training routines, accuracy cost and hardware costs are generated. A hardware complexity cost generator generates costs for weights near bit-depth steps where the number of binary bits required to represent a weight decreases, such as from 2N to 2N−1, where one less binary bit is required. Gradients are generated from costs for each weight, and weights near bit-depth steps are easily selected since they have a large gradient, while weights far away from a bit-depth step have near-zero gradients. The selected weights are reduced during optimization. Over many cycles of optimization, a low-bit-depth neural network is generated that uses fewer binary bits per weight, resulting in lower hardware costs when the low-bit-depth neural network is manufactured on an Application-Specific Integrated Circuit (ASIC).
An Integrated Circuit process (IC) manufacturing process comprising:
A computer-implemented process to convert a neural network into a low-bit-depth neural network comprising:
A non-transitory computer-readable medium for storing computer-executable instructions, the computer-executable instructions when executed by a processor implementing a method comprising:
Dependencies are inferred from the claim text.
Drawings are temporarily unavailable. Open the PDF at USPTO
From the US filing history. Foreign family members aren’t listed.
Neural networks: learning methods
Architecture, e.g. interconnection topology: convolutional networks [CNN, ConvNet]
Architecture, e.g. interconnection topology: quantised networks; Sparse networks; Compressed networks
Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons: using electronic means
About 13 years 8 months of term left · ends 4 Jul 2040 (estimated)
This invention relates to Neural Networks, and more particularly to optimization of weight sizes.
Artificial Neural Networks are especially useful for processing large amounts of data in complex ways that are hard to define using traditional computer programs. Instead of being programmed with instructions, training data is input to a neural network and compared to the expected output, then adjustments are made within the neural network and the training data is again processed and outputs compared to generate further adjustments to the neural network. After many such training cycles, the neural network is altered to efficiently process data similar to the training data and expected outputs. Neural networks are an example of machine learning, since the neural network learns how to generate the expected outputs for the training data. Real data similar to the training data can then be input to the neural network to process live data.
FIG. 1 illustrates a prior art neural network. Input nodes 102, 104, 106, 108 receive input data I1, I2, I3, . . . I4, while output nodes 103, 105, 107, 109 output the result of the neural network's operations, output data O1, O2, O3, . . . O4. Three layers of operations are performed within this neural network. Nodes 110, 112, 114, 116, 118, each take inputs from one or more of input nodes 102, 104, 106, 108, perform some operation, such as addition, subtraction, multiplication, or more complex operations, and send and output to nodes in the second layer. Second-layer nodes 120, 122, 124, 126, 128, 129 also receive multiple inputs, combines these inputs to generate an output, and sends the outputs on to third-level nodes 132, 134, 136, 138, 139, which similarly combine inputs and generate outputs.
FIG. 1 illustrates a prior art neural network.
FIG. 2 shows a neural network used for facial recognition.
FIG. 3 shows a neural network implementing an image recognition processor.
The present invention relates to an improvement in neural networks. The following description is presented to enable one of ordinary skill in the art to make and use the invention as provided in the context of a particular application and its requirements. Various modifications to the preferred embodiment will be apparent to those with skill in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the particular embodiments shown and described, but is to be accorded the widest scope consistent with the principles and novel features herein disclosed.
The inventors have discovered that the size or bit depths of weights in a neural network can be reduced to reduce the complexity, cost, and size of the neural network while still optimizing accuracy. Specifically, a bit-depth complexity penalty is added during optimization to urge a neural network towards a smaller weights memory. A numerical gradient of the size of the weights is used during optimization iterations. The inventors reduce the number of bits required for weights in a neural network through a training process, which can then be deployed in dedicated hardware such as in an ASIC. The smaller optimized neural network reduces the cost and size of the ASIC.
FIG. 4 shows a neural network optimizer. Neural network 36 can be a neural network generated by design engineers for a particular application, or a generic neural network that is adjusted for a particular application. For example, the number of intermediate or hidden layers in the neural network may be adjusted for a particular application, and the type of operations performed in nodes and the connectivity among nodes may be adjusted for certain applications or problems to be solved. For example, a generic neural network might be expanded to 10 layers for the facial recognition application of FIGS. 2, or to 16 layers for the image processing application of FIG. 3.