Back to results
US 11,106,974 B2Granted
Pre-training of neural network by parameter decomposition
AssigneeInternational Business Machines Corporation
InventorsTakashi Fukuda, Osamu Ichikawa
Life of this patent
Estimating the term…
Loading patent…
Loading patent…
AssigneeInternational Business Machines Corporation
InventorsTakashi Fukuda, Osamu Ichikawa
Estimating the term…
A technique for training a neural network including an input layer, one or more hidden layers and an output layer, in which the trained neural network can be used to perform a task such as speech recognition. In the technique, a base of the neural network having at least a pre-trained hidden layer is prepared. A parameter set associated with one pre-trained hidden layer in the neural network is decomposed into a plurality of new parameter sets. The number of hidden layers in the neural network is increased by using the plurality of the new parameter sets. Pre-training for the neural network is performed.
A computer-implemented method for training a neural network including an input layer, one or more hidden layers, and an output layer, the method comprising:
A computer system training a neural network including an input layer, one or more hidden layers and an output layer, by executing program instructions, the computer system comprising:
A computer program product for training a neural network including an input layer, one or more hidden layers and an output layer, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
Dependencies are inferred from the claim text.
Drawings are temporarily unavailable. Open the PDF at USPTO
From the US filing history. Foreign family members aren’t listed.
Neural networks: learning methods
Architecture, e.g. interconnection topology: combinations of networks
Architecture, e.g. interconnection topology: convolutional networks [CNN, ConvNet]
Learning methods: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
About 13 years 8 months of term left · ends 27 Jun 2040 (estimated)
The present disclosure, generally, relates to machine learning techniques, and more particularly, to technique for training a neural network including an input layer, one or more hidden layers and an output layer.
It has been known that pre-training neural networks before fine-tuning can improve automatic speech recognition (ASR) performance, especially when the amount of training data is relatively small. There are several known pre-training techniques including discriminative pre-training that is based on an error back-propagation and generative pre-training that does not use discriminative information.
In conventional pre-training processes, a new layer initialized with random parameters is inserted to the top of the hidden layers just below the output layer. Then, the neural network is pre-trained using the training data.
The subject matter, which is regarded as the invention, is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The forgoing and other features and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:
FIG. 1 illustrates a block diagram of a speech recognition system that includes an acoustic training system for training a neural network based acoustic model in accordance with at least one embodiment of the present invention;
FIG. 2 depicts an architecture of a neural network used for an acoustic model, which is a target of a novel training function in accordance with at least one embodiment of the present invention;
Now, the present invention will be described using particular embodiments, and the embodiments described hereafter are understood to be only referred to as examples and are not intended to limit the scope of the present invention.
One or more embodiments according to the present invention are directed to computer-implemented methods, computer systems and computer program products for training a neural network including an input layer, one or more hidden layers and an output layer, in which the trained neural network can be used for a task such as speech recognition.
Referring to the series of FIGS. 1-6, there are shown a computer system and a method for training an acoustic model based on a neural network according to some embodiments of the present invention.