Systems and methods for combining stochastic average gradient and hessian-free optimization for sequence training of deep neural networks
AssigneeInternational Business Machines Corporation
InventorsPierre Dognin, Vaibhava Goel
Life of this patent
Estimating the term…
Abstract
A method for training a deep neural network (DNN), comprises receiving and formatting speech data for the training, performing Hessian-free sequence training (HFST) on a first subset of a plurality of subsets of the speech data, and iteratively performing the HFST on successive subsets of the plurality of subsets of the speech data, wherein iteratively performing the HFST comprises reusing information from at least one previous iteration.
Claims10
A system for training a deep neural network, comprising:
- a memory and at least one processor coupled to the memory;
- an input component, executed via the at least one processor, which receives and formats speech data for the training;
- a training component, executed via the at least one processor, which performs Hessian-free sequence training on a first subset of a plurality of subsets of the speech data received from the input component, and iteratively performs the Hessian-free sequence training on successive subsets of the plurality of subsets of the speech data;
- wherein the training component reuses information from at least one previous iteration when iteratively performing the Hessian-free sequence training.
A computer program product for training a deep neural network, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method comprising:
- receiving and formatting speech data for the training;
- performing Hessian-free sequence training on a first subset of a plurality of subsets of the speech data;
- and
- iteratively performing the Hessian-free sequence training on successive subsets of the plurality of subsets of the speech data;
- wherein iteratively performing the Hessian-free sequence training comprises reusing information from at least one previous iteration.
Dependencies are inferred from the claim text.