US 11,106,975 B2 · Fast neural network implementations by increasing parallelism of cell computations

Abstract

The amount of time required to train a neural network may be decreased by modifying the neural network to allow for greater parallelization of computations. The computations for cells of the neural network may be modified so that the matrix-vector multiplications of the cell do not depend on a previous cell and thus allowing the matrix-vector computations to be performed outside of the cells. Because the matrix-vector multiplications can be performed outside of the cells, they can be performed in parallel to decrease the computation time required for processing a sequence of training vectors with the neural network. The trained neural network may be applied to a wide variety of applications, such as performing speech recognition, determining a sentiment of text, determining a subject matter of text, answering a question in text, or translating text to another language.

Drawings

Drawings are temporarily unavailable. Open the PDF at USPTO

Drawings

Claims20

Dependent claims

A computer-implemented method for implementing a neural network, the method comprising:

obtaining a sequence of training vectors comprising a first training vector;
obtaining a first gate matrix corresponding to a first gate of the neural network;
obtaining a second gate matrix corresponding to a second gate of the neural network;
creating a combined gate matrix by combining the first gate matrix and the second gate matrix;
creating a combined training matrix by combining the sequence of training vectors;
computing precomputed data, wherein computing the precomputed data comprises multiplying the combined gate matrix and the combined training matrix and wherein the precomputed data comprises:
a first precomputed vector computed using a result of a multiplication of the first gate matrix with the first training vector, and
a second precomputed vector computed using a result of the multiplication of the second gate matrix with the first training vector;
performing a sequence of computations for a plurality of cells of the neural network, wherein computations for a first cell of the plurality of cells comprise:
obtaining a state vector from a previous cell,
obtaining the first training vector,
obtaining the first precomputed vector from the precomputed data,
obtaining the second precomputed vector from the precomputed data,
computing a first state vector for the first cell using the first precomputed vector and the state vector from the previous cell, and
computing a first output vector for the first cell using the second precomputed vector and the first state vector;
training the neural network using one or more output vectors of the sequence of computations;
obtaining a sequence of input vectors corresponding to input from a user;
processing the sequence of input vectors with the trained neural network to obtain a result;
and
providing the result to a device for presentation to the user.

17 more elements

A system for implementing a neural network, the system comprising:

at least one computer comprising at least one processor and at least one memory, the at least one computer configured to:
obtain a sequence of training vectors comprising a first training vector;
obtain a first gate matrix corresponding to a first gate of the neural network;
obtain a second gate matrix corresponding to a second gate of the neural network;
create a combined gate matrix by combining the first gate matrix and the second gate matrix;
create a combined training matrix by combining the sequence of training vectors;
compute precomputed data, wherein computing the precomputed data comprises multiplying the combined gate matrix and the combined training matrix and wherein the precomputed data comprises:
a first precomputed vector computed using a result of a multiplication of the first gate matrix with the first training vector, and
a second precomputed vector computed using a result of a multiplication of the second gate matrix with the first training vector;
perform a sequence of computations for a plurality of cells of the neural network, wherein computations for a first cell of the plurality of cells comprise:
obtaining a state vector from a previous cell,
obtaining the first training vector,
obtaining the first precomputed vector from the precomputed data,
obtaining the second precomputed vector from the precomputed data,
computing a first state vector for the first cell using the first precomputed vector and the state vector from the previous cell, and
computing a first output vector for the first cell using the second precomputed vector and the first state vector;
training the neural network using one or more output vectors of the sequence of computations;
obtaining a sequence of input vectors corresponding to input from a user;
process the sequence of input vectors with the trained neural network to obtain a result;
and
providing the result to a device for presentation to the user.

18 more elements

One or more non-transitory computer-readable media comprising computer executable instructions that, when executed, cause at least one processor to perform actions comprising:

obtaining a sequence of input vectors comprising a first input vector, wherein the sequence of input vectors correspond to input from a user;
obtaining a first gate matrix corresponding to a first gate of a neural network;
obtaining a second gate matrix corresponding to a second gate of the neural network;
creating a combined gate matrix by combining the first gate matrix and the second gate matrix;
creating a combined input matrix by combining the sequence of input vectors;
computing precomputed data wherein computing the precomputed data comprises multiplying the combined gate matrix and the combined input matrix and wherein the precomputed data comprises:
a first precomputed vector computed using a result of a multiplication of the first gate matrix with the first input vector, and
a second precomputed vector computed using a result of a multiplication of the second gate matrix with the first input vector;
performing a sequence of computations for a plurality of cells of the neural network, wherein computations for a first cell of the plurality of cells comprise:
obtaining a state vector from a previous cell,
obtaining the first input vector,
obtaining the first precomputed vector from the precomputed data,
obtaining the second precomputed vector from the precomputed data,
computing a first state vector for the first cell using the first precomputed vector and the state vector from the previous cell, and
computing a first output vector for the first cell using the second precomputed vector and the first state vector;
processing the sequence of input vectors with the neural network to obtain a result;
and
providing the result to a device for presentation to the user.

15 more elements

Dependencies are inferred from the claim text.

Description12,054 words

Summary410 words

#A portion of the disclosure of this patent document contains material, which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever.

#A Computer Program Listing Appendix is submitted herewith electronically via EFS-Web, the files of which are incorporated herein by reference. The filename, size, and date for the files submitted include: 1. DrQA_README.txt, 1628 bytes, Sep. 25, 2017; 2. DrQA_download.txt, 537 bytes, Sep. 25, 2017; 3. DrQA_drqa_layers.txt, 9071 bytes, Sep. 25, 2017; 4. DrQA_drqa_model.txt, 5679 bytes, Sep. 25, 2017; 5. DrQA_drqa_rnn_reader.txt, 7594 bytes, Sep. 25, 2017; 6. DrQA_drqa_utils.txt, 1299 bytes, Sep. 25, 2017; 7. DrQA_prepro.txt, 12796 bytes, Sep. 25, 2017; 8. DrQA_train.txt, 14267 bytes, Sep. 25, 2017; 9. README.txt, 3371 bytes, Sep. 25, 2017; 10. classification_README.txt, 1652 bytes, Sep. 25, 2017; 11. classification_dataloader.txt, 7134 bytes, Sep. 25, 2017; 12. classification_modules.txt, 2761 bytes, Sep. 25, 2017; 13. classification_train_classifier.txt, 6760 bytes, Sep. 25, 2017; 14. cuda_functional_py.txt, 19394 bytes, Sep. 25, 2017; 15. language_model_README.txt, 1085 bytes, Sep. 25, 2017; 16. language_model_train_lm.txt, 8758 bytes, Sep. 25, 2017; 17. speech_BLSTM_5L.txt, 5681 bytes, Sep. 25, 2017; 18. speech_Bi-SRU_12L.txt, 7458 bytes, Sep. 25, 2017; 19. speech_CNTK_Bi-SRU.txt, 2349 bytes, Sep. 25, 2017; 20. speech_CNTK_Bi-SRU_smbr.txt, 2547 bytes, Sep. 25, 2017; 21. speech_CNTK_SRU.txt, 2237 bytes, Sep. 25, 2017; 22. speech_CNTK_SRU_smbr.txt, 2455 bytes, Sep. 25, 2017; 23. speech_LSTM_5L.txt, 7198 bytes, Sep. 25, 2017; 24. speech_README.txt, 971 bytes, Oct. 20, 2017; 25. speech_SRU_12L.txt, 3729 bytes, Sep. 25, 2017; 26. speech_make_fbank_pitch.txt, 1251 bytes, Sep. 25, 2017; 27. speech_run_bi-sru.txt, 7374 bytes, Sep. 25, 2017; 28. speech_run_sru.txt, 7029 bytes, Sep. 25, 2017; and 29. speech_run_swbd.txt, 9591 bytes, Sep. 25, 2017.

#The present invention relates to fast implementations of artificial neural networks.

Brief description of the drawings10 figures

#The invention and the following detailed description of certain embodiments thereof may be understood by reference to the following figures:

#FIG. 1 is an example system where a neural network processes a sequence of inputs and generates a sequence of outputs.

#FIG. 2 is an example system for training a neural network.

Detailed description11,466 words

#Described herein are techniques for increasing the parallelism of artificial neural networks, which will be referred to herein as simply neural networks. A neural network is a mathematical model that is loosely based on the structure of neural networks in animal brains. Neural networks are typically structured as one or more layers of neurons, where each layer processes the outputs of neurons of a previous layer and whose outputs are processed by neurons of a subsequent layer. The processing of a layer may also include a non-linear process, such as computing a hyperbolic tangent, a sigmoid function, a softmax function, or a rectifier function. The non-linearities of a neural network allow the neural network to provide better performance than linear models.

#Some neural networks, such as recurrent neural networks may be implemented as a sequence of cells, where the computations of a subsequent cell depend on one or more outputs of a previous cell. Because the computations of a cell depend on one or more outputs of a previous cell, at least some computations of a cell cannot be performed until the computations of a previous cell have completed. Accordingly, there are limitations to parallelizing the computations of the sequence of cells. Further, because cell computations may include computationally intensive matrix-vector multiplications, the inability to parallelize the cell computations causes the recurrent neural network to require significant computing resources.

#Described herein are techniques for increasing the parallelization of cell computations of a neural network, thus increasing the computational efficiency for applications utilizing neural networks. In particular, computational efficiency may be increased by modifying the most demanding part of cell computations (matrix-vector multiplications) to allow those computationally demanding parts to be performed in parallel. The remaining parts of the cell computations that cannot be performed in parallel have lower computational demands, and thus the overall computation time may be significantly decreased. To obtain a neural network that allows for greater parallelization, the structure of the cell computations is modified. In particular, the computations for a cell are modified so that computationally intensive matrix-vector multiplications no longer depend on the previous cell. Modifying the structure of a neural network may also impact the performance of the neural network, but the experiments described below show that the modified neural network is not only more computationally efficient but also provides better performance in some applications.

Facts

Application

15/789,241Filed 20 Oct 2017

Published

US 2019/0122101 A125 Apr 2019

Granted

31 Aug 2021

Term ends

2 Jun 2040

Estimated · how

20 years from earliest known nonprovisional domestic filing plus recorded patent term adjustment, including 956 days of adjustment. Maintenance-fee lapses, terminal disclaimers and statutory extensions have not been checked. The continuity chain may be incomplete; this is not an enforceability determination.

Classification14 codes

G06N 3/08
Neural networks: learning methods
G06F 17/16
Complex mathematical operations: matrix or vector computation , e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
G06N 3/044
Architecture, e.g. interconnection topology: recurrent networks, e.g. Hopfield networks
G06N 3/0442
Recurrent networks, e.g. Hopfield networks: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]

Fast neural network implementations by increasing parallelism of cell computations

Life of this patent

Fast neural network implementations by increasing parallelism of cell computations

Life of this patent

Life of this patent

Abstract

Drawings

Drawings

Claims20

Description12,054 words

Drawings

Classification14 codes

Life of this patent

Description12,054 words