Back to results
US 11,080,595 B2Granted
Quasi-recurrent neural network based encoder-decoder model
Assigneesalesforce.com, inc.
InventorsJames Bradbury, Stephen Joseph Merity, Caiming Xiong, Richard Socher
Life of this patent
Estimating the term…
Abstract
The technology disclosed provides a quasi-recurrent neural network (QRNN) encoder-decoder model that alternates convolutional layers, which apply in parallel across timesteps, and minimalist recurrent pooling layers that apply in parallel across feature dimensions.
Claims20
A quasi-recurrent neural network (QRNN) system that increases computational efficiency in neural network sequence-to-sequence modeling, the system comprising:
- a QRNN encoder that comprises one or more encoder convolutional layers and one or more one encoder pooling layers,
- at least one encoder convolutional layer including:
- a first plurality of convolutional filters placed in parallel that operate on a plurality of consecutive time series windows respectively, wherein each of the first plurality of convolutional filters receives and performs masked convolution on a respective time series of encoder input vectors within a respective time series window by excluding input vectors that are received at a future timestep and concurrently outputs an encoded convolutional vector in parallel for the respective time series window, and
- at least one encoder pooling layer receives the encoded convolutional vectors for the time series windows, concurrently accumulates an ordered set of feature sums in an encoded state vector for a current time series window, and sequentially outputs an encoded state vector for each successive time series window among the time series windows;
- a QRNN decoder that comprises one or more decoder convolutional layers and one or more one decoder pooling layers,
- at least one decoder convolutional layer including a second plurality of convolutional filters placed in parallel that operate on the consecutive time series windows respectively, each of the second plurality of convolutional filters receives a time series of decoder input vectors, performs masked convolution on the times series of decoder input by excluding decoder input vectors that are received at any future timestep and concurrently outputs decoded convolutional vectors for the time series window, and
- at least one decoder pooling layer receives the decoded convolutional vectors for the time series windows respectively concatenated with an encoded state vector outputted by an encoder pooling layer for a final time series window, concurrently accumulates an ordered set of feature sums in a decoded state vector for a current time series window, and sequentially outputs a decoded state vector for each successive time series window among the time series windows;
- a state comparator that calculates linguistic similarity between the encoded state vectors and the decoded state vectors to produce an affinity matrix with encoding-wise and decoding-wise axes;
- an exponential normalizer that normalizes the affinity matrix encoding-wise to produce respective encoding-to-decoding attention weights;
- an encoding mixer that respectively combines the encoded state vectors with the encoding-to-decoding attention weights to generate respective contextual summaries of the encoded state vectors;
- and
- an attention encoder that respectively combines the decoded state vectors with the respective contextual summaries of the encoded state vectors to produce an attention encoding for each of the time series windows.
9 more elements
A method of increasing computational efficiency in neural network sequence-to-sequence modeling, the method including:
- receiving a time series of encoder input vectors at an encoder convolutional layer of a QRNN encoder including a plurality of convolutional filters placed in parallel that operate on a plurality of consecutive time series windows respectively;
- performing masked convolution by the plurality of convolutional filters on the time series of encoder input vectors by excluding input vectors that are received at a future timestep and concurrently outputting encoded convolutional vectors for time series windows corresponding to the convolutional filters;
- receiving the encoded convolutional vectors for the time series windows at an encoder pooling layer of the QRNN encoder, concurrently accumulating an ordered set of feature sums in an encoded state vector for a current time series window, and sequentially outputting an encoded state vector for each successive time series window among the time series windows;
- receiving a time series of decoder input vectors at a decoder convolutional layer of a QRNN decoder including a second plurality of convolutional filters placed in parallel that operate on the plurality of consecutive time series windows respectively;
- performing masked convolution on the times series of decoder input by excluding decoder input vectors that are received at any future timestep and concurrently outputting decoded convolutional vectors for the plurality of consecutive time series windows;
- receiving the decoded convolutional vectors for the time series windows at a decoder pooling layer of the QRNN decoder respectively concatenated with an encoded state vector outputted by an encoder pooling layer for a final time series window, concurrently accumulating an ordered set of feature sums in an decoded state vector for a current time series window, and sequentially outputting an decoded state vector for each successive time series window among the time series windows;
- calculating linguistic similarity between the encoded state vectors and the decoded state vectors to produce an affinity matrix with encoding-wise and decoding-wise axes;
- exponentially normalizing the affinity matrix encoding-wise to produce respective encoding-to-decoding attention weights;
- combining the encoded state vectors with the encoding-to-decoding attention weights to generate respective contextual summaries of the encoded state vectors;
- and
- combining the decoded state vectors with the respective contextual summaries of the encoded state vectors to produce an attention encoding for each of the time series windows.
8 more elements
A non-transitory computer readable storage medium impressed with computer program instructions to increase computational efficiency in neural network sequence-to-sequence modeling, the instructions, when executed on a processor, implement a method comprising:
- receiving a time series of encoder input vectors at an encoder convolutional layer of a QRNN encoder including a plurality of convolutional filters placed in parallel that operate on a plurality of consecutive time series windows respectively;
- performing masked convolution by the plurality of convolutional filters on the time series of encoder input vectors by excluding input vectors that are received at a future timestep and concurrently outputting encoded convolutional vectors for time series windows corresponding to the convolutional filters;
- receiving the encoded convolutional vectors for the time series windows at an encoder pooling layer of the QRNN encoder, concurrently accumulating an ordered set of feature sums in an encoded state vector for a current time series window, and sequentially outputting an encoded state vector for each successive time series window among the time series windows;
- receiving a time series of decoder input vectors at a decoder convolutional layer of a QRNN decoder including a second plurality of convolutional filters placed in parallel that operate on the plurality of consecutive time series windows respectively;
- performing masked convolution on the times series of decoder input by excluding decoder input vectors that are received at any future timestep and concurrently outputting decoded convolutional vectors for the plurality of time series windows;
- receiving the decoded convolutional vectors for the time series windows at a decoder pooling layer of the QRNN decoder respectively concatenated with an encoded state vector outputted by an encoder pooling layer for a final time series window, concurrently accumulating an ordered set of feature sums in an decoded state vector for a current time series window, and sequentially outputting an decoded state vector for each successive time series window among the time series windows;
- calculating linguistic similarity between the encoded state vectors and the decoded state vectors to produce an affinity matrix with encoding-wise and decoding-wise axes;
- exponentially normalizing the affinity matrix encoding-wise to produce respective encoding-to-decoding attention weights;
- combining the encoded state vectors with the encoding-to-decoding attention weights to generate respective contextual summaries of the encoded state vectors;
- and
- combining the decoded state vectors with the respective contextual summaries of the encoded state vectors to produce an attention encoding for each of the time series windows.
8 more elements
Dependencies are inferred from the claim text.