Deep reinforced model for abstractive summarization

A neural network-based abstractive text summarization system running on parallel processors, the system comprising:

• an encoder that produces encoder hidden states for input tokens of a document to be summarized;
• an attentive decoder that emits summary tokens which summarize the document by attending over the encoder hidden states and its previously produced decoder hidden states;
• the decoder attention over the encoder hidden states encouraging input tokens not previously attended by the decoder and penalizing input tokens previously attended by the decoder;
• and
• the decoder attention over the previously produced decoder hidden states encouraging summary token candidates not previously emitted by the decoder and penalizing summary token candidates previously emitted by the decoder.

A neural network-based abstractive text summarization system, including one or more processors and memory, coupled to the processors, containing computer instructions that, when executed on the processors, cause the system to implement a process that implements an abstractive summarization model, the system comprising:

• a decoder attention over encoder hidden states encouraging input tokens not previously attended by a decoder and penalizing input tokens previously attended by the decoder;
• and
• embeddings of input document tokens of document sequences in a word embedding space;
• a bidirectional recurrent neural network (abbreviated RNN)-based encoder to produce an overall bidirectional encoding for the input document and token bidirectional encodings for the input document tokens of the document sequences using the input document token embeddings provided;
• a RNN-based decoder initialized with the overall bidirectional encoding as an initial current decoder hidden state;
• the RNN-based decoder calculating an intra-temporal encoder attention vector at each decoding step by
• applying a current decoder hidden state to the token bidirectional encodings to produce encoded state attention weights for the input document tokens;
• exponentially normalizing the encoded state attention weights;
• and
• calculating a weighted sum of the token bidirectional encodings based on the normalized encoded state attention weights;
• the RNN-based decoder calculating an intra-decoder attention vector at each decoding step by
• applying the current decoder hidden state to previously calculated decoder hidden states to produce decoded state attention weights for each of the previously calculated decoder hidden states;
• exponentially normalizing the decoded state attention weights;
• and
• calculating a weighted sum of the previously calculated decoder hidden states based on the normalized decoded state attention weights;
• and
• at each decoding step, concatenating the intra-decoder attention vector with the intra-temporal encoder attention vector and with the current decoder hidden state and processing the concatenated vector to emit a summary token.

A tangible non-transitory computer readable storage medium impressed with computer program instructions that implement an abstractive summarization model, the instructions, when executed on a processor, implement a method comprising:

• processing input token embeddings of a document through a recurrent encoder that produces encoder hidden states for each of the embeddings;
• initializing a recurrent attentive decoder with an ultimate encoder hidden state and a special start-of-summary token to produce decoder hidden states at successive decoding steps;
• and
• at each decoding step, the decoder emitting a summary token using a current intra-temporal encoder attention vector, a current intra-decoder attention vector, and a current decoder hidden state, including:
• applying the current decoder hidden state to each of the encoder hidden states to produce current encoder attention scores for each of the encoder hidden states;
• generating current encoder temporal scores for each of the encoder hidden states by exponentially normalizing a particular encoder hidden state's current encoder attention score over its previous encoder attention scores;
• generating current normalized encoder temporal scores by unity normalizing the current encoder temporal scores;
• producing the current intra-temporal encoder attention vector as convex combination of the encoder hidden states scaled by corresponding current normalized encoder temporal scores;
• applying the current decoder hidden state to each of previous decoder hidden states to produce current decoder attention scores for each of the previous decoder hidden states;
• generating current normalized decoder attention scores for each of the previous decoder hidden states by exponentially normalizing each of the current decoder attention scores;
• and
• producing the current intra-decoder attention vector as a convex combination of the previous decoder hidden states scaled by corresponding current normalized decoder attention scores and processing the vector to emit a summary token.