# US 2021/0174784 A1 · Contextual text generation for question answering and text summarization with supervised representation disentanglement and mutual information minimization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174784 A1
- Document ID: US20210174784A1
- Status: Published application
- Kind: A1
- Application number: 17/114,946
- Filed: 2020-12-08
- Published: 2021-06-10
- Earliest claimed: 2019-12-09
- Assignees: NEC Laboratories America, Inc.
- Applicants: NEC Laboratories America, Inc.
- Inventors: Renqiang Min; Christopher Malon; Hans Peter Graf
- Web page: https://patentgrep.com/patent/US20210174784A1

## CPC

- G06F 40/56: Natural language generation
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06F 40/279: Recognition of textual entities
- G06F 40/30: Semantic analysis
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/086: using evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G10L 15/063: {Training}
- G10L 15/16: using artificial neural networks
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06N 3/02: Neural networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Methods and systems for disentangled data generation include accessing a dataset including pairs, each formed from a given input text structure and a given style label for the input text structures. An encoder is trained to disentangle a sequential text input into disentangled representations, including a content embedding and a style embedding, based on a subset of the dataset, using an objective function that includes a regularization term that minimizes mutual information between the content embedding and the style embedding. A generator is trained to generate a text output that includes content from the style embedding, expressed in a style other than that represented by the style embedding of the text input.

## Claims

### Claim 1

A computer-implemented method for disentangled data generation, comprising:
accessing a dataset including a plurality of pairs, each formed from a given one of a plurality of input text structures and a given one of a plurality of style labels for the plurality of input text structures;
training an encoder to disentangle a sequential text input into disentangled representations, including a content embedding and a style embedding, based on a subset of the dataset, using an objective function that includes a regularization term that minimizes mutual information between the content embedding and the style embedding; and
training a generator to generate a text output that includes content from the style embedding, expressed in a style other than that represented by the style embedding of the text input.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the encoder includes a multi-head attention neural network unit for performing disentanglement.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising updating model parameters using the subset of the dataset as mini-batches by maximizing a log-likelihood of an input text in a variational autoencoder framework.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the generator includes a two-layer unidirectional LSTM neural network having an output dimension equal to a vocabulary size of a vocabulary from which the next word prediction is generated.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein in a subsequent testing sequence, the method further comprises:
fixing one of the disentangled representations while randomly sampling another one of the disentangled representations; and
decoding the fixed one of the disentangled representations with the randomly sampled other one of the disentangled representations.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the input text structures are selected from the group consisting of sentences and paragraphs.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the objective function is:
VAE+λreg
wherein λ is a hyperparameter reweighting a regularization reg and a variational autoencoder objective VAE, where reg is expressed as Dis+MI(s; c), including a disentanglement loss Dis and a mutual information term MI(s; c) based on a style embedding s and a content embedding c.

### Claim 7 (depends on 11)

The system of claim 11, wherein the objective function is:
VAE+λreg
wherein λ is a hyperparameter reweighting a regularization reg and a variational autoencoder objective VAE, where reg is expressed as Dis+MI(s; c), including a disentanglement loss Dis and a mutual information term MI(s; c) based on a style embedding s and a content embedding c.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising disentangling a new text input, using the trained encoder, into a new content embedding and a new style embedding.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, further comprising generating a new text output using the generator, based on the new content embedding, to perform at least one of style transfer, conditional text generation for question answering or text summarization, or personalized dialogue systems.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein generating the new text output is further based on a style embedding that is different from the new style embedding.

### Claim 11

A system for disentangled data generation, comprising:
a hardware processor; and
a memory that stores:
a dataset including a plurality of pairs, each formed from a given one of a plurality of input text structures and a given one of a plurality of style labels for the plurality of input text structures; and
computer program code which, when executed by the hardware processor, implements:
training code that trains an encoder to disentangle a sequential text input into disentangled representations, including a content embedding and a style embedding, based on a subset of the dataset, using an objective function that includes a regularization term that minimizes mutual information between the content embedding and the style embedding, and that trains a generator to generate a text output that includes content from the style embedding, expressed in a style other than that represented by the style embedding of the text input.

### Claim 12 (depends on 11)

The system of claim 11, wherein the encoder includes a multi-head attention neural network unit for performing disentanglement.

### Claim 13 (depends on 11)

The system of claim 11, wherein the training code further updates model parameters using the subset of the dataset as mini-batches by maximizing a log-likelihood of an input text in a variational autoencoder framework.

### Claim 14 (depends on 11)

The system of claim 11, wherein the generator includes a two-layer unidirectional LSTM neural network having an output dimension equal to a vocabulary size of a vocabulary from which the next word prediction is generated.

### Claim 15 (depends on 11)

The system of claim 11, wherein the training code further:
fixes one of the disentangled representations while randomly sampling another one of the disentangled representations; and
decodes the fixed one of the disentangled representations with the randomly sampled other one of the disentangled representations.

### Claim 16 (depends on 11)

The system of claim 11, wherein the input text structures are selected from the group consisting of sentences and paragraphs.

### Claim 18 (depends on 11)

The system of claim 11, wherein the computer program code further implements disentangling code that disentangles a new text input, using the trained encoder, into a new content embedding and a new style embedding.

### Claim 19 (depends on 18)

The system of claim 18, wherein the computer program code further implements generating code that generates a new text output using the generator, based on the new content embedding, to perform at least one of style transfer, conditional text generation for question answering or text summarization, or personalized dialogue systems.

### Claim 20 (depends on 19)

The system of claim 19, wherein the new text output is further based on a style embedding that is different from the new style embedding.