# US 2021/0342552 A1 · Natural language text generation from a set of keywords using machine learning and templates

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342552 A1
- Document ID: US20210342552A1
- Status: Published application
- Kind: A1
- Application number: 16/864,253
- Filed: 2020-05-01
- Published: 2021-11-04
- Earliest claimed: 2020-05-01
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Abhijit Mishra; Md Faisal Mahbub Chowdhury; Sagar Manohar; Dan Gutfreund
- Web page: https://patentgrep.com/patent/US20210342552A1

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06F 40/56: Natural language generation
- G06F 40/186: Templates
- G06F 40/30: Semantic analysis
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An embodiment of the present invention generates natural language content from a set of keywords in accordance with a template. Keyword vectors representing a context for the keywords are generated. The keywords are associated with language tags, while the template includes a series of language tags indicating an arrangement for the generated natural language content. Template vectors are generated from the series of language tags of the template and represent a context for the template. Contributions from the contexts for the keywords and the template are determined based on a comparison of the series of language tags of the template with the associated language tags of the keywords. One or more words for each language tag of the template are generated to produce the natural language content based on combined contributions from the contexts for the keywords and the template.

## Claims

### Claim 1

A method of generating natural language content from a set of keywords in accordance with a template comprising:
generating, via a processor, keyword vectors representing a context for the keywords, wherein the keywords are associated with language tags and the template includes a series of language tags indicating an arrangement for the generated natural language content;
generating, via the processor, template vectors from the series of language tags of the template, wherein the template vectors represent a context for the template;
determining, via the processor, contributions from the contexts for the keywords and the template based on a comparison of the series of language tags of the template with the associated language tags of the keywords; and
generating, via the processor, one or more words for each language tag of the template to produce the natural language content based on combined contributions from the contexts for the keywords and the template.

### Claim 2 (depends on 1)

The method of claim 1, wherein the language tags of the template and the associated language tags of the keywords include part-of-speech tags.

### Claim 3 (depends on 1)

The method of claim 1, wherein determining contributions comprises:
determining a probability for each language tag of the template indicating a likelihood of that language tag of the template matching one of the associated language tags of the keywords, wherein the probability for a corresponding language tag of the template indicates the contribution for the context of the keywords for generating a word for the corresponding language tag of the template, and wherein a complement of the probability indicates the contribution for the context of the template for generating the word for the corresponding language tag of the template.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
applying the probability for the corresponding language tag of the template to a keyword vector associated with the corresponding language tag of the template to produce the contribution of the context for the keywords;
applying the complement of the probability for the corresponding language tag of the template to a template vector associated with the corresponding language tag of the template to produce the contribution of the context for the template; and
combining the contributions of the contexts for the keywords and the template to produce the combined contributions.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
determining the associated language tags for the keywords via a machine learning model, wherein the machine learning model is trained with a data set including complete sentences and the complete sentences without function words.

### Claim 6 (depends on 1)

The method of claim 1, wherein the keywords are in a first natural language, and the generated natural language content is in a second different natural language.

### Claim 7 (depends on 1)

The method of claim 1, wherein generating the keyword vectors comprises:
generating word embeddings for each of the keywords;
encoding the word embeddings using a machine learning model to produce encoded vector representations of the keywords, wherein the machine learning model is trained to produce the same encoded vector representations for a set of keywords regardless of an order of the keywords; and
generating the keyword vectors based on the encoded vector representations.

### Claim 8 (depends on 7)

The method of claim 7, wherein generating the keyword vectors based on the encoded vector representations further comprises:
applying attention weights to the encoded vector representations of the keywords to produce a keyword vector for a corresponding language tag of the template as a weighted combination of the encoded vector representations, wherein the attention weights indicate importance of individual keywords and are based on the corresponding language tag of the template.

### Claim 9 (depends on 1)

The method of claim 1, wherein generating the template vectors comprises:
generating word embeddings for each of the language tags of the template;
encoding the word embeddings for the language tags of the template using a bidirectional recurrent machine learning model; and
producing the template vectors based on the encoded word embeddings for the language tags of the template, wherein each template vector is produced based on adjacent language tags within the template.

### Claim 10 (depends on 1)

The method of claim 1, wherein generating one or more words for each language tag of the template comprises:
determining for each language tag of the template a probability distribution over a word vocabulary using a recurrent machine learning model, wherein the word vocabulary is learned from training data during training of the recurrent machine learning model; and
selecting one or more words from the word vocabulary for a corresponding language tag of the template based on the probability distribution.

### Claim 11

A system for generating natural language content from a set of keywords in accordance with a template comprising:
a processor configured to:
generate keyword vectors representing a context for the keywords, wherein the keywords are associated with language tags and the template includes a series of language tags indicating an arrangement for the generated natural language content;
generate template vectors from the series of language tags of the template, wherein the template vectors represent a context for the template;
determine contributions from the contexts for the keywords and the template based on a comparison of the series of language tags of the template with the associated language tags of the keywords; and
generate one or more words for each language tag of the template to produce the natural language content based on combined contributions from the contexts for the keywords and the template.

### Claim 12 (depends on 11)

The system of claim 11, wherein determining contributions comprises:
determining a probability for each language tag of the template indicating a likelihood of that language tag of the template matching one of the associated language tags of the keywords, wherein the probability for a corresponding language tag of the template indicates the contribution for the context of the keywords for generating a word for the corresponding language tag of the template, and wherein a complement of the probability indicates the contribution for the context of the template for generating the word for the corresponding language tag of the template.

### Claim 13 (depends on 11)

The system of claim 11, wherein the processor is further configured to:
determine the associated language tags for the keywords via a machine learning model, wherein the machine learning model is trained with a data set including complete sentences and the complete sentences without function words.

### Claim 14 (depends on 11)

The system of claim 11, wherein generating the keyword vectors comprises:
generating word embeddings for each of the keywords;
encoding the word embeddings using a machine learning model to produce encoded vector representations of the keywords, wherein the machine learning model is trained to produce the same encoded vector representations for a set of keywords regardless of an order of the keywords; and
generating the keyword vectors based on the encoded vector representations, wherein generating the keyword vectors based on the encoded vector representations further comprises:
applying attention weights to the encoded vector representations of the keywords to produce a keyword vector for a corresponding language tag of the template as a weighted combination of the encoded vector representations, wherein the attention weights indicate importance of individual keywords and are based on the corresponding language tag of the template; and
wherein generating the template vectors comprises:
generating word embeddings for each of the language tags of the template;
encoding the word embeddings for the language tags of the template using a bidirectional recurrent machine learning model; and
producing the template vectors based on the encoded word embeddings for the language tags of the template, wherein each template vector is produced based on adjacent language tags within the template.

### Claim 15 (depends on 11)

The system of claim 11, wherein generating one or more words for each language tag of the template comprises:
determining for each language tag of the template a probability distribution over a word vocabulary using a recurrent machine learning model, wherein the word vocabulary is learned from training data during training of the recurrent machine learning model; and
selecting one or more words from the word vocabulary for a corresponding language tag of the template based on the probability distribution.

### Claim 16

A computer program product for generating natural language content from a set of keywords in accordance with a template, the computer program product comprising one or more computer readable storage media having program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by a processor to cause the processor to:
generate keyword vectors representing a context for the keywords, wherein the keywords are associated with language tags and the template includes a series of language tags indicating an arrangement for the generated natural language content;
generate template vectors from the series of language tags of the template, wherein the template vectors represent a context for the template;
determine contributions from the contexts for the keywords and the template based on a comparison of the series of language tags of the template with the associated language tags of the keywords; and
generate one or more words for each language tag of the template to produce the natural language content based on combined contributions from the contexts for the keywords and the template.

### Claim 17 (depends on 16)

The computer program product of claim 16, wherein determining contributions comprises:
determining a probability for each language tag of the template indicating a likelihood of that language tag of the template matching one of the associated language tags of the keywords, wherein the probability for a corresponding language tag of the template indicates the contribution for the context of the keywords for generating a word for the corresponding language tag of the template, and wherein a complement of the probability indicates the contribution for the context of the template for generating the word for the corresponding language tag of the template.

### Claim 18 (depends on 16)

The computer program product of claim 16, wherein the program instructions further cause the processor to:
determine the associated language tags for the keywords via a machine learning model, wherein the machine learning model is trained with a data set including complete sentences and the complete sentences without function words.

### Claim 19 (depends on 16)

The computer program product of claim 16, wherein generating the keyword vectors comprises:
generating word embeddings for each of the keywords;
encoding the word embeddings using a machine learning model to produce encoded vector representations of the keywords, wherein the machine learning model is trained to produce the same encoded vector representations for a set of keywords regardless of an order of the keywords; and
generating the keyword vectors based on the encoded vector representations, wherein generating the keyword vectors based on the encoded vector representations further comprises:
applying attention weights to the encoded vector representations of the keywords to produce a keyword vector for a corresponding language tag of the template as a weighted combination of the encoded vector representations, wherein the attention weights indicate importance of individual keywords and are based on the corresponding language tag of the template; and
wherein generating the template vectors comprises:
generating word embeddings for each of the language tags of the template;
encoding the word embeddings for the language tags of the template using a bidirectional recurrent machine learning model; and
producing the template vectors based on the encoded word embeddings for the language tags of the template, wherein each template vector is produced based on adjacent language tags within the template.

### Claim 20 (depends on 16)

The computer program product of claim 16, wherein generating one or more words for each language tag of the template comprises:
determining for each language tag of the template a probability distribution over a word vocabulary using a recurrent machine learning model, wherein the word vocabulary is learned from training data during training of the recurrent machine learning model; and
selecting one or more words from the word vocabulary for a corresponding language tag of the template based on the probability distribution.
