# US 2021/0342380 A1 · Generative ontology learning and natural language processing with predictive language models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342380 A1
- Document ID: US20210342380A1
- Status: Published application
- Kind: A1
- Application number: 16/862,509
- Filed: 2020-04-29
- Published: 2021-11-04
- Earliest claimed: 2020-04-29
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Francois Pierre Luus; Etienne Eben Vos; Ndivhuwo Makondo; Naweed Aghmad Khan; Ismail Yunus Akhalwaya
- Web page: https://patentgrep.com/patent/US20210342380A1

## CPC

- G06F 16/367: {Ontology}
- G06F 40/30: Semantic analysis
- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06F 40/242: Dictionaries
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/56: Natural language generation
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06N 5/025: Extracting rules from data
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An ontology topic is selected and a pretrained predictive language model is primed to create a predictive primed model based on one or more ontological rules corresponding to the selected ontology topic. Using the predictive primed model, natural language text is generated based on the ontology topic and guidance of a prediction steering component. The predictive primed model is guided in selecting text that is predicted to be appropriate for the ontology topic and the generated natural language text. The generated natural language text is processed to generate extracted ontology rules and the extracted ontology rules are compared to one or more rules of an ontology rule database that correspond to the ontology topic. A check is performed to determine if a performance of the ontology extractor is acceptable.

## Claims

### Claim 1

A method for training an ontology system, the method comprising:
selecting an ontology topic;
priming, by an ontology topic primer, a pretrained predictive language model to create a predictive primed model based on one or more ontological rules corresponding to the selected ontology topic;
generating, using the predictive primed model, natural language text based on the ontology topic and guidance of a prediction steering component;
guiding, by the prediction steering component, the predictive primed model in selecting text that is predicted to be appropriate for the ontology topic and the generated natural language text;
processing, by an ontology extractor, the generated natural language text to generate extracted ontology rules;
comparing the extracted ontology rules to one or more rules of an ontology rule database that correspond to the ontology topic; and
performing, by an ontology comparator, a check to determine if a performance of the ontology extractor is acceptable.

### Claim 2 (depends on 1)

The method of claim 1, further comprising repeating the priming, generating, guiding, processing, comparing, and performing operations, wherein the priming of the pretrained predictive language model uses results generated by the ontology comparator and a sequence of natural language tokens generated by the ontology topic primer that provides context to the pretrained predictive language model.

### Claim 3 (depends on 1)

The method of claim 1, further comprising pairing a coherent selection of ontological rules with natural language text that expressively and verbosely embeds the selection of ontological rules to generate a training dataset for training the ontology system.

### Claim 4 (depends on 1)

The method of claim 1, further comprising training the ontology extractor, the ontology topic primer, and the ontology comparator using an adversarial training algorithm implemented as an iterative stochastic gradient descent optimization with a backpropagation of errors in one or more objectives.

### Claim 5 (depends on 1)

The method of claim 1, wherein the ontology topic primer comprises a sequence-to-sequence model implemented as a recurrent neural network model having a time dimension, the method further comprising training the ontology topic primer to produce natural language text that embodies given ontological rules, taking as input a sequence of tokens from one or more ontological rules of the ontology rule database, and outputting a sequence of natural language tokens.

### Claim 6 (depends on 1)

The method of claim 1, further comprising seeding the predictive primed model using a topic selection output to prime the predictive primed model with one or more of dictionary definitions and topic concepts prompting an ontology-to-text generator comprising the pretrained predictive language model to sequentially output words and consequently generate a corpus.

### Claim 7 (depends on 1)

The method of claim 1, further comprising training the prediction steering component based on a ranked list of probabilities of a given word appearing in natural language text corresponding to the topic selection.

### Claim 8 (depends on 1)

The method of claim 1, further comprising attempting production of ontologies in order to maximize a loss of an ontology comparator.

### Claim 9 (depends on 1)

The method of claim 1, wherein the ontology extractor comprises a sequence-to-sequence model implemented as a recurrent neural network that takes as input a sequence of tokens from the generated natural language text, the method further comprising outputting a sequence of tokens that express ontological rules.

### Claim 10 (depends on 1)

The method of claim 1, the extracted ontology rules capture semantics and relations between entities in the generated natural language text.

### Claim 11 (depends on 1)

The method of claim 1, wherein the ontology comparator comprises an ontology matching algorithm using analogical similarity, the method further comprising computing a similarity score between two ontologies by mapping corresponding relations between each ontology and one or more corresponding role-filling objects.

### Claim 12 (depends on 1)

The method of claim 1, wherein the ontology comparator comprises a binary classifier implemented as a neural network that takes a set of token sequences as input, the method further comprising converting the set into a vector embedding, and assessing whether the set was one of generatively extracted and originating from the ontology rule database.

### Claim 13 (depends on 1)

The method of claim 1, further comprising attempting to correctly classify the extracted ontology rules as one of one or more rules of an ontology rule database that correspond to one or more of the ontology topic and a synthesized ontology to minimize a loss metric.

### Claim 14 (depends on 1)

The method of claim 1, wherein a result of the comparison comprises an indication of the performance of the ontology extractor and is used to train the ontology extractor and the ontology topic primer.

### Claim 15 (depends on 1)

The method of claim 1, further comprising attempting to correctly classify given natural language text as one of ontology-synthesized text and genuine natural language text to minimize a loss metric.

### Claim 16

A non-transitory computer readable medium comprising computer executable instructions which when executed by a computer cause the computer to perform the method of:
implementing an ontology topic primer, a prediction steering component, an ontology extractor, and an ontology comparator;
selecting an ontology topic;
priming, by the ontology topic primer, a pretrained predictive language model to create a predictive primed model based on one or more ontological rules corresponding to the selected ontology topic;
generating, using the predictive primed model, natural language text based on the ontology topic and guidance of the prediction steering component;
guiding, by the prediction steering component, the predictive primed model in selecting text that is predicted to be appropriate for the ontology topic and the generated natural language text;
processing, by the ontology extractor, the generated natural language text to generate extracted ontology rules;
comparing, by the ontology comparator, the extracted ontology rules to one or more rules of an ontology rule database that correspond to the ontology topic; and
performing, by the ontology comparator, a check to determine if a performance of the ontology extractor is acceptable.

### Claim 17

An apparatus comprising:
a memory; and
at least one processor, coupled to the memory, and operative to perform operations comprising:
implementing an ontology topic primer, a prediction steering component, an ontology extractor, and an ontology comparator
selecting an ontology topic;
priming, by the ontology topic primer, a pretrained predictive language model to create a predictive primed model based on one or more ontological rules corresponding to the selected ontology topic;
generating, using the predictive primed model, natural language text based on the ontology topic and guidance of a prediction steering component;
guiding, by the prediction steering component, the predictive primed model in selecting text that is predicted to be appropriate for the ontology topic and the generated natural language text;
processing, by the ontology extractor, the generated natural language text to generate extracted ontology rules;
comparing the extracted ontology rules to one or more rules of an ontology rule database that correspond to the ontology topic; and
performing, by the ontology comparator, a check to determine if a performance of the ontology extractor is acceptable.

### Claim 18 (depends on 17)

The apparatus of claim 17, the operations further comprising training the ontology extractor, the ontology topic primer, and the ontology comparator using an adversarial training algorithm implemented as an iterative stochastic gradient descent optimization with a backpropagation of errors in one or more objectives.

### Claim 19 (depends on 17)

The apparatus of claim 17, wherein the ontology topic primer comprises a sequence-to-sequence model implemented as a recurrent neural network model having a time dimension, the operations further comprising training the ontology topic primer to produce natural language text that embodies given ontological rules, taking as input a sequence of tokens from one or more ontological rules of the ontology rule database, and outputting a sequence of natural language tokens.

### Claim 20 (depends on 17)

The apparatus of claim 17, wherein the ontology comparator comprises an ontology matching algorithm using analogical similarity, the operations further comprising computing a similarity score between two ontologies by mapping corresponding relations between each ontology and one or more corresponding role-filling objects.
