# US 2021/0081799 A1 · Compressing neural networks for natural language understanding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0081799 A1
- Document ID: US20210081799A1
- Status: Published application
- Kind: A1
- Application number: 16/938,098
- Filed: 2020-07-24
- Published: 2021-03-18
- Earliest claimed: 2019-09-12
- Assignees: Oracle International Corporation
- Applicants: Oracle International Corporation
- Inventors: Mark Edward Johnson
- Web page: https://patentgrep.com/patent/US20210081799A1

## CPC

- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06F 40/205: Parsing
- G06F 40/295: Named entity recognition
- G06F 40/30: Semantic analysis
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G06N 5/04: Inference or reasoning models
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A model for a natural language understanding task is generated based on labeled data generated by a labeling model. The model for the natural language understanding task is smaller than the labeling model (i.e., with lower computational and memory requirements than the combined model), but with substantially the same performance as the labeling model. In some cases, the labeling model may be generated based on a large pre-trained model.

## Claims

### Claim 1

A computer-implemented method comprising:
obtaining first labeled conversational data;
training a first neural network using the first labeled conversational data;
obtaining first unlabeled conversational data;
executing the trained first neural network on the first unlabeled conversational data to label the first unlabeled conversational data, thereby generating second labeled conversational data; and
training a second neural network for performing a natural language understanding task using the second labeled conversational data.

### Claim 2 (depends on 1)

The method of claim 1, further comprising, prior to training the first neural network:
obtaining second unlabeled conversational data;
training a third neural network to perform a proxy task using the second unlabeled conversational data; and
generating the first neural network based on the third neural network.

### Claim 3 (depends on 2)

The method of claim 2, wherein generating the first neural network based on the third neural network comprises constructing the first neural network to include at least a portion of the third neural network.

### Claim 4 (depends on 2)

The method of claim 2, wherein the third neural network is a component of the first neural network.

### Claim 5 (depends on 2)

The method of claim 2, wherein the proxy task comprises a predictive language task.

### Claim 6 (depends on 1)

The method of claim 1, wherein the natural language understanding task comprises one or more of semantic parsing, intent classification, or named entity classification.

### Claim 7 (depends on 1)

The method of claim 1, wherein training the second neural network comprises:
for a first training input, of the second labeled conversational data, outputting by the second neural network a predicted output;
computing a loss measuring an error between the predicted output and a first label associated with the first training input;
computing, based upon the loss, updated values for a first set of parameters of the second neural network; and
updating the second neural network by changing values of the first set of parameters to the updated values.

### Claim 8

A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising instructions that when executed by the one or more processors cause the one or more processors to perform processing comprising:
obtaining first labeled conversational data;
training a first neural network using the first labeled conversational data;
obtaining first unlabeled conversational data;
executing the trained first neural network on the first unlabeled conversational data to label the first unlabeled conversational data, thereby generating second labeled conversational data; and
training a second neural network for performing a natural language understanding task using the second labeled conversational data.

### Claim 9 (depends on 8)

The non-transitory computer-readable memory of claim 8, the processing further comprising, prior to training the first neural network:
obtaining second unlabeled conversational data;
training a third neural network to perform a proxy task using the second unlabeled conversational data; and
generating the first neural network based on the third neural network.

### Claim 10 (depends on 9)

The non-transitory computer-readable memory of claim 9, wherein generating the first neural network based on the third neural network comprises constructing the first neural network to include at least a portion of the third neural network.

### Claim 11 (depends on 9)

The non-transitory computer-readable memory of claim 9, wherein the third neural network is a component of the first neural network.

### Claim 12 (depends on 9)

The non-transitory computer-readable memory of claim 9, wherein the proxy task comprises a predictive language task.

### Claim 13 (depends on 8)

The non-transitory computer-readable memory of claim 8, wherein the natural language understanding task comprises one or more of semantic parsing, intent classification, or entity recognition.

### Claim 14 (depends on 8)

The non-transitory computer-readable memory of claim 8, wherein training the second neural network comprises:
for a first training input, of the second labeled conversational data, outputting by the second neural network a predicted output;
computing a loss measuring an error between the predicted output and a first label associated with the first training input;
computing, based upon the loss, updated values for a first set of parameters of the second neural network; and
updating the second neural network by changing values of the first set of parameters to the updated values.

### Claim 15

A system comprising:
one or more processors;
a memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions comprising instructions that when executed by the one or more processors cause the one or more processors to perform processing comprising:
obtaining first labeled conversational data;
training a first neural network using the first labeled conversational data;
obtaining first unlabeled conversational data;
executing the trained first neural network on the first unlabeled conversational data to label the first unlabeled conversational data, thereby generating second labeled conversational data; and
training a second neural network for performing a natural language understanding task using the second labeled conversational data.

### Claim 16 (depends on 15)

The system of claim 15, the processing further comprising, prior to training the first neural network:
obtaining second unlabeled conversational data;
training a third neural network to perform a proxy task using the second unlabeled conversational data; and
generating the first neural network based on the third neural network.

### Claim 17 (depends on 16)

The system of claim 16, wherein generating the first neural network based on the third neural network comprises constructing the first neural network to include at least a portion of the third neural network.

### Claim 18 (depends on 16)

The system of claim 16, wherein the third neural network is a component of the first neural network.

### Claim 19 (depends on 15)

The system of claim 15, wherein the natural language understanding task comprises one or more of semantic parsing, intent classification, or named entity classification.

### Claim 20 (depends on 15)

The system of claim 15, wherein training the second neural network comprises:
for a first training input, of the second labeled conversational data, outputting by the second neural network a predicted output;
computing a loss measuring an error between the predicted output and a first label associated with the first training input;
computing, based upon the loss, updated values for a first set of parameters of the second neural network; and
updating the second neural network by changing values of the first set of parameters to the updated values.
