# US 2022/0012567 A1 · Training neural network classifiers using classification metadata from other ml classifiers

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0012567 A1
- Document ID: US20220012567A1
- Status: Published application
- Kind: A1
- Application number: 16/924,015
- Filed: 2020-07-08
- Published: 2022-01-13
- Earliest claimed: 2020-07-08
- Applicants: VMware, Inc.
- Inventors: Yaniv Ben-Itzhak; Shay Vargaftik
- Web page: https://patentgrep.com/patent/US20220012567A1

## CPC

- G06N 20/20: Ensemble learning
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/24133: {Distances to prototypes}
- G06F 18/24323: {Tree-organised classifiers}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Techniques for training a neural network classifier using classification metadata from another, non-neural network (non-NN) classifier are provided. In one set of embodiments, a computer system can train the non-NN classifier using a training data set, where the training results in a trained version of the non-NN network classifier. The computer system can further classify a data instance in the plurality of data instances using the trained non-NN classifier, the classifying generating a first class distribution for the data instance, and provide the data instance's feature set as input to a neural network classifier, the providing causing the neural network classifier to generate a second class distribution for the data instance. The computer system can then compute a loss value indicating a degree of divergence between the first and second class distributions and provide the loss value as feedback to the neural network classifier, which can cause the neural network classifier to adjust one or more internal edge weights in an manner that reduces the degree of divergence.

## Claims

### Claim 1

A method comprising:
training, by a computer system, a non-neural network classifier using a training data set, wherein the training data set comprises a plurality of data instances, wherein each data instance includes a feature set and a corresponding class label, and wherein the training results in a trained version of the non-neural network classifier;
classifying, by the computer system, a data instance in the plurality of data instances using the trained version of the non-neural network classifier, the classifying generating a first class distribution for the data instance;
providing, by the computer system, the data instance's feature set as input to a neural network classifier, the providing causing the neural network classifier to generate a second class distribution for the data instance;
computing, by the computer system, a loss value indicating a degree of divergence between the first class distribution and the second class distribution; and
providing, by the computer system, the loss value as feedback to the neural network classifier, the providing of the loss value as feedback causing the neural network classifier to adjust one or more internal edge weights in an manner that reduces the degree of divergence between the first class distribution and the second class distribution.

### Claim 2 (depends on 1)

The method of claim 1 wherein computing the loss value comprises:
calculating a loss function or a distance metric between the first class distribution and the second class distribution.

### Claim 3 (depends on 1)

The method of claim 1 wherein the classifying of the data instance using the trained version of the non-neural network classifier and the providing of the data instance's feature set as input to the neural network classifier are performed in parallel.

### Claim 4 (depends on 1)

The method of claim 1 wherein the non-neural network classifier is a random forest classifier, and wherein the steps of claim 1 result in a version of the neural network classifier that incorporates properties of the random forest classifier.

### Claim 5 (depends on 1)

The method of claim 1 further comprising:
generating a new training data instance using the trained version of the non-neural network classifier.

### Claim 6 (depends on 5)

The method of claim 5 wherein generating the new training data instance comprises:
selecting a random feature set for the new training data instance;
classifying the new training data instance via the trained version of the non-neural network classifier, the classifying of the new training data instance resulting in a predicted classification; and
setting the predicted classification as a class label for the new training data instance.

### Claim 7 (depends on 5)

The method of claim 5 further comprising:
classifying the new training data instance using the trained version of the non-neural network classifier, the classifying generating a first class distribution for the new training data instance;
providing the new training data instance's feature set as input to the neural network classifier, the providing causing the neural network classifier to generate a second class distribution for the new training data instance;
computing another loss value indicating a degree of divergence between the first class distribution for the new training data instance and the second class distribution for the new training data instance; and
providing said another loss value as feedback to the neural network classifier, the providing of said another loss value as feedback causing the neural network classifier to adjust one or more internal edge weights in an manner that reduces the degree of divergence between the first class distribution for the new training data instance and the second class distribution for the new training data instance.

### Claim 8

A non-transitory computer readable storage medium having stored thereon program code executable by a computer system, the program code causing the computer system to execute a method comprising:
training a non-neural network classifier using a training data set, wherein the training data set comprises a plurality of data instances, wherein each data instance includes a feature set and a corresponding class label, and wherein the training results in a trained version of the non-neural network classifier;
classifying a data instance in the plurality of data instances using the trained version of the non-neural network classifier, the classifying generating a first class distribution for the data instance;
providing the data instance's feature set as input to a neural network classifier, the providing causing the neural network classifier to generate a second class distribution for the data instance;
computing a loss value indicating a degree of divergence between the first class distribution and the second class distribution; and
providing the loss value as feedback to the neural network classifier, the providing of the loss value as feedback causing the neural network classifier to adjust one or more internal edge weights in an manner that reduces the degree of divergence between the first class distribution and the second class distribution.

### Claim 9 (depends on 8)

The non-transitory computer readable storage medium of claim 8 wherein computing the loss value comprises:
calculating a loss function or a distance metric between the first class distribution and the second class distribution.

### Claim 10 (depends on 8)

The non-transitory computer readable storage medium of claim 8 wherein the classifying of the data instance using the trained version of the non-neural network classifier and the providing of the data instance's feature set as input to the neural network classifier are performed in parallel.

### Claim 11 (depends on 8)

The non-transitory computer readable storage medium of claim 8 wherein the non-neural network classifier is a random forest classifier, and wherein the method of claim 8 results in a version of the neural network classifier that incorporates properties of the random forest classifier.

### Claim 12 (depends on 8)

The non-transitory computer readable storage medium of claim 8 wherein the method further comprises:
generating a new training data instance using the trained version of the non-neural network classifier.

### Claim 13 (depends on 12)

The non-transitory computer readable storage medium of claim 12 wherein generating the new training data instance comprises:
selecting a random feature set for the new training data instance;
classifying the new training data instance via the trained version of the non-neural network classifier, the classifying of the new training data instance resulting in a predicted classification; and
setting the predicted classification as a class label for the new training data instance.

### Claim 14 (depends on 12)

The non-transitory computer readable storage medium of claim 12 wherein the method further comprises:
classifying the new training data instance using the trained version of the non-neural network classifier, the classifying generating a first class distribution for the new training data instance;
providing the new training data instance's feature set as input to the neural network classifier, the providing causing the neural network classifier to generate a second class distribution for the new training data instance;
computing another loss value indicating a degree of divergence between the first class distribution for the new training data instance and the second class distribution for the new training data instance; and
providing said another loss value as feedback to the neural network classifier, the providing of said another loss value as feedback causing the neural network classifier to adjust one or more internal edge weights in an manner that reduces the degree of divergence between the first class distribution for the new training data instance and the second class distribution for the new training data instance.

### Claim 15

A computer system comprising:
a processor; and
a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to:
train a non-neural network classifier using a training data set, wherein the training data set comprises a plurality of data instances, wherein each data instance includes a feature set and a corresponding class label, and wherein the training results in a trained version of the non-neural network classifier;
classify a data instance in the plurality of data instances using the trained version of the non-neural network classifier, the classifying generating a first class distribution for the data instance;
provide the data instance's feature set as input to a neural network classifier, the providing causing the neural network classifier to generate a second class distribution for the data instance;
compute a loss value indicating a degree of divergence between the first class distribution and the second class distribution; and
provide the loss value as feedback to the neural network classifier, the providing of the loss value as feedback causing the neural network classifier to adjust one or more internal edge weights in an manner that reduces the degree of divergence between the first class distribution and the second class distribution.

### Claim 16 (depends on 15)

The computer system of claim 15 wherein the program code that causes the processor to compute the loss value comprises program code that causes the processor to:
calculate a loss function or a distance metric between the first class distribution and the second class distribution.

### Claim 17 (depends on 15)

The computer system of claim 15 wherein the classifying of the data instance using the trained version of the non-neural network classifier and the providing of the data instance's feature set as input to the neural network classifier are performed in parallel.

### Claim 18 (depends on 15)

The computer system of claim 15 wherein the non-neural network classifier is a random forest classifier, and wherein the steps performed by the processor result in a version of the neural network classifier that incorporates properties of the random forest classifier.

### Claim 19 (depends on 15)

The computer system of claim 15 wherein the program code further causes the processor to:
generate a new training data instance using the trained version of the non-neural network classifier.

### Claim 20 (depends on 19)

The computer system of claim 19 wherein the program code that causes the processor to generate the new training data instance comprises program code that causes the processor to:
select a random feature set for the new training data instance;
classify the new training data instance via the trained version of the non-neural network classifier, the classifying of the new training data instance resulting in a predicted classification; and
set the predicted classification as a class label for the new training data instance.

### Claim 21 (depends on 19)

The computer system of claim 19 wherein the program code further causes the processor to:
classify the new training data instance using the trained version of the non-neural network classifier, the classifying generating a first class distribution for the new training data instance;
provide the new training data instance's feature set as input to the neural network classifier, the providing causing the neural network classifier to generate a second class distribution for the new training data instance;
compute another loss value indicating a degree of divergence between the first class distribution for the new training data instance and the second class distribution for the new training data instance; and
provide said another loss value as feedback to the neural network classifier, the providing of said another loss value as feedback causing the neural network classifier to adjust one or more internal edge weights in an manner that reduces the degree of divergence between the first class distribution for the new training data instance and the second class distribution for the new training data instance.
