# US 2022/0019859 A1 · System and method for configuring a classifier to achieve a target error rate

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0019859 A1
- Document ID: US20220019859A1
- Status: Published application
- Kind: A1
- Application number: 16/931,810
- Filed: 2020-07-17
- Published: 2022-01-20
- Earliest claimed: 2020-07-17
- Assignees: BlackBerry Limited
- Applicants: BlackBerry Limited
- Inventors: Edward Snow WILLIS; Steven John HENKEL
- Web page: https://patentgrep.com/patent/US20220019859A1

## CPC

- G06F 18/2178: {based on feedback of a supervisor}
- G06N 3/08: Learning methods
- G06F 1/03: working, at least partly, by table look-up (G06F1/025 takes precedence)
- G06F 18/24: Classification techniques
- G06N 3/045: Combinations of networks
- G06N 3/048: Activation functions
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/0499: Feedforward networks
- G06N 3/063: using electronic means
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer-implemented method comprises training, using a validation set of input, a first classifier to predict when a second classifier will issue a classification error on a particular input, the first classifier generating a number of data buckets based on the validation set of input and populating a threshold lookup table for each data bucket based on a number of thresholds set for the second classifier during the training; storing each threshold lookup table in memory; obtaining a target error rate; obtaining a new input and running the new input through the first classifier, the first classifier selecting one of the data buckets for the new input; and selecting a threshold for the second classifier using the stored threshold lookup table for the selected data bucket and the target error rate.

## Claims

### Claim 1

A computer-implemented method comprising:
training, using a validation set of input, a first classifier to predict when a second classifier will issue a classification error on a particular input, the first classifier generating a number of data buckets based on the validation set of input and populating a threshold lookup table for each data bucket based on a number of thresholds set for the second classifier during the training;
storing each threshold lookup table in memory;
obtaining a target error rate;
obtaining a new input and running the new input through the first classifier, the first classifier selecting one of the data buckets for the new input; and
selecting a threshold for the second classifier using the stored threshold lookup table for the selected data bucket and the target error rate.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first classifier runs each input of the validation set of the input through the second classifier at different thresholds to determine when the second classifier issues the classification error.

### Claim 3 (depends on 2)

The method of claim 2, wherein the different thresholds are incremented by a predefined value.

### Claim 4 (depends on 1)

The method of claim 1, wherein the classification error is one of a false positive and a false negative.

### Claim 5 (depends on 1)

The method of claim 1, wherein the number of data buckets is 2k data buckets, where k is a number of nodes in a chokepoint layer in the first classifier.

### Claim 6 (depends on 1)

The method of claim 1, wherein obtaining the target error rate comprises:
providing the user with a plurality of error rates for selection as the target error rate; and
receiving a user selection of the target error rate.

### Claim 7 (depends on 1)

The method of claim 1, wherein obtaining the target error rate comprises:
providing the user with a plurality of policy levels, each policy level associated with a particular error rate; and
receiving a user selection of one of the policy levels.

### Claim 8 (depends on 1)

The method of claim 1, wherein the target error rate is one of a false positive rate or a false negative rate.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
setting the threshold for the second classifier; and
running the new input through the second classifier to classify the new input.

### Claim 10

A system comprising:
at least one processor; and
a memory coupled to the at least one processor and storing instructions that, when executed by the at least one processor, configure the at least one processor to:
train, using a validation set of input, a first classifier to predict when a second classifier will issue a classification error on a particular input, the first classifier generating a number of data buckets based on the validation set of input and populating a threshold lookup table for each data bucket based on a number of thresholds set for the second classifier during the training;
store each threshold lookup table in memory;
obtain a target error rate;
obtain a new input and run the new input through the first classifier, the first classifier selecting one of the data buckets for the new input; and
select a threshold for the second classifier using the stored threshold lookup table for the selected data bucket and the target error rate.

### Claim 11 (depends on 10)

The system of claim 10, wherein the first classifier runs each input of the validation set of the input through the second classifier at different thresholds to determine when the second classifier issues the classification error.

### Claim 12 (depends on 11)

The system of claim 11, wherein the different thresholds are incremented by a predefined value.

### Claim 13 (depends on 10)

The system of claim 10, wherein the classification error is one of a false positive and a false negative.

### Claim 14 (depends on 10)

The system of claim 10, wherein the number of data buckets is 2k data buckets, where k is a number of nodes in a chokepoint layer in the first classifier.

### Claim 15 (depends on 10)

The system of claim 10, wherein obtaining the target error rate comprises:
providing the user with a plurality of error rates for selection as the target error rate; and
receiving a user selection of the target error rate.

### Claim 16 (depends on 10)

The system of claim 10, wherein obtaining the target error rate comprises:
providing the user with a plurality of policy levels, each policy level associated with a particular error rate; and
receiving a user selection of one of the policy levels.

### Claim 17 (depends on 10)

The system of claim 10, wherein the target error rate is one of a false positive rate or a false negative rate.

### Claim 18 (depends on 10)

The system of claim 10, wherein the instructions, when executed by the at least one processor, configure the at least one processor to:
set the threshold for the second classifier; and
run the new input through the second classifier to classify the new input.

### Claim 19

A non-transitory computer readable storage medium comprising computer-executable instructions which, when executed, configure a processor to:
train, using a validation set of input, a first classifier to predict when a second classifier will issue a classification error on a particular input, the first classifier generating a number of data buckets based on the validation set of input and populating a threshold lookup table for each data bucket based on a number of thresholds set for the second classifier during the training;
store each threshold lookup table in memory;
obtain a target error rate;
obtain a new input and run the new input through the first classifier, the first classifier selecting one of the data buckets for the new input; and
select a threshold for the second classifier using the stored threshold lookup table for the selected data bucket and the target error rate.

### Claim 20 (depends on 19)

The non-transitory computer readable storage medium of claim 19, wherein the target error rate is one of a false positive rate or a false negative rate.
