# US 2025/0061287 A1 · Enhanced machine learning model accuracy through post-hoc confidence score calibration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0061287 A1
- Document ID: US20250061287A1
- Status: Published application
- Kind: A1
- Application number: 18/677,561
- Filed: 2024-05-29
- Published: 2025-02-20
- Earliest claimed: 2023-08-16
- Assignees: Snowflake Inc.
- Applicants: Snowflake Inc.
- Inventors: Andrzej Szwabe
- Web page: https://patentgrep.com/patent/US20250061287A1

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Examples provide enhanced machine learning model accuracy through post-hoc confidence score calibration. A machine learning (ML) system receives results generated by an ML model, the results comprising at least one confidence score and electronic documents. The ML system processes the results generated by the ML model comprising performing document understanding by extracting data points from the electronic documents. The ML system associates the confidence score with the extracted data points and calibrates a confidence score associated with the extracted data points using a post-hoc calibration solution set. The ML system implements confidence scoring recalibration comprising aligning the confidence score with prediction accuracy and adjusting the generated confidence score by the recalibration. Based on adjusting the confidence score, the ML system extracts an individual element of information from the electronic documents comprising an extracted value. The ML system generates an output comprising the extracted values and an adjusted confidence score.

## Claims

### Claim 1

A method comprising:
receiving, by at least one hardware processor, results generated by a machine learning (ML) model, the results comprising at least one confidence score and a plurality of electronic documents;
processing the results generated by the ML model, the processing comprising performing document understanding by extracting data points from the plurality of electronic documents;
associating the at least one confidence score with the extracted data points;
calibrating the at least one confidence score associated with the extracted data points using a post-hoc calibration (PHC) solution set;
implementing confidence scoring recalibration, the confidence scoring recalibration comprising functionality to assess reliability of the results generated by the ML model, the confidence scoring recalibration comprising aligning the at least one confidence score with prediction accuracy;
adjusting the at least one confidence score generated by the confidence scoring recalibration;
based on the adjusting of the at least one confidence score, extracting an individual element of information from the plurality of electronic documents, the extracting of the individual element of the information comprising one or more extracted values; and
generating an output comprising the one or more extracted values in a database comprising an adjusted confidence score associated with each of the one or more extracted values and the results generated by a ML model.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
utilizing selective prediction accuracy to improve the performance of the ML model, the selective prediction accuracy comprising the functionality to predict an accuracy of the results generated by the ML model; and
improving both calibration quality and the selective prediction accuracy of the ML model performing the processing of the results and the document understanding.

### Claim 3 (depends on 1)

The method of claim 1, wherein the PHC solution set comprises at least one of temperature scaling, softmax denominator top-k probability selection, and polynomial regression.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
training the ML model comprising the temperature scaling on a validation set, wherein the validation set is a subset of a dataset used during the training of the ML model.

### Claim 5 (depends on 1)

The method of claim 1, wherein the ML model further comprises:
using a pre-trained large language model;
generating one or more outputs from the pre-trained large language model; and
enabling the pre-trained large language model to make predictions on data categories not used for pre-training.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
processing a first set of electronic documents, using the pre-trained large language model, to extract a set of values for a set of data points for each of the first set of electronic documents; and
recalibrating the at least one confidence score based on polynomial regression.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
processing, by the ML model, a second set of electronic documents, comprising extracting one or more data points for each of the first set of electronic documents;
providing a new confidence score solution comprising improving both calibration quality and selective prediction accuracy of the second set of electronic documents based on a combination of temperature scaling, top-k softmax denomination and recalibrating of the at least one confidence score of the first set of electronic documents; and
indicating, by the ML model, confidence scores for one or more data point values extracted from the second set of electronic documents, when processed by the ML model for extraction.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
a sequence of per-token logits comprising a series of numerical values that represent the predictions for each token in a sequence of electronic documents based on the ML model.

### Claim 9 (depends on 8)

The method of claim 8, wherein logits include raw, unnormalized output of a neural network prior to application of a softmax function, and comprising:
using the softmax function to convert the logits into probabilities, the logits providing a preliminary measurement of the at least one confidence score for the ML model.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
identifying a quality of confidence score;
recalibrating the confidence score based on the quality of the confidence score; and
applying the combination of the temperature scaling, softmax denominator top-k probability selection, and the polynomial regression to the confidence score.

### Claim 11

A system comprising:
one or more hardware processors of a machine; and
at least one memory storing instructions that, when executed by the one or more hardware processors, cause the system to perform operations comprising:
receiving results generated by a machine learning (ML) model, the results comprising at least one confidence score and a plurality of electronic documents;
processing the results generated by the ML model, the processing comprising performing document understanding by extracting data points from the plurality of electronic documents;
associating the at least one confidence score with the extracted data points;
calibrating the at least one confidence score associated with the extracted data points using a post-hoc calibration (PHC) solution set;
implementing confidence scoring recalibration, the confidence scoring recalibration comprising functionality to assess reliability of the results generated by the ML model, the confidence scoring recalibration comprising aligning the at least one confidence score with prediction accuracy;
adjusting the at least one confidence score generated by the confidence scoring recalibration;
based on the adjusting of the at least one confidence score, extracting an individual element of information from the plurality of electronic documents, the extracting of the individual element of the information comprising one or more extracted values; and
generating an output comprising the one or more extracted values in a database comprising an adjusted confidence score associated with each of the one or more extracted values and the results generated by a ML model.

### Claim 12 (depends on 11)

The system of claim 11, the operations further comprising:
utilizing selective prediction accuracy to improve performance of the ML model, the selective prediction accuracy comprising the functionality to predict an accuracy of the results generated by the ML model; and
improving both calibration quality and the selective prediction accuracy of the ML model performing the processing of the results and the document understanding.

### Claim 13 (depends on 11)

The system of claim 11, wherein the PHC solution set comprises at least one of temperature scaling, softmax denominator top-k probability selection, and polynomial regression.

### Claim 14 (depends on 13)

The system of claim 13, the operations further comprising:
training the ML model comprising the temperature scaling on a validation set, wherein the validation set is a subset of a dataset used during the training of the ML model.

### Claim 15 (depends on 11)

The system of claim 11, wherein the ML model further comprises:
using a pre-trained large language model;
generating one or more outputs from the pre-trained large language model; and
enabling the pre-trained large language model to make predictions on data categories not used for pre-training.

### Claim 16 (depends on 15)

The system of claim 15, the operations further comprising:
processing a first set of electronic documents, using the pre-trained large language model, to extract a set of values for a set of data points for each of the first set of electronic documents; and
recalibrating the at least one confidence score based on polynomial regression.

### Claim 17 (depends on 16)

The system of claim 16, the operations further comprising:
processing, by the ML model, a second set of electronic documents, comprising extracting one or more data points for each of the first set of electronic documents;
providing a new confidence score solution comprising improving both calibration quality and selective prediction accuracy of the second set of electronic documents based on a combination of temperature scaling, top-k softmax denomination and recalibrating of the at least one confidence score of the first set of electronic documents; and
indicating, by the ML model, confidence scores for one or more data point values extracted from the second set of electronic documents, when processed by the ML model for extraction.

### Claim 18 (depends on 17)

The system of claim 17, the operations further comprising:
a sequence of per-token logits comprising a series of numerical values that represent the predictions for each token in a sequence of electronic documents based on the ML model.

### Claim 19 (depends on 18)

The system of claim 18, wherein logits include raw, unnormalized output of a neural network prior to application of a softmax function, and comprising:
using the softmax function to convert the logits into probabilities, the logits providing a preliminary measurement of the at least one confidence score for the ML model.

### Claim 20 (depends on 19)

The system of claim 19, the operations further comprising:
identifying a quality of confidence score;
recalibrating the confidence score based on the quality of the confidence score; and
applying the combination of the temperature scaling, softmax denominator top-k probability selection, and the polynomial regression to the confidence score.

### Claim 21

A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving results generated by a machine learning (ML) model, the results comprising at least one confidence score and a plurality of electronic documents;
processing the results generated by the ML model, the processing comprising performing document understanding by extracting data points from the plurality of electronic documents;
associating the at least one confidence score with the extracted data points;
calibrating the at least one confidence score associated with the extracted data points using a post-hoc calibration (PHC) solution set;
implementing confidence scoring recalibration, the confidence scoring recalibration comprising functionality to assess reliability of the results generated by the ML model, the confidence scoring recalibration comprising aligning the at least one confidence score with prediction accuracy;
adjusting the at least one confidence score generated by the confidence scoring recalibration;
based on the adjusting of the at least one confidence score, extracting an individual element of information from the plurality of electronic documents, the extracting of the individual element of the information comprising one or more extracted values; and
generating an output comprising the one or more extracted values in a database comprising an adjusted confidence score associated with each of the one or more extracted values and the results generated by a ML model.

### Claim 22 (depends on 21)

The machine-storage medium of claim 21, the operations further comprising:
utilizing selective prediction accuracy to improve the performance of the ML model, the selective prediction accuracy comprising the functionality to predict an accuracy of the results generated by the ML model; and
improving both calibration quality and the selective prediction accuracy of the ML model performing the processing of the results and the document understanding.

### Claim 23 (depends on 21)

The machine-storage medium of claim 21, wherein the PHC solution set comprises at least one of temperature scaling, softmax denominator top-k probability selection, and polynomial regression.

### Claim 24 (depends on 23)

The machine-storage medium of claim 23, the operations further comprising:
training the ML model comprising the temperature scaling on a validation set, wherein the validation set is a subset of a dataset used during the training of the ML model.

### Claim 25 (depends on 21)

The machine-storage medium of claim 21, wherein the ML model further comprises:
using a pre-trained large language model;
generating one or more outputs from the pre-trained large language model; and
enabling the pre-trained large language model to make predictions on data categories not used for pre-training.

### Claim 26 (depends on 25)

The machine-storage medium of claim 25, the operations further comprising:
processing a first set of electronic documents, using the pre-trained large language model, to extract a set of values for a set of data points for each of the first set of electronic documents; and
recalibrating the at least one confidence score based on polynomial regression.

### Claim 27 (depends on 26)

The machine-storage medium of claim 26, the operations further comprising:
processing, by the ML model, a second set of electronic documents, comprising extracting one or more data points for each of the first set of electronic documents;
providing a new confidence score solution comprising improving both calibration quality and selective prediction accuracy of the second set of electronic documents based on a combination of temperature scaling, top-k softmax denomination and recalibrating of the at least one confidence score of the first set of electronic documents; and
indicating, by the ML model, confidence scores for one or more data point values extracted from the second set of electronic documents, when processed by the ML model for extraction.

### Claim 28 (depends on 27)

The machine-storage medium of claim 27, the operations further comprising:
a sequence of per-token logits comprising a series of numerical values that represent the predictions for each token in a sequence of electronic documents based on the ML model.

### Claim 29 (depends on 28)

The machine-storage medium of claim 28, wherein logits include raw, unnormalized output of a neural network prior to application of a softmax function, and comprising:
using the softmax function to convert the logits into probabilities, the logits providing a preliminary measurement of the at least one confidence score for the ML model.

### Claim 30 (depends on 29)

The machine-storage medium of claim 29, the operations further comprising:
identifying a quality of confidence score;
recalibrating the confidence score based on the quality of the confidence score; and
applying the combination of the temperature scaling, softmax denominator top-k probability selection, and the polynomial regression to the confidence score.
