# US 2025/0156459 A1 · Training data identification and model selection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0156459 A1
- Document ID: US20250156459A1
- Status: Published application
- Kind: A1
- Application number: 18/509,435
- Filed: 2023-11-15
- Published: 2025-05-15
- Earliest claimed: 2023-11-15
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Anna Lisa Gentile; Daniel Gruhl; Shubhi Asthana; Chad Eric DeLuca; Pawan Raghunath Chowdhary; Alfredo Alba; Steven R. Welch; Hirenkumar Ashokbhai Thummar; Linda Ha Kato; Bing Zhang
- Web page: https://patentgrep.com/patent/US20250156459A1

## CPC

- G06F 16/3344: {using natural language analysis}
- G06F 16/24578: {using ranking}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Techniques are provided for training data identification and model selection. In one embodiment, the techniques involve querying a first language learning model with a first query, wherein the first language learning model generates a text output response to the first query, generating groupings of the text output response, wherein the groupings include multiple n-grams, generating a source score of the groupings based on a first training data and a second training data, and identifying the first training data as training data of the first language learning model based on the source score.

## Claims

### Claim 1

A method comprising:
querying a first language learning model with a first query, wherein the first language learning model generates a text output response to the first query;
generating groupings of the text output response, wherein the groupings include multiple n-grams;
generating a source score of the groupings based on a first training data and a second training data; and
identifying the first training data as training data of the first language learning model based on the source score.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
generating a ranking of training data based on queries and ratings of corresponding responses of a set of language learning models, wherein the ranking of training data includes a ranking of the first training data;
selecting a second language learning model of the set of language learning models based on the ranking of training data and a second query; and
generating a response to the second query based on the second language learning model.

### Claim 3 (depends on 1)

The method of claim 1, wherein the source score represents a likelihood that the first training data was used to train the first language learning model, and wherein generating the source score involves:
generating a first score of the groupings based on the first training data, wherein the first training data represents a potential source of the training data used by the first language learning model; and
generating a second score of the groupings based on the second training data, wherein the second training data represents standard communication data of a given population.

### Claim 4 (depends on 3)

The method of claim 3, wherein the source score is determined as ScoreS=max(0, (Score1−Score2)), wherein ScoreS represents the source score, wherein Score1 represents the first score of the groupings, and wherein Score2 represents the second score of the groupings.

### Claim 5 (depends on 4)

The method of claim 4, wherein the first score is determined as
Score
1
=
∑
log
10
⁢
(
n
-
grams
overlaps
+
1
)
(
n
-
grams
unique
)
log
10
(
firsttrainingdata
Bytes
)
,
wherein Score1 represents the first score, wherein n-gramsoverlaps represents a number of overlaps between each of the n-grams and the first training data, wherein n-gramsunique represents a number of unique n-grams in the text output response, and wherein the first training data Bytes represents a size of the first training data in bytes.

### Claim 6 (depends on 4)

The method of claim 4, wherein the second score is determined as
Score
2
=
∑
log
10
⁢
(
n
-
grams
overlaps
+
1
)
(
n
-
grams
unique
)
log
10
(
secondtrainingdata
Bytes
)
,
wherein Score2 represents the second score, wherein n-gramsoverlaps represents a number of overlaps between each of the n-grams and the second training data, wherein n-gramsunique represents a number of unique n-grams in the text output response, and wherein the second training data Bytes represents a size of the second training data in bytes.

### Claim 7

A system, comprising:
a processor; and
memory or storage comprising an algorithm or computer instructions, which when executed by the processor, performs an operation comprising:
querying a first language learning model with a first query, wherein the first language learning model generates a text output response to the first query;
generating groupings of the text output response, wherein the groupings include multiple n-grams;
generating a source score of the groupings based on a first training data and a second training data; and
identifying the first training data as training data of the first language learning model based on the source score.

### Claim 8 (depends on 7)

The system of claim 7, the operation further comprising:
generating a ranking of training data based on queries and ratings of corresponding responses of a set of language learning models, wherein the ranking of training data includes a ranking of the first training data;
selecting a second language learning model of the set of language learning models based on the ranking of training data and a second query; and
generating a response to the second query based on the second language learning model.

### Claim 9 (depends on 7)

The system of claim 7, wherein the source score represents a likelihood that the first training data was used to train the first language learning model, and wherein generating the source score involves:
generating a first score of the groupings based on the first training data, wherein the first training data represents a potential source of the training data used by the first language learning model; and
generating a second score of the groupings based on the second training data, wherein the second training data represents standard communication data of a given population.

### Claim 10 (depends on 9)

The system of claim 9, wherein the source score is determined as ScoreS=max(0, (Score1−Score2)), wherein ScoreS represents the source score, wherein Score1 represents the first score of the groupings, and wherein Score2 represents the second score of the groupings.

### Claim 11 (depends on 10)

The system of claim 10, wherein the first score is determined as
Score
1
=
∑
log
10
⁢
(
n
-
grams
overlaps
+
1
)
(
n
-
grams
unique
)
log
10
(
firsttrainingdata
Bytes
)
,
wherein Score1 represents the first score, wherein n-gramsoverlaps represents a number of overlaps between each of the n-grams and the first training data, wherein n-gramsunique represents a number of unique n-grams in the text output response, and wherein the first training data Bytes represents a size of the first training data in bytes.

### Claim 12 (depends on 10)

The system of claim 10, wherein the second score is determined as
Score
2
=
∑
log
10
⁢
(
n
-
grams
overlaps
+
1
)
(
n
-
grams
unique
)
log
10
(
secondtrainingdata
Bytes
)
,
wherein Score2 represents the second score, wherein n-gramsoverlaps represents a number of overlaps between each of the n-grams and the second training data, wherein n-gramsunique represents a number of unique n-grams in the text output response, and wherein the second training data Bytes represents a size of the second training data in bytes.

### Claim 13

A computer-readable storage medium having a computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising:
querying a first language learning model with a first query, wherein the first language learning model generates a text output response to the first query;
generating groupings of the text output response, wherein the groupings include multiple n-grams;
generating a source score of the groupings based on a first training data and a second training data; and
identifying the first training data as training data of the first language learning model based on the source score.

### Claim 14 (depends on 13)

The computer-readable storage medium of claim 13, the operation further comprising:
generating a ranking of training data based on queries and ratings of corresponding responses of a set of language learning models, wherein the ranking of training data includes a ranking of the first training data;
selecting a second language learning model of the set of language learning models based on the ranking of training data and a second query; and
generating a response to the second query based on the second language learning model.

### Claim 15 (depends on 13)

The computer-readable storage medium of claim 13, wherein the source score represents a likelihood that the first training data was used to train the first language learning model, and wherein generating the source score involves:
generating a first score of the groupings based on the first training data, wherein the first training data represents a potential source of the training data used by the first language learning model; and
generating a second score of the groupings based on the second training data, wherein the second training data represents standard communication data of a given population.

### Claim 16 (depends on 15)

The computer-readable storage medium of claim 15, wherein the source score is determined as ScoreS=max(0, (Score1−Score2)), wherein ScoreS represents the source score, wherein Score1 represents the first score of the groupings, and wherein Score2 represents the second score of the groupings.

### Claim 17 (depends on 16)

The computer-readable storage medium of claim 16, wherein the first score is determined as
Score
1
=
∑
log
10
⁢
(
n
-
grams
overlaps
+
1
)
(
n
-
grams
unique
)
log
10
(
firsttrainingdata
Bytes
)
,
wherein Score1 represents the first score, wherein n-gramsoverlaps represents a number of overlaps between each of the n-grams and the first training data, wherein n-gramsunique represents a number of unique n-grams in the text output response, and wherein the first training data Bytes represents a size of the first training data in bytes.

### Claim 18 (depends on 16)

The computer-readable storage medium of claim 16, wherein the second score is determined as
Score
2
=
∑
log
10
⁢
(
n
-
grams
overlaps
+
1
)
(
n
-
grams
unique
)
log
10
(
secondtrainingdata
Bytes
)
,
wherein Score2 represents the second score, wherein n-gramsoverlaps represents a number of overlaps between each of the n-grams and the second training data, wherein n-gramsunique represents a number of unique n-grams in the text output response, and wherein the second training data Bytes represents a size of the second training data in bytes.

### Claim 19

A system, comprising:
a processor; and
memory or storage comprising an algorithm or computer instructions, which when executed by the processor, performs an operation comprising:
identifying a first training data as training data of a first language learning model based on a source score;
generating a ranking of training data based on queries and ratings of corresponding responses of a set of language learning models, wherein the ranking of training data includes a ranking of a first training data;
selecting a second language learning model of the set of language learning models based on the ranking of training data and a query; and
generating a response to the query based on the second language learning model.

### Claim 20 (depends on 19)

The system of claim 19, the operation further comprising:
querying the first language learning model with a first query, wherein the first language learning model generates a text output response to the first query;
generating groupings of the text output response, wherein the groupings include multiple n-grams; and
generating the source score of the groupings based on a first training data and a second training data.

### Claim 21 (depends on 20)

The system of claim 20, wherein the source score represents a likelihood that the first training data was used to train the first language learning model, and wherein generating the source score involves:
generating a first score of the groupings based on the first training data, wherein the first training data represents a potential source of the training data used by the first language learning model; and
generating a second score of the groupings based on the second training data, wherein the second training data represents standard communication data of a given population.

### Claim 22 (depends on 21)

The system of claim 21, wherein the source score is determined as ScoreS=max(0, (Score1−Score2)), wherein ScoreS represents the source score, wherein Score1 represents the first score of the groupings, and wherein Score2 represents the second score of the groupings;
wherein the first score is determined as
Score
1
=
∑
log
10
⁢
(
n
-
grams
overlaps
+
1
)
(
n
-
grams
unique
)
log
10
(
firsttrainingdata
Bytes
)
,
wherein Score1 represents the first score, wherein n-gramsoverlaps represents a number of overlaps between each of the n-grams and the first training data, wherein n-gramsunique represents a number of unique n-grams in the text output response, and wherein the first training data Bytes represents a size of the first training data in bytes; and
wherein the second score is determined as
Score
2
=
∑
log
10
⁢
(
n
-
grams
overlaps
+
1
)
(
n
-
grams
unique
)
log
10
(
secondtrainingdata
Bytes
)
,
wherein Score2 represents the second score, wherein n-gramsoverlaps represents a number of overlaps between each of the n-grams and the second training data, wherein n-gramsunique represents a number of unique n-grams in the text output response, and wherein the second training data Bytes represents a size of the second training data in bytes.

### Claim 23

A computer-readable storage medium having a computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising:
identifying a first training data as training data of a first language learning model based on a source score;
generating a ranking of training data based on queries and ratings of corresponding responses of a set of language learning models, wherein the ranking of training data includes a ranking of a first training data;
selecting a second language learning model of the set of language learning models based on the ranking of training data and a query; and
generating a response to the query based on the second language learning model.

### Claim 24 (depends on 23)

The computer-readable storage medium of claim 23, the operation further comprising:
querying the first language learning model with a first query, wherein the first language learning model generates a text output response to the first query;
generating groupings of the text output response, wherein the groupings include multiple n-grams; and
generating the source score of the groupings based on a first training data and a second training data.

### Claim 25 (depends on 24)

The computer-readable storage medium of claim 24, wherein the source score represents a likelihood that the first training data was used to train the first language learning model, and wherein generating the source score involves:
generating a first score of the groupings based on the first training data, wherein the first training data represents a potential source of the training data used by the first language learning model; and
generating a second score of the groupings based on the second training data, wherein the second training data represents standard communication data of a given population.
