US 2026/0228581 A1Published application

Foundation model evaluation using synthetic data

AssigneeInternational Business Machines Corporation

InventorsPin-Yu Chen, Payel Das, Ching-Yun Ko

Life of this patent

Estimating the term…

Loading patent…

Abstract

A method, computer program product, and computer system for evaluating foundation models. Synthetic data is generated for binary classification for a downstream task, by random sampling of a mixture of two Gaussian probability distributions. The synthetic data is classified into two classes. Multiple trained foundation models are executed using the synthetic data as input, resulting in each model outputting features data corresponding to features of the synthetic data and classifying the features data using a ϵ-robust Bayes optimal classifier, where e denotes robustness. A ratio of area under the curve of E(a)/E(θm, ε, a) is computed for each model of the multiple models, where a denotes accuracy of the classified randomly sampled data, E(a) is an expected scaled bound of the classified randomly sampled data, and E(θm, ε, a) is an accuracy-constrained expected bound of the features data. The model having the largest ratio is selected.

Claims20

Dependent claims

A method for evaluating foundation models, said method comprising:

generating synthetic data for binary classification for a downstream task characterized by d independent variables, by randomly sampling data from a mixture Gaussian k(G(y{tilde over (μ)}k, Σk))(k=1, . . . , K) of (i) a first d-dimensional Gaussian probability density function (PDF) specific to a first class characterized by y=+1 and having a mean of μ1k and a covariance of Σk and (ii) a second d-dimensional Gaussian PDF specific to a second class characterized by y=−1 and having a mean of μ2k and a covariance of Σk, wherein {tilde over (μ)}k=(μ1k−μ2k)/2, wherein d≥1 and K≥2;
classifying the randomly sampled data, using a Bayes optimal classifier, into the first class and the second class;
executing M trained foundation models using the synthetic data as input and applying an ϵ-robust Bayes optimal classifier, resulting in foundation model m outputting features data corresponding to features of the synthetic data and classifying the features data using the ϵ-robust Bayes optimal classifier, wherein each foundation model m is characterized by model parameters θm(m=1, . . . , M), wherein e is an inputted robustness, and wherein M≥2;
computing an expected scaled bound E(a) of the classified randomly sampled data as a function of an accuracy (a) of the classified randomly sampled data, each accuracy (a) being at least an inputted threshold accuracy (at);
computing an accuracy-constrained expected bound E(θm, ε, a) of the features data as a function of the accuracy (a) for each foundation model m (m=1, . . . , M) for the inputted e;
computing
R
m
=
∫
a
t
1
E
⁡
(
θ
m
,
ϵ
,
a
)
⁢
da
/
∫
a
t
1
E
⁡
(
a
)
⁢
da
for each model m (m=1, . . . , M);
selecting m*=argmax (Rm) for m ∈ (1, . . . , M), which identifies a best foundation model m*.

36 more elements

A computer program product, comprising one or more computer readable storage media storing computer readable program instructions, said program instructions executable by one or more processors of a computer system to cause the computer system to perform operations for evaluating foundation models, said operations comprising:

generating synthetic data for binary classification for a downstream task characterized by d independent variables, by randomly sampling data from a mixture Gaussian k (G(y{tilde over (μ)}k, Σk)) (k=1, . . . , K) of (i) a first d-dimensional Gaussian probability density function (PDF) specific to a first class characterized by y=+1 and having a mean of μ1k and a covariance of Σk and (ii) a second d-dimensional Gaussian PDF specific to a second class characterized by y=−1 and having a mean of μ2k and a covariance of Σk, wherein {tilde over (μ)}k=(μ1k−μ2k)/2, wherein d≥1 and K≥2;
classifying the randomly sampled data, using a Bayes optimal classifier, into the first class and the second class;
executing M trained foundation models using the synthetic data as input and applying an ϵ-robust Bayes optimal classifier, resulting in foundation model m outputting features data corresponding to features of the synthetic data and classifying the features data using the ϵ-robust Bayes optimal classifier, wherein each foundation model m is characterized by model parameters θm(m=1, . . . , M), wherein e is an inputted robustness, and wherein M≥2;
computing an expected scaled bound E(a) of the classified randomly sampled data as a function of an accuracy (a) of the classified randomly sampled data, each accuracy (a) being at least an inputted threshold accuracy (at);
computing an accuracy-constrained expected bound E(θm, ε, a) of the features data as a function of the accuracy (a) for each foundation model m (m=1, . . . , M) for the inputted E;
computing
R
m
=
∫
a
t
1
E
⁡
(
θ
m
,
ϵ
,
a
)
⁢
da
/
∫
a
t
1
E
⁡
(
a
)
⁢
da
for each model m (m=1, . . . , M);
selecting m*=argmax (Rm) for m ε (1, . . . , M), which identifies a best foundation model m *.

36 more elements

A computer system, comprising one or more processors, one or more memories, one or more computer readable storage media, and computer readable program instructions stored on the one or more computer readable storage media for execution by the one or more processors via the one or more memories to cause the computer system to perform operations for evaluating foundation models, said operations comprising:

generating synthetic data for binary classification for a downstream task characterized by d independent variables, by randomly sampling data from a mixture Gaussian k (G(y{tilde over (μ)}k, μk)) (k=1, . . . , K) of (i) a first d-dimensional Gaussian probability density function (PDF) specific to a first class characterized by y=+1 and having a mean of μ1k and a covariance of Σk and (ii) a second d-dimensional Gaussian PDF specific to a second class characterized by y=−1 and having a mean of μ2k and a covariance of Σk, wherein {tilde over (μ)}k=(μ1k−μ2k)/2, wherein d≥1 and K≥2;
classifying the randomly sampled data, using a Bayes optimal classifier, into the first class and the second class;
executing M trained foundation models using the synthetic data as input and applying an ϵ-robust Bayes optimal classifier, resulting in foundation model m outputting features data corresponding to features of the synthetic data and classifying the features data using the ϵ-robust Bayes optimal classifier, wherein each foundation model m is characterized by model parameters θm(m=1, . . . , M), wherein e is an inputted robustness, and wherein M≥2;
computing an expected scaled bound E(a) of the classified randomly sampled data as a function of an accuracy (a) of the classified randomly sampled data, each accuracy (a) being at least an inputted threshold accuracy (at);
computing an accuracy-constrained expected bound E(θm, ε, a) of the features data as a function of the accuracy (a) for each foundation model m (m=1, . . . , M) for the inputted ϵ;
computing
R
m
=
∫
a
t
1
E
⁡
(
θ
m
,
ϵ
,
a
)
⁢
da
/
∫
a
t
1
E
⁡
(
a
)
⁢
da
for each model m (m=1, . . . , M);
selecting m*=argmax (Rm) for m ∈ (1, . . . , M), which identifies a best foundation model m *.

36 more elements

Dependencies are inferred from the claim text.