# US 12,111,881 B2 · Item recommendation with application to automated artificial intelligence

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,111,881 B2
- Document ID: US12111881B2
- Status: Granted
- Kind: B2
- Application number: 17/108,107
- Filed: 2020-12-01
- Published: 2022-06-02
- Granted: 2024-10-08
- Earliest claimed: 2020-12-01
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Nico Stephan Gorbach; Adelmo Cristiano Innocenza Malossi; Andrea Bartezzaghi
- Web page: https://patentgrep.com/patent/US12111881B2

## CPC

- G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/232: Non-hierarchical techniques
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A plurality of rating values corresponding to the performance of each of a plurality of pipelines for a dataset of a plurality datasets is stored. The distance between each pair of datasets within the plurality is determined in a latent space. The distance corresponds to an expected value of a regret incurred when a selected pipeline is rated for a performance of the pipeline for a first dataset. The regret is a monotonically decreasing function of the rating value, according to the pipeline's performance. The plurality of datasets is clustered in the latent space, and a representative dataset in each cluster is identified. In response to receiving a new dataset, a set of preferred pipelines is selected from the plurality, according to a set of rating values corresponding to each of the selected pipelines' performance on the representative dataset. The new dataset is processed in the set of preferred pipelines.

## Claims

### Claim 1

A computer-implemented method for selecting preferred machine learning pipelines for processing new datasets, the method comprising:
for a plurality of machine learning pipelines and a plurality N of datasets previously-processed by the pipelines, storing a plurality of rating values, each rating value corresponding to a performance of a pipeline of the plurality of machine learning pipelines and for a dataset of the plurality N of datasets;
for each pair ui=1 to N, uj=1 to N, i±j of the plurality N of previously-processed datasets, determining a distance di,j from ui to uj in a latent space, wherein the distance di,j corresponds to an expected value of a regret incurred when the pipeline, selected in a predetermined manner based on a set of rating values for a dataset uj, is rated for a performance of the selected pipeline for a dataset ui, and wherein the regret for the selected pipeline includes a monotonically decreasing function of the rating value for the performance of the pipeline for the dataset ui,
clustering the plurality N of previously-processed datasets in the latent space and identifying a representative dataset in each cluster for which each distance to the dataset from other datasets in the cluster is minimized over the cluster;
in response to receiving a new dataset, selecting a set of preferred pipelines from the plurality of machine learning pipelines for processing the new dataset, each preferred pipeline being selected according to a set of rating values for the representative dataset; and
processing the new dataset in the set of preferred pipelines.

### Claim 2 (depends on 1)

The method as claimed in claim 1, wherein the predetermined manner for selecting the pipeline based on the rating values for the dataset uj comprises:
selecting the pipeline as a random variable from a probability distribution conditioned on the rating values for the dataset uj; and
wherein the expected value of the regret comprises a probability-weighted average of a set of regrets incurred for a respective set of preferred pipelines.

### Claim 3 (depends on 2)

The method as claimed in claim 2, further comprising:
determining a rating value for a performance of each pipeline of the set of preferred pipelines for the new dataset;
calculating a regret incurred for each pipeline of the set of preferred pipelines for the new dataset;
selecting at least one further preferred pipeline in the predetermined manner, based on the rating values for the performance and the representative dataset corresponding to the at least one further preferred pipeline with a lowest regret for the new dataset; and
processing the new dataset in each of the at least one further preferred pipeline.

### Claim 4 (depends on 3)

The method as claimed in claim 3, further comprising:
determining a rating value for the performance of each of the at least one further preferred pipeline for the new dataset; and
identifying, in response to the determination of the rating value for the performance of each of the at least one further preferred pipeline for the new dataset, a pipeline among the at least one further pipeline with a highest rating value for the new dataset.

### Claim 5 (depends on 4)

The method as claimed in claim 4, wherein the set of preferred pipelines comprises a set of highest-rated pipelines for the corresponding representative datasets.

### Claim 6 (depends on 2)

The method as claimed in claim 2, wherein a probability of selecting the pipeline from said probability distribution for the dataset uj varies directly with the rating value for performance of the selected pipeline on the dataset uj.

### Claim 7 (depends on 2)

The method as claimed in claim 2, wherein:
the probability distribution for the dataset uj is dependent on a parameter β, whose value determines a degree of the dependence of the probability distribution on the rating values for the dataset; and
the method further comprises selecting a value of the parameter ß to provide a desired degree of dependence of the probability distribution on the set of rating values of the representative dataset.

### Claim 8 (depends on 7)

The method as claimed in claim 7, further comprising selecting an optimal value of the parameter β, by cross-validation using the plurality N datasets.

### Claim 9 (depends on 7)

The method as claimed in claim 7, wherein the probability P (k|uj) of selecting a pipeline Pk from the probability distribution conditioned on the rating values for the dataset uj satisfies
P(k|uj)∝exp(−β(Mj−Rj,k)); and
wherein Mj is a maximum rating for performance of pipelines on dataset uj, and Rj,k is the rating for performance of pipeline Pk on dataset uj.

### Claim 10 (depends on 1)

The method as claimed in claim 1, wherein the regret for a pipeline Pk rated for the dataset ui is a function of the difference between a maximum rating value for performance of pipelines on dataset ui and a rating value for performance of the pipeline Pk on dataset ui.

### Claim 11

A computing system for selecting preferred machine learning pipelines for processing new datasets, the system comprising:
a memory subsystem, with program instructions included thereon; and
a processor in communication with the memory subsystem, wherein the program instructions cause the processor to:
for a plurality of machine learning pipelines and a plurality N of datasets previously-processed by the pipelines, store a plurality of rating values, each rating value corresponding to a performance of a pipeline of the plurality of machine learning pipelines and for a dataset of the plurality N of datasets;
for each pair ui=1 to N, uj=1 to N, i±j of the plurality N of previously-processed datasets, determine a distance di,j from ui to uj in a latent space, wherein the distance di,j corresponds to an expected value of a regret incurred when the pipeline, selected in a predetermined manner based on a set of rating values for a dataset uj, is rated for a performance of the selected pipeline for a dataset ui, and wherein the regret for the selected pipeline includes a monotonically decreasing function of the rating value for the performance of the pipeline for the dataset ui,
cluster the plurality N of previously-processed datasets in the latent space and identify a representative dataset in each cluster for which each distance to the dataset from other datasets in the cluster is minimized over the cluster;
in response to receiving a new dataset, select a set of preferred pipelines from the plurality of machine learning pipelines for processing the new dataset, each preferred pipeline being selected according to a set of rating values for the representative dataset; and
process the new dataset in the set of preferred pipelines.

### Claim 12 (depends on 11)

The system as claimed in claim 11, wherein the predetermined manner for selecting the pipeline based on the rating values for the dataset uj comprises:
selecting the pipeline as a random variable from a probability distribution conditioned on the rating values for the dataset uj; and
wherein the expected value of the regret comprises a probability-weighted average of a set of regrets incurred for a respective set of preferred pipelines.

### Claim 13 (depends on 12)

The system as claimed in claim 12, wherein the program instructions further cause the processor to:
determine a rating value for a performance of each pipeline of the set of preferred pipelines for the new dataset;
calculate a regret incurred for each pipeline of the set of preferred pipelines for the new dataset;
select at least one further preferred pipeline in the predetermined manner, based on the rating values for the performance and the representative dataset corresponding to the at least one further preferred pipeline with a lowest regret for the new dataset; and
process the new dataset in each of the at least one further preferred pipeline.

### Claim 14 (depends on 13)

The system as claimed in claim 13, wherein the program instructions further cause the processor to:
determine a rating value for the performance of each of the at least one further preferred pipeline for the new dataset; and
identify, in response to the determination of the rating value for the performance of each of the at least one further preferred pipeline for the new dataset, a pipeline among the at least one further pipeline with a highest rating value for the new dataset.

### Claim 15 (depends on 14)

The system as claimed in claim 14, wherein the set of preferred pipelines comprises a set of highest-rated pipelines for the corresponding representative datasets.

### Claim 16

A computer program product for selecting preferred machine learning pipelines for processing new datasets, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a device to cause the device to:
for a plurality of machine learning pipelines and a plurality N of datasets previously-processed by the pipelines, store a plurality of rating values, each rating value corresponding to a performance of a pipeline of the plurality of machine learning pipelines and for a dataset of the plurality N of datasets;
for each pair ui=1 to N, uj=1 to N, i±j of the plurality N of previously-processed datasets, determine a distance di,j from ui to uj in a latent space, wherein the distance di,j corresponds to an expected value of a regret incurred when the pipeline, selected in a predetermined manner based on a set of rating values for a dataset uj, is rated for a performance of the selected pipeline for a dataset ui, and wherein the regret for the selected pipeline includes a monotonically decreasing function of the rating value for the performance of the pipeline for the dataset ui;
cluster the plurality N of previously-processed datasets in the latent space and identify a representative dataset in each cluster for which each distance to the dataset from other datasets in the cluster is minimized over the cluster;
in response to receiving a new dataset, select a set of preferred pipelines from the plurality of machine learning pipelines for processing the new dataset, each preferred pipeline being selected according to a set of rating values for the representative dataset; and
process the new dataset in the set of preferred pipelines.

### Claim 17 (depends on 16)

The computer program product as claimed in claim 16, wherein the predetermined manner for selecting the pipeline based on the rating values for the dataset uj comprises:
selecting the pipeline as a random variable from a probability distribution conditioned on the rating values for the dataset uj; and
wherein the expected value of the regret comprises a probability-weighted average of a set of regrets incurred for a respective set of preferred pipelines.

### Claim 18 (depends on 17)

The computer program product as claimed in claim 17, wherein the program instructions further cause the device to:
determine a rating value for a performance of each pipeline of the set of preferred pipelines for the new dataset;
calculate a regret incurred for each pipeline of the set of preferred pipelines for the new dataset;
select at least one further preferred pipeline in the predetermined manner, based on the rating values for the performance and the representative dataset corresponding to the at least one further preferred pipeline with a lowest regret for the new dataset; and
process the new dataset in each of the at least one further preferred pipeline.

### Claim 19 (depends on 18)

The computer program product as claimed in claim 18, wherein the program instructions further cause the device to:
determine a rating value for the performance of each of the at least one further preferred pipeline for the new dataset; and
identify, in response to the determination of the rating value for the performance of each of the at least one further preferred pipeline for the new dataset, a pipeline among the at least one further pipeline with a highest rating value for the new dataset.

### Claim 20 (depends on 19)

The computer program product as claimed in claim 19, wherein the set of preferred pipelines comprises a set of highest-rated pipelines for the corresponding representative datasets.