# US 12,118,474 B2 · Techniques for adaptive pipelining composition for machine learning (ML)

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,474 B2
- Document ID: US12118474B2
- Status: Granted
- Kind: B2
- Application number: 18/132,859
- Filed: 2023-04-10
- Published: 2023-10-19
- Granted: 2024-10-15
- Earliest claimed: 2019-09-14
- Assignees: Oracle International Corporation
- Applicants: Oracle International Corporation
- Inventors: Alberto Polleri; Larissa Cristina Dos Santos Romualdo Suzuki; Sergio Aldea Lopez; Marc Michiel Bron; Dan David Golding; Alexander Ioannides; Maria del Rosario Mestre; Hugo Alexandre Pereira Monteiro; Oleg Gennadievich Shevelev; Xiaoxue Zhao; Matthew Charles Rowe
- Web page: https://patentgrep.com/patent/US12118474B2

## CPC

- G06F 11/3003: {Monitoring arrangements specially adapted to the computing system or computing system component being monitored}
- G06N 5/02: Knowledge representation; Symbolic representation
- G06F 8/77: Software metrics
- G06F 11/3409: {for performance assessment}
- G06F 11/3433: {for load management (allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029)}
- G06F 11/3452: {Performance evaluation by statistical analysis}
- G06F 11/3466: {Performance evaluation by tracing or monitoring}
- G06F 16/907: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/2193: {based on specific statistical tests}
- G06F 18/285: {Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system}
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence
- G06N 20/00: Machine learning
- G06N 20/20: Ensemble learning
- H04L 9/088: {Usage controlling of secret information, e.g. techniques for restricting cryptographic keys to pre-authorized uses, different access levels, validity of crypto-period, different key- or password length, or different strong and weak cryptographic algorithms (network architectures or network communication protocols for using time-dependent keys in a packet data network H04L63/068)}
- H04L 9/3247: {involving digital signatures}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

The present disclosure relates to systems and methods for an adaptive pipelining composition service that can identify and incorporate one or more new models into the machine learning application. The machine learning application with the new model can be tested off-line with the results being compared with ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and auto-promoted to production. One or more parameters may also be discovered. The new parameters may be incorporated into the existing model in an off-line mode. The machine learning application with the new parameters can be tested off-line and the results can be compared with previous results with existing parameters. If the new parameters outperform the existing parameters as compared with ground-truth data, the machine learning application can be auto-promoted to production.

## Claims

### Claim 1

A computer-implemented method, the method comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters to identify and define properties of the first machine learning model;
generating a first result for the first machine learning model using a first data set as an input to the first machine learning model;
identifying one or more metrics for the first machine learning model, wherein the metrics define how a performance of the first machine learning model is measured;
generating a second result for the first machine learning model using a second data set as the input to the first machine learning model, wherein the second data set comprises a labeled data set;
comparing the first result with the second result to calculate a first scoring;
analyzing the one or more first parameters to identify an ontology for the first machine learning model;
using the ontology to identify a second machine learning model based at least in part on comparing first metadata of the second machine learning model with second metadata for the first machine learning model;
testing the second machine learning model on the first data set to produce a third result;
generating a fourth result for the second machine learning model using the second data set as the input to the second machine learning model;
comparing the third result to the fourth result to generate a second scoring; and
based on the second scoring being less than the first scoring, replacing the first machine learning model with the second machine learning model in a machine learning application.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
storing the second machine learning model in a memory.

### Claim 3 (depends on 1)

The method of claim 1, wherein the replacing the first machine learning model with the second machine learning model is performed in a shadow mode until it the second machine learning model satisfies one or more conditions for auto-promoting the second machine learning model to production.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
generating a log comprising the one or more first parameters, the first machine learning model, the second machine learning model, the first result, and the second result; and
storing the log in a memory.

### Claim 5 (depends on 4)

The method of claim 4, further comprising analyzing the log to determine one or more patterns.

### Claim 6 (depends on 4)

The method of claim 4, further comprising saving supplemental metadata concerning the second machine learning model, wherein the supplemental metadata includes at least the one or more first parameters the second result.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first metadata comprises at least one of a number of levels for a decision tree and a number of parameters of an algorithm for the second machine learning model.

### Claim 8

A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a data processing apparatus to perform operations executed as a background process of a machine learning application, the operations comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters to identify and define properties of the first machine learning model;
generating a first result for the first machine learning model using a first data set as an input to the first machine learning model;
identifying one or more metrics for the first machine learning model, wherein the metrics determine a performance of the first machine learning model;
generating a second result for the first machine learning model using a second data set as the input to the first machine learning model, wherein the second data set comprises ground truth data;
comparing the first result with the second result to calculate a first scoring;
analyzing one or more first parameters to identify an ontology for the first machine learning model;
using the ontology to identify a second machine learning model based at least in part on comparing first metadata of the second machine learning model with second metadata for the first machine learning model;
testing the second machine learning model on the first data set to produce a third result;
generating a fourth result for the second machine learning model using the second data set as the input to the second machine learning model;
comparing the third result to the fourth result to generate a second scoring; and
based on the second scoring being less than the first scoring, replacing the first machine learning model with the second machine learning model for the machine learning application.

### Claim 9 (depends on 8)

The computer-program product of claim 8, including instructions configured to cause a data processing apparatus to perform further operations comprising storing the second machine learning model in a memory.

### Claim 10 (depends on 8)

The computer-program product of claim 8, wherein the replacing the first machine learning model is replaced by the second machine learning model in a shadow mode until the second machine learning model satisfies one or more conditions for auto-promoting the second machine learning model to production.

### Claim 11 (depends on 8)

The computer-program product of claim 8, including instructions configured to cause a data processing apparatus to perform further operations comprising:
generating a log comprising the one or more first parameters, the first machine learning model, the second machine learning model, the first result, and the second result; and
storing the log in a memory.

### Claim 12 (depends on 11)

The computer-program product of claim 11, including instructions configured to cause a data processing apparatus to perform further operations comprising analyzing the log to determine one or more patterns.

### Claim 13 (depends on 11)

The computer-program product of claim 11, including instructions configured to cause a data processing apparatus to perform further operations comprising saving supplemental metadata concerning the second machine learning model based at least in part on the one or more first parameters the second result.

### Claim 14 (depends on 8)

The computer-program product of claim 8, wherein the first metadata comprises at least one of a number of levels for a decision tree and a number of parameters of an algorithm for the second machine learning model.

### Claim 15

A system for executed as a background process of a machine learning application, comprising:
one or more data processors; and
a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations comprising:
identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters to identify and define properties of the first machine learning model;
generating a first result for the machine learning model using a first data set as an input to the first machine learning model;
identifying one or more metrics for the first machine learning model, wherein the metrics determine a performance of the first machine learning model;
generating a second result for the first machine learning model using a second data set as the input to the first machine learning model, wherein the second data set comprises ground truth data;
comparing the first result to the second result to calculate a first scoring;
analyzing the one or more first parameters to identify an ontology for the first machine learning model;
using the ontology to identify a second machine learning model based at least in part on comparing first metadata of the second machine learning model with second metadata for the first machine learning model;
testing the second machine learning model on the first data set to produce a third result;
generating a fourth result for the second machine learning model using the second data set as the input to the second machine learning model;
comparing the third result to the fourth result to generate a second scoring; and
based on the second scoring being less than the first scoring, replacing the first machine learning model with the second machine learning model for the machine learning application.

### Claim 16 (depends on 15)

The system of claim 15, wherein operations further comprise storing the second machine learning model in a memory.

### Claim 17 (depends on 15)

The system of claim 15, wherein the replacing the first machine learning model is replaced by the second machine learning model in a shadow mode until the second machine learning model satisfies one or more conditions for auto-promoting the second machine learning model to production.

### Claim 18 (depends on 15)

The system of claim 15, wherein the operations further comprise:
generating a log comprising the one or more first parameters, the first machine learning model, the second machine learning model, the first result, and the second result; and
storing the log in a memory.

### Claim 19 (depends on 18)

The system of claim 18, wherein the a non-transitory computer-readable storage medium includes further instructions which, when executed on the one or more data processors, cause the one or more data processors to perform further operations comprising saving supplemental metadata concerning the second machine learning model based at least in part on the one or more first parameters the second result.

### Claim 20 (depends on 15)

The system of claim 15, wherein the first metadata comprises at least one of a number of levels for a decision tree and a number of parameters of an algorithm for the second machine learning model.