# US 2021/0065048 A1 · Automated artificial intelligence radial visualization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0065048 A1
- Document ID: US20210065048A1
- Status: Published application
- Kind: A1
- Application number: 16/557,760
- Filed: 2019-08-30
- Published: 2021-03-04
- Earliest claimed: 2019-08-30
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Theodoros SALONIDIS; John EVERSMAN; Dakuo WANG; Alex SWAIN; Gregory BRAMBLE; Lin JU; Nicholas MAZZITELLI; Voranouth SUPADULYA
- Web page: https://patentgrep.com/patent/US20210065048A1

## CPC

- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06N 20/00: Machine learning
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 9/3867: {using instruction pipelines}
- G06F 9/542: {Event management; Broadcasting; Multicasting; Notifications}
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/20: Ensemble learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Embodiments for providing automated machine learning visualization. Machine learning tasks, transformers, and estimators may be received into one or more machine learning composition modules. The machine learning composition modules generate one or more machine learning models. A machine learning model pipeline is a sequence of transformers and estimators and an ensemble of machine learning pipelines are an ensemble of machine learning pipelines. A machine learning model pipeline, an ensemble of a plurality of machine learning model pipelines, or a combination thereof, along with corresponding metadata, may be generated using the machine learning composition modules. Metadata may be extracted from the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination thereof. An interactive visualization graphical user interface of the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination thereof, and the extracted metadata may be generated.

## Claims

### Claim 1

A method for providing automated machine learning visualization by one or more processors, comprising:
receiving one or more machine learning tasks, one or more transformers, and one or more estimators into one or more machine learning composition modules;
generating a machine learning model pipeline, an ensemble of a plurality of machine learning model pipelines, or a combination thereof along with corresponding metadata using the one or more machine learning composition modules, wherein a machine learning model pipeline is a sequence of transformers and estimators and an ensemble of machine learning pipelines are an ensemble of machine learning pipelines;
extracting metadata from the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination thereof; and
generating an interactive visualization graphical user interface (“GUI”) of the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination thereof, the extracted metadata, or a combination thereof.

### Claim 2 (depends on 1)

The method of claim 1, wherein receiving the one or more machine learning tasks further includes receiving training data, hold out data, test data, optimization metrics, evaluation metrics, target variable, the one or more transformers, and the one or more estimators into the one or more machine learning composition modules.

### Claim 3 (depends on 1)

The method of claim 1, further including:
defining metadata for the machine learning pipeline to include structure metadata, performance metadata, provenance metadata, or a combination thereof relating to the machine learning pipeline; or
defining metadata for the ensemble of a plurality of machine learning model pipelines to include structure metadata, performance metadata, provenance metadata, or a combination thereof relating to the ensemble of a plurality of machine learning model pipelines

### Claim 4 (depends on 1)

The method of claim 1, further including:
ranking the extracted metadata and the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or a combination thereof according to pipeline ranking criteria and metadata ranking criteria;
ranking structure metadata, performance metadata, provenance metadata, or a combination thereof for the machine learning pipeline according to the metadata ranking criteria; or
ranking the structure metadata, the performance metadata, the provenance metadata, or a combination thereof for the ensemble of a plurality of machine learning model pipelines according to the metadata ranking criteria.

### Claim 5 (depends on 1)

The method of claim 1, further including displaying the interactive visualization GUI as a radial structure having a plurality of concentric rings having one or more nodes displayed therein, wherein the plurality of concentric rings include at least machine learning pipeline rings, data rings, estimator rings, transformer rings, and composition module rings, wherein the one or more nodes represent the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or a combination thereof, data, the one or more estimators, the one or more transformers, and the machine learning composition modules used to generate the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or a combination thereof, wherein the having one or more nodes in the plurality of concentric rings are displayed in sequence based on different ranking criteria,

### Claim 6 (depends on 5)

The method of claim 5, further including:
associating one or more nodes with one or more of the plurality of concentric rings based;
associating and displaying details relating to the machine learning model pipeline or the ensemble of a plurality of machine learning model pipelines upon the user interacting with the one or more of the plurality of concentric rings; or
displaying each of the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination associated with the one or more selected data transformers, the one or more data estimators, the one or more machine learning composition modules, one or more data partitions, or a combination thereof upon the user interacting with the one or more of the plurality of concentric rings.

### Claim 7 (depends on 5)

The method of claim 5, further including:
automatically updating the interactive visualization GUI upon occurrence of one or more triggering events, or
automatically updating the interactive visualization upon a user:
selecting the pipeline ranking criteria and the metadata ranking criteria of the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination thereof the one or more transformers, the one or more estimators, the one or more machine learning composition modules, or a combination thereof for visualization in one or more corresponding rings of the interactive visualization;
selecting or interacting with one or more nodes located within the interactive visualization of the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination thereof; or
selecting or interacting with one or more nodes for of one or more of a plurality of metadata rings within the interactive visualization GUI, wherein the plurality of metadata rings include transformer rings, estimators rings, data rings, and composition modules rings.

### Claim 8

A system for providing automated machine learning visualization, comprising:
one or more computers with executable instructions that when executed cause the system to:
receive one or more machine learning tasks, one or more transformers, and one or more estimators into one or more machine learning composition modules;
generate a machine learning model pipeline, an ensemble of a plurality of machine learning model pipelines, or a combination thereof along with corresponding metadata using the one or more machine learning composition modules, wherein a machine learning model pipeline is a sequence of transformers and estimators and an ensemble of machine learning pipelines are an ensemble of machine learning pipelines;
extract metadata from the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination thereof; and
generate an interactive visualization graphical user interface (“GUI”) of the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination thereof, the extracted metadata, or a combination thereof.

### Claim 9 (depends on 8)

The system of claim 8, wherein the executable instructions for receiving the one or more machine learning tasks further include receiving training data, hold out data, test data, optimization metrics, evaluation metrics, target variable, the one or more transformers, and the one or more estimators into the one or more machine learning composition modules.

### Claim 10 (depends on 8)

The system of claim 8, wherein the executable instructions further:
define metadata for the machine learning pipeline to include structure metadata, performance metadata, provenance metadata, or a combination thereof relating to the machine learning pipeline; or
define metadata for the ensemble of a plurality of machine learning model pipelines to include structure metadata, performance metadata, provenance metadata, or a combination thereof relating to the ensemble of a plurality of machine learning model pipelines

### Claim 11 (depends on 8)

The system of claim 8, wherein the executable instructions further:
rank the extracted metadata and the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or a combination thereof according to pipeline ranking criteria and metadata ranking criteria;
rank structure metadata, performance metadata, provenance metadata, or a combination thereof for the machine learning pipeline according to the metadata ranking criteria; or
rank the structure metadata, the performance metadata, the provenance metadata, or a combination thereof for the ensemble of a plurality of machine learning model pipelines according to the metadata ranking criteria.

### Claim 12 (depends on 8)

The system of claim 8, wherein the executable instructions further display the interactive visualization GUI as a radial structure having a plurality of concentric rings having one or more nodes displayed therein, wherein the plurality of concentric rings include at least machine learning pipeline rings, data rings, estimator rings, transformer rings, and composition module rings, wherein the one or more nodes represent the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or a combination thereof, data, the one or more estimators, the one or more transformers, and the machine learning composition modules used to generate the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or a combination thereof, wherein the having one or more nodes in the plurality of concentric rings are displayed in sequence based on different ranking criteria,

### Claim 13 (depends on 8)

The system of claim 8, wherein the executable instructions further:
associate one or more nodes with one or more of the plurality of concentric rings based;
associate and display details relating to the machine learning model pipeline or the ensemble of a plurality of machine learning model pipelines upon the user interacting with the one or more of the plurality of concentric rings; or
display each of the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination associated with the one or more selected data transformers, the one or more data estimators, the one or more machine learning composition modules, one or more data partitions, or a combination thereof upon the user interacting with the one or more of the plurality of concentric rings.

### Claim 14 (depends on 8)

The system of claim 8, wherein the executable instructions further:
automatically update the interactive visualization GUI upon occurrence of one or more triggering events, or
automatically update the interactive visualization upon a user:
selecting the pipeline ranking criteria and the metadata ranking criteria of the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination thereof the one or more transformers, the one or more estimators, the one or more machine learning composition modules, or a combination thereof for visualization in one or more corresponding rings of the interactive visualization;
selecting or interacting with one or more nodes located within the interactive visualization of the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination thereof; or
selecting or interacting with one or more nodes for of one or more of a plurality of metadata rings within the interactive visualization GUI, wherein the plurality of metadata rings include transformer rings, estimators rinds, data rings, and composition modules rings.

### Claim 15

A computer program product for providing automated machine learning visualization by a processor, the computer program product comprising a non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:
an executable portion that receives one or more machine learning tasks, one or more transformers, and one or more estimators into one or more machine learning composition modules;
an executable portion that generates a machine learning model pipeline, an ensemble of a plurality of machine learning model pipelines, or a combination thereof along with corresponding metadata using the one or more machine learning composition modules, wherein a machine learning model pipeline is a sequence of transformers and estimators and an ensemble of machine learning pipelines are an ensemble of machine learning pipelines;
an executable portion that extracts metadata from the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination thereof; and
an executable portion that generates an interactive visualization graphical user interface (“GUI”) of the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination thereof, the extracted metadata, or a combination thereof.

### Claim 16 (depends on 15)

The computer program product of claim 15, wherein the executable portion that executable portion that receives the one or more machine learning tasks further receives in the one or more machine learning tasks training data, hold out data, test data, optimization metrics, evaluation metrics, target variable, the one or more transformers, and the one or more estimators into the one or more machine learning composition modules.

### Claim 17 (depends on 15)

The computer program product of claim 15, further including an executable portion that:
defines metadata for the machine learning pipeline to include structure metadata, performance metadata, provenance metadata, or a combination thereof relating to the machine learning pipeline; or
defines metadata for the ensemble of a plurality of machine learning model pipelines to include structure metadata, performance metadata, provenance metadata, or a combination thereof relating to the ensemble of a plurality of machine learning model pipelines

### Claim 18 (depends on 15)

The computer program product of claim 15, further including an executable portion that:
ranks the extracted metadata and the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or a combination thereof according to pipeline ranking criteria and metadata ranking criteria;
ranks structure metadata, performance metadata, provenance metadata, or a combination thereof for the machine learning pipeline according to the metadata ranking criteria; or
ranks the structure metadata, the performance metadata, the provenance metadata, or a combination thereof for the ensemble of a plurality of machine learning model pipelines according to the metadata ranking criteria.

### Claim 19 (depends on 15)

The computer program product of claim 15, further including an executable portion that display the interactive visualization GUI as a radial structure having a plurality of concentric rings having one or more nodes displayed therein, wherein the plurality of concentric rings include at least machine learning pipeline rings, data rings, estimator rings, transformer rings, and composition module rings, wherein the one or more nodes represent the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or a combination thereof, data, the one or more estimators, the one or more transformers, and the machine learning composition modules used to generate the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or a combination thereof, wherein the having one or more nodes in the plurality of concentric rings are displayed in sequence based on different ranking criteria,

### Claim 20 (depends on 15)

The computer program product of claim 15, further including an executable portion that:
associates one or more nodes with one or more of the plurality of concentric rings based;
associates and display details relating to the machine learning model pipeline or the ensemble of a plurality of machine learning model pipelines upon the user interacting with the one or more of the plurality of concentric rings; or
displays each of the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination associated with the one or more selected data transformers, the one or more data estimators, the one or more machine learning composition modules, one or more data partitions, or a combination thereof upon the user interacting with the one or more of the plurality of concentric rings.

### Claim 21 (depends on 15)

The computer program product of claim 15, further including an executable portion that:
automatically updates the interactive visualization GUI upon occurrence of one or more triggering events, or
automatically updates the interactive visualization upon a user:
selecting the pipeline ranking criteria and the metadata ranking criteria of the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination thereof the one or more transformers, the one or more estimators, the one or more machine learning composition modules, or a combination thereof for visualization in one or more corresponding rings of the interactive visualization;
selecting or interacting with one or more nodes located within the interactive visualization of the machine learning model pipeline, the ensemble of a plurality of machine learning model pipelines, or combination thereof; or
selecting or interacting with one or more nodes for of one or more of a plurality of metadata rings within the interactive visualization GUI, wherein the plurality of metadata rings include transformer rings, estimators rinds, data rings, and composition modules rings.
