# US 12,118,406 B2 · Interactive augmented reality based optimization of machine learning model execution on hybrid cloud

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,406 B2
- Document ID: US12118406B2
- Status: Granted
- Kind: B2
- Application number: 17/443,983
- Filed: 2021-07-29
- Published: 2023-02-02
- Granted: 2024-10-15
- Earliest claimed: 2021-07-29
- Assignees: Kyndryl, Inc.
- Applicants: Kyndryl, Inc.
- Inventors: Manjit Singh Sodhi; Prerna Agarwal; Shridhar Gurunath Domanal; Sri Harsha Varada
- Web page: https://patentgrep.com/patent/US12118406B2

## CPC

- G06F 9/5072: {Grid computing}
- H04L 41/16: using machine learning or artificial intelligence
- G06N 20/00: Machine learning
- G06Q 10/101: Collaborative creation, e.g. joint development of products or services
- G06Q 30/04: Billing or invoicing
- G06T 11/00: Two-dimensional [2D] image generation
- G06F 2209/501: Performance criteria
- G06F 2209/503: Resource availability
- G06F 2209/508: Monitor
- H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

According to one embodiment, a method, computer system, and computer program product for cloud service brokerage. The embodiment may include receiving a data set and user defined contextual parameters relating to a machine learning (ML) problem of a user to be performed on the data set. The embodiment may include identifying a resource requirement of the ML problem and available resources. The embodiment may include enabling user configuration of the contextual parameters in an interactive augmented reality (AR) view. The embodiment may include identifying a set of clusters upon which to execute computing tasks of the ML problem. The set of clusters is identified out of the available resources. The embodiment may include implementing a ML evaluation process to determine an optimized load distribution model for execution of the computing tasks within the set of clusters. The embodiment may include implementing the optimized load distribution model.

## Claims

### Claim 1

A computer-based method for a cloud service brokerage, the method comprising:
receiving a data set and user defined contextual parameters relating to a machine learning (ML) problem of a user to be performed on the data set;
identifying a resource requirement of the ML problem and available resources of the user;
enabling a user configuration of the user defined contextual parameters in an interactive augmented reality (AR) view;
identifying a set of clusters for executing computing tasks of the ML problem, wherein the set of clusters is identified out of the available resources;
defining data pre-processing and feature engineering definitions for the data set based on associated metadata of the data set;
in response to identifying available memory space within individual clusters of the set of clusters, partitioning the data set across the available memory space within the individual clusters of the set of clusters such that a single cluster stores a single partition;
implementing a ML evaluation process to determine an optimized load distribution model for executing the computing tasks within the set of clusters, wherein the ML evaluation process comprises distributing the computing tasks across the set of clusters; and
executing the computing tasks to optimize the load distribution model.

### Claim 2 (depends on 1)

The method of claim 1, wherein the ML evaluation process further comprises:
applying the defined data pre-processing and feature engineering definitions within the individual clusters; and
combining computing task execution results from the clusters to derive final model parameters for the optimized load distribution model.

### Claim 3 (depends on 1)

The method of claim 1, wherein the data set comprises:
associated metadata, and
wherein the user defined contextual parameters are selected from a group consisting of:
one or more parameters relating to a time constraint for the executing of the computing tasks of the ML problem,
a monetary cost constraint for performing the ML problem,
computing resources available to solve the ML problem,
preferred geographic locations for executing the ML problem,
desired computing resource security standards for executing the ML problem,
preferred cloud service providers,
preferred weightages on cloud service provider utilization,
excluded computing resources, and
computing performance metrics of computing resources.

### Claim 4 (depends on 1)

The method of claim 1, wherein the resource requirement is based, at least, on the ML problem to be performed on the data set and a size of the data set, and
wherein the available resources comprise:
one or more cloud service providers for which the user has an identifiable account or one or more on-premises computing devices.

### Claim 5 (depends on 1)

The method of claim 1, wherein the interactive AR view comprises a virtual representation of the ML problem, and
wherein the interactive AR view contains an element selected from a group consisting of:
a rendered virtual object representation for the given ML problem,
rendered virtual object representations for computing tasks of the ML problem,
a rendered virtual object representation for the resource requirement,
rendered virtual object representations for user defined contextual parameters, and
rendered virtual object representations for identified available resources.

### Claim 6 (depends on 5)

The method of claim 5, wherein the actions of the user are implemented within the AR view as drag-and-drop operations of the user, hand gestures of the user, and/or natural language input of the user, and
wherein the actions of the user are selected from the group consisting of:
an addition of a contextual parameter,
a deletion of a contextual parameter,
a selection of a contextual parameter,
a deselection of a contextual parameter,
an adjustment of contextual parameters,
an assignment of percentage utilization of identified available resources,
a distribution of computing tasks across identified available resources,
a selection of preferred available resources, and
a selection of available resources to exclude.

### Claim 7 (depends on 5)

The method of claim 5, wherein the interactive AR view comprises:
one or more rendered artificially intelligent (AI) avatars that are trained on the user defined contextual parameters relating to the ML problem and on attributes of the identified available resources, and
wherein the user interacts with the one or more rendered AI avatars within the AR view via text and/or natural language input to implement actions of the user in the AR view.

### Claim 8

A computer system, comprising:
a processor that, when executing instructions stored in an associated memory, is configured to:
receive a data set and user defined contextual parameters relating to a machine learning (ML) problem of a user to be performed on the data set;
identify a resource requirement of the ML problem and available resources of the user;
defining data pre-processing and feature engineering definitions for the data set based on associated metadata of the data set;
in response to identifying available memory space within individual clusters of the set of clusters, partitioning the data set across the available memory space within the individual clusters of the set of clusters such that a single cluster stores a single partition;
enable a user configuration of the user defined contextual parameters in an interactive augmented reality (AR) view;
identify a set of clusters for executing computing tasks of the ML problem, wherein the set of clusters is identified out of the available resources;
implement a ML evaluation process to determine an optimized load distribution model for execution of the computing tasks within the set of clusters, wherein the ML evaluation process comprises distributing the computing tasks across the set of clusters; and
execute the computing tasks to optimize load distribution model.

### Claim 9 (depends on 8)

The computer system of claim 8, wherein the ML evaluation process further comprises:
applying the defined data pre-processing and feature engineering definitions within the individual clusters; and
combining computing task execution results from the clusters to derive final model parameters for the optimized load distribution model.

### Claim 10 (depends on 8)

The computer system of claim 8, wherein the data set comprises:
associated metadata, and
wherein the user defined contextual parameters are selected from a group consisting of:
one or more parameters relating to a time constraint for execution of the ML problem,
a monetary cost constraint for performing the ML problem,
computing resources available to solve the ML problem,
preferred geographic locations for execution of the ML problem,
desired computing resource security standards for execution of the ML problem,
preferred cloud service providers,
preferred weightages on cloud service provider utilization,
excluded computing resources, and
computing performance metrics of computing resources.

### Claim 11 (depends on 8)

The computer system of claim 8, wherein the resource requirement is based, at least, on the ML problem to be performed on the data set and a size of the data set, and
wherein the available resources comprise:
one or more cloud service providers for which the user has an identifiable account or one or more on-premises computing devices.

### Claim 12 (depends on 8)

The computer system of claim 8, wherein the interactive AR view comprises a virtual representation of the ML problem, and
wherein the interactive AR view contains an element selected from a group consisting of:
a rendered virtual object representation for the given ML problem,
rendered virtual object representations for computing tasks of the ML problem,
a rendered virtual object representation for the resource requirement,
rendered virtual object representations for user defined contextual parameters, and
rendered virtual object representations for identified available resources.

### Claim 13 (depends on 12)

The computer system of claim 12, wherein the actions of the user are implemented within the AR view as drag-and-drop operations of the user, hand gestures of the user, and/or natural language input of the user, and
wherein the actions of the user are selected from the group consisting of:
an addition of a contextual parameter,
a deletion of a contextual parameter,
a selection of a contextual parameter,
a deselection of a contextual parameter,
an adjustment of contextual parameters,
an assignment of percentage utilization of identified available resources,
a distribution of computing tasks across identified available resources,
a selection of preferred available resources, and
a selection of available resources to exclude.

### Claim 14 (depends on 12)

The computer system of claim 12, wherein the interactive AR view comprises:
one or more rendered artificially intelligent (AI) avatars that are trained on the user defined contextual parameters relating to the ML problem and on attributes of the identified available resources, and
wherein the user interacts with the one or more rendered AI avatars within the AR view via text and/or natural language input to implement actions of the user in the AR view.

### Claim 15

A computer program product comprising a computer readable storage medium storing instructions that, when executed by a processor, cause the processor to perform:
receiving a data set and user defined contextual parameters relating to a machine learning (ML) problem of a user to be performed on the data set;
identifying a resource requirement of the ML problem and available resources of the user;
enabling a user configuration of the user defined contextual parameters in an interactive augmented reality (AR) view;
identifying a set of clusters for executing computing tasks of the ML problem, wherein the set of clusters is identified out of the available resources;
defining data pre-processing and feature engineering definitions for the data set based on associated metadata of the data set;
in response to identifying available memory space within individual clusters of the set of clusters, partitioning the data set across the available memory space within the individual clusters of the set of clusters such that a single cluster stores a single partition;
implementing an ML evaluation process to determine an optimized load distribution model for executing the computing tasks within the set of clusters, wherein the ML evaluation process comprises distributing the computing tasks across the set of clusters; and
executing the computing tasks to optimize the load distribution model.

### Claim 16 (depends on 15)

The computer program product of claim 15, wherein the distributing the computing tasks across the set of clusters further comprises:
distributing the computing tasks across the set of clusters based on the user defined contextual parameters,
wherein the executing the computing tasks to optimize the load distribution model further comprises:
executing the computing tasks on the partitions within the individual clusters of the set of clusters, and
wherein the implementing the ML evaluation process further comprises:
applying the defined data pre-processing and feature engineering definitions within the individual clusters; and
combining computing task execution results from the set of clusters to derive final model parameters for the optimized load distribution model.

### Claim 17 (depends on 15)

The computer program product of claim 15, wherein the data set comprises associated metadata, and
wherein the user defined contextual parameters are selected from a group consisting of:
one or more parameters relating to a time constraint for the executing of the ML problem,
a monetary cost constraint for performing the ML problem,
computing resources available to solve the ML problem,
preferred geographic locations for executing the ML problem,
desired computing resource security standards for executing the ML problem,
preferred cloud service providers,
preferred weightages on cloud service provider utilization,
excluded computing resources, and
computing performance metrics of computing resources.

### Claim 18 (depends on 15)

The computer program product of claim 15, wherein the resource requirement is based, at least, on the ML problem to be performed on the data set and a size of the data set, and
wherein the available resources comprise:
one or more cloud service providers for which the user has an identifiable account or one or more on-premises computing devices.

### Claim 19 (depends on 15)

The computer program product of claim 15, wherein the interactive AR view comprises a virtual representation of the ML problem, and
wherein the interactive AR view contains an element from a group consisting of:
a rendered virtual object representation for the given ML problem,
rendered virtual object representations for computing tasks of the ML problem,
a rendered virtual object representation for the resource requirement,
rendered virtual object representations for user defined contextual parameters, and
rendered virtual object representations for identified available resources.

### Claim 20 (depends on 19)

The computer program product of claim 19, wherein the actions of the user are implemented within the AR view as drag-and-drop operations of the user, hand gestures of the user, and/or natural language input of the user, and
wherein the actions of the user are selected from the group consisting of:
an addition of a contextual parameter,
a deletion of a contextual parameter,
a selection of a contextual parameter,
a deselection of a contextual parameter,
an adjustment of contextual parameters,
an assignment of percentage utilization of identified available resources,
a distribution of computing tasks across identified available resources,
a selection of preferred available resources, and a selection of available resources to exclude.