# US 2021/0279062 A1 · Automated runtime configuration for dataflows

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279062 A1
- Document ID: US20210279062A1
- Status: Published application
- Kind: A1
- Application number: 16/811,448
- Filed: 2020-03-06
- Published: 2021-09-09
- Earliest claimed: 2020-03-06
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Abhishek Uday Kumar Shah; Anudeep Sharma; Mark A. Kromer; Jikai Ma
- Web page: https://patentgrep.com/patent/US20210279062A1

## CPC

- G06F 9/328: {for runtime instruction patching}
- G06F 9/4494: {data driven}
- G06F 9/30036: {Instructions to perform operations on packed data, e.g. vector, tile or matrix operations}
- G06F 9/50: Allocation of resources, e.g. of the central processing unit [CPU]
- G06F 9/5005: {to service a request}
- G06F 11/3006: {where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems (multiprogramming arrangements G06F9/46; allocation of resources G06F9/50)}
- G06F 16/164: {File meta data generation}
- G06F 16/254: {Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses}
- G06N 3/09: Supervised learning
- G06N 20/20: Ensemble learning
- G06F 9/5044: {considering hardware capabilities}
- G06F 9/5066: {Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs (mappping at compile time, see G06F8/451)}
- G06F 9/5072: {Grid computing}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems and computer program products are provided for automated runtime configuration for dataflows to automatically select or adapt a runtime environment or resources to a dataflow plan prior to execution. Metadata generated for dataflows indicates dataflow information, such as numbers and types of sources, sinks and operations, and the amount of data being consumed, processed and written. Weighted dataflow plans are created from unweighted dataflow plans based on metadata. Weights that indicate operation complexity or resource consumption are generated for data operations. A runtime environment or resources to execute a dataflow plan is/are selected based on the weighted dataflow and/or a maximum flow. Preferences may be provided to influence weighting and runtime selections.

## Claims

### Claim 1

A system, comprising:
a metadata generator configured to:
receive a dataflow plan comprising data operations in a dataflow pipeline; and
generate metadata for the received dataflow plan comprising information about each of the data operations,
a weight generator configured to:
create a weighted dataflow plan from the received dataflow plan by associating a weight with each of the data operations based on the metadata;
a max flow detector configured to:
determine a maximum dataflow for the weighted dataflow plan; and
a mapper configured to:
select at least one of a runtime environment or one or more runtime resources to execute the received dataflow plan based at least on the maximum dataflow.

### Claim 2 (depends on 1)

The system of claim 1, wherein the weight generator is configured to generate a feature set comprising a plurality of features from the received dataflow plan; and
wherein the weight generator is configured to create the weighted dataflow plan by generating a feature weight set for the feature set based on the metadata.

### Claim 3 (depends on 1)

The system of claim 1, wherein the max flow detector is configured to determine the maximum dataflow based on application of a max-flow min-cut theorem to the weighted dataflow plan.

### Claim 4

A method, comprising:
receiving a dataflow plan comprising data operations in a dataflow pipeline;
creating a weighted dataflow plan from the received dataflow plan by associating a weight with each of the data operations; and
selecting at least one of a runtime environment or one or more runtime resources to execute the received dataflow plan based at least on the weighted dataflow plan.

### Claim 5 (depends on 4)

The method of claim 4, wherein each weight associated with each of the data operations is selected from a weight range corresponding to an execution complexity range for each of the data operations in terms of consumption of at least one of computing, memory, storage and network resources.

### Claim 6 (depends on 5)

The method of claim 5, wherein each weight associated with each of the data operations comprises a combined weight or a weight vector indicating at least two of computing, memory, storage and network resources.

### Claim 7 (depends on 4)

The method of claim 4, further comprising:
generating metadata for the received dataflow plan comprising information about each of the data operations,
wherein the weight associated with each of the data operations is based on the metadata.

### Claim 8 (depends on 4)

The method of claim 4, further comprising:
determining a maximum dataflow for the weighted dataflow plan,
wherein the selection of the runtime environment or the runtime resources is based on the maximum dataflow.

### Claim 9 (depends on 8)

The method of claim 8, wherein determining the maximum dataflow comprises applying a max-flow min-cut theorem to the weighted dataflow plan.

### Claim 10 (depends on 4)

The method of claim 4, further comprising:
selecting the runtime environment or the runtime resources based on an indication of a preference related to at least one of execution time and execution cost to execute the received dataflow plan.

### Claim 11 (depends on 4)

The method of claim 4, further comprising:
providing feedback to a user interface indicating a plurality of runtime environments or a plurality of runtime resource alternatives and their costs in terms of at least one of time and price; and
prompting user input to select from among the plurality of runtime environments or the plurality of runtime resource alternatives to execute the received dataflow plan.

### Claim 12 (depends on 4)

The method of claim 4, wherein the selection of the runtime environment or the runtime resources comprises selecting between at least two of: compute optimized, memory optimized, network optimized, and storage optimized runtime environments or runtime resources.

### Claim 13 (depends on 4)

The method of claim 4, wherein the received dataflow plan comprises a graph with nodes and edges provided by a user.

### Claim 14 (depends on 4)

The method of claim 4, further comprising:
restricting the selection of the runtime environment or the runtime resources based on one or more caps or limits specified in a user agreement.

### Claim 15 (depends on 4)

The method of claim 4, further comprising:
generating a feature set comprising a plurality of features from the received dataflow plan; and
generating, by a machine learning (ML) model, the weighted dataflow plan as a weighted feature set from the feature set.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
providing the feature weight set to a second ML model; and
selecting the runtime environment or the runtime resources based on an output of the second ML model.

### Claim 17 (depends on 15)

The method of claim 15, wherein at least one of the plurality of features is generated based on application of term frequency-inverse document frequency (TF-IDF) to the received dataflow plan.

### Claim 18

A computer-readable storage medium having program instructions recorded thereon that, when executed by a processing circuit, perform a method comprising:
receiving a dataflow plan comprising data operations in a dataflow pipeline;
generating metadata for the received dataflow plan comprising information about each of the data operations;
creating a weighted dataflow plan from the received dataflow plan by associating a weight with each of the data operations based on the metadata; and
selecting at least one of a runtime environment or one or more runtime resources to execute the received dataflow plan based at least on the weighted dataflow plan.

### Claim 19 (depends on 18)

The computer-readable storage medium of claim 18, the method further comprising:
determining a maximum dataflow for the weighted dataflow plan,
wherein the selection of the runtime environment or the runtime resources is based on the maximum dataflow.

### Claim 20 (depends on 19)

The computer-readable storage medium of claim 19, wherein the weighted dataflow plan is created based on the output of a first machine learning (ML) model; or
wherein the runtime environment or the runtime resources are selected based on the output of a second ML model.
