# US 2023/0168895 A1 · Automated runtime configuration for dataflows

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0168895 A1
- Document ID: US20230168895A1
- Status: Published application
- Kind: A1
- Application number: 18/159,968
- Filed: 2023-01-26
- Published: 2023-06-01
- Earliest claimed: 2020-03-06
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Abhishek Uday Kumar Shah; Anudeep Sharma; Mark A. Kromer; Jikai Ma
- Web page: https://patentgrep.com/patent/US20230168895A1

## CPC

- G06F 9/328: {for runtime instruction patching}
- G06F 9/4494: {data driven}
- G06F 9/30036: {Instructions to perform operations on packed data, e.g. vector, tile or matrix operations}
- G06F 9/50: Allocation of resources, e.g. of the central processing unit [CPU]
- G06F 9/5005: {to service a request}
- G06F 11/3006: {where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems (multiprogramming arrangements G06F9/46; allocation of resources G06F9/50)}
- G06F 16/164: {File meta data generation}
- G06F 16/254: {Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses}
- G06N 3/09: Supervised learning
- G06N 20/20: Ensemble learning
- G06F 9/5044: {considering hardware capabilities}
- G06F 9/5066: {Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs (mappping at compile time, see G06F8/451)}
- G06F 9/5072: {Grid computing}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems and computer program products are provided for automated runtime configuration for dataflows to automatically select or adapt a runtime environment or resources to a dataflow plan prior to execution. Metadata generated for dataflows indicates dataflow information, such as numbers and types of sources, sinks and operations, and the amount of data being consumed, processed and written. Weighted dataflow plans are created from unweighted dataflow plans based on metadata. Weights that indicate operation complexity or resource consumption are generated for data operations. A runtime environment or resources to execute a dataflow plan is/are selected based on the weighted dataflow and/or a maximum flow. Preferences may be provided to influence weighting and runtime selections.

## Claims

### Claim 1

A method in a server, comprising:
receiving a dataflow plan comprising data operations in a dataflow pipeline;
generating a feature set comprising a plurality of features from the received dataflow plan;
generating, by a machine learning (ML) model, a weighted dataflow plan as a weighted feature set from the generated feature set, the weighted dataflow plan associating a weight with each of the data operations; and
selecting an execution environment to execute the received dataflow plan based at least on the weighted dataflow plan.

### Claim 2 (depends on 1)

The method of claim 1, wherein the weight associated a data operation is selected from a weight range corresponding to an execution complexity range for the data operation in terms of consumption of at least one of computing, memory, storage or network resources.

### Claim 3 (depends on 1)

The method of claim 1, wherein the weight associated a data operation comprises a combined weight or a weight vector indicating at least two of computing, memory, storage, and network resources.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
generating metadata for each of the data operations,
wherein said generating comprises:
generating the weight associated with each of the data operations based at least on the corresponding metadata.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
determining a maximum flow for the weighted dataflow plan,
wherein said selecting comprises:
selecting the execution environment based on the maximum flow.

### Claim 6 (depends on 5)

The method of claim 5, wherein said determining the maximum dataflow comprises:
applying a max-flow min-cut theorem to the weighted dataflow plan.

### Claim 7 (depends on 1)

The method of claim 1, wherein said selecting comprises:
selecting the execution environment based on an indication of a preference related to at least one of execution time to execute the received dataflow plan or execution cost to execute the received dataflow plan.

### Claim 8 (depends on 1)

The method of claim 1, wherein said selecting comprises:
providing feedback to a user interface indicating a plurality of execution environments and associated costs; and
prompting user input to select an execution environment from among the plurality of execution environments to execute the received dataflow plan.

### Claim 9 (depends on 1)

The method of claim 1, wherein said selecting comprises:
providing the weighted feature set to a second ML model; and
selecting the execution environment based at least on an output generated by the second ML model based at least on the provided weighted feature set.

### Claim 10

A system, comprising:
a processor; and
memory that stores program code executable by the processor to perform a method, the method comprising:
receiving a dataflow plan comprising data operations in a dataflow pipeline;
generating a feature set comprising a plurality of features from the received dataflow plan;
generating, by a machine learning (ML) model, a weighted dataflow plan as a weighted feature set from the generated feature set, the weighted dataflow plan associating a weight with each of the data operations; and
selecting an execution environment to execute the received dataflow plan based at least on the weighted dataflow plan.

### Claim 11 (depends on 10)

The system of claim 10, wherein the weight associated a data operation is selected from a weight range corresponding to an execution complexity range for the data operation in terms of consumption of at least one of computing, memory, storage or network resources.

### Claim 12 (depends on 10)

The system of claim 10, wherein the weight associated a data operation comprises a combined weight or a weight vector indicating at least two of computing, memory, storage, and network resources.

### Claim 13 (depends on 10)

The system of claim 10, wherein the method further comprises:
generating metadata for each of the data operations,
wherein said generating comprises:
generating the weight associated with each of the data operations based at least on the corresponding metadata.

### Claim 14 (depends on 10)

The system of claim 10, wherein the method further comprises:
determining a maximum flow for the weighted dataflow plan,
wherein said selecting comprises:
selecting the execution environment based on the maximum flow.

### Claim 15 (depends on 14)

The system of claim 14, wherein said determining the maximum dataflow comprises:
applying a max-flow min-cut theorem to the weighted dataflow plan.

### Claim 16 (depends on 10)

The system of claim 10, wherein said selecting comprises:
selecting the execution environment based on an indication of a preference related to at least one of execution time to execute the received dataflow plan or execution cost to execute the received dataflow plan.

### Claim 17 (depends on 10)

The system of claim 10, wherein said selecting comprises:
providing feedback to a user interface indicating a plurality of execution environments and associated costs; and
prompting user input to select an execution environment from among the plurality of execution environments to execute the received dataflow plan.

### Claim 18 (depends on 10)

The system of claim 10, wherein said selecting comprises:
providing the weighted feature set to a second ML model; and
selecting the execution environment based at least on an output generated by the second ML model based at least on the provided weighted feature set.

### Claim 19

A method in a server, comprising:
receiving a dataflow plan comprising data operations in a dataflow pipeline;
generating, from the received dataflow plan, metadata that comprises information regarding the data operations;
generating, by a machine learning (ML) model, a weighted dataflow plan based at least one the received dataflow plan and the generated metadata, the weighted dataflow plan associating a weight with each of the data operations; and
selecting an execution environment to execute the received dataflow plan based at least on the weighted dataflow plan.

### Claim 20 (depends on 19)

The method of claim 19, wherein said selecting comprises:
determining a maximum flow for the weighted dataflow plan; and
selecting the execution environment based on the maximum flow.
