Controlled data distribution in container orchestration
AssigneeInternational Business Machines Corporation
InventorsParitosh RANJAN, Prodip ROY, Bhubaneswar PADHAN, SWARUP KUMAR SIKDER
Life of this patent
Estimating the term…
Abstract
Controlled distribution of workloads among container groups in orchestration services is performed according to an orchestration service description configuration file, which supports validating, formatting, selecting container sets, and distributing formatted input data for processing. Individual container feedback drives ongoing workload distribution to completion. Container feedback includes status reports and data processing statistics. Ongoing workload distribution includes reading portions of data from a target dataset, validating and formatting the portions of data until no more data is available to be read from the target dataset.
Claims18
A computer-implemented method for controlled parallel distribution of a data workload via container orchestration, the method comprising:
- reading input data from a dataset in storage, the input data being a portion of the dataset, the portion determined according to pre-defined capacities of a set of containers including individual containers, the set of containers being assigned to process the dataset;
- distributing, in parallel, batches of the input data to the set of containers, the batches assigned to the individual containers according to configured distribution rules;
- retrieving data processing status reports from the individual containers while processing their respectively assigned batches;
- and
- distributing additional input data to the individual containers based on the retrieved data processing status reports.
A computer program product comprising a computer readable storage medium having a set of instructions stored therein which, when executed by a processor, causes the processor to control parallel distribution of a data workload via container orchestration by:
- reading input data from a dataset in storage, the input data being a portion of the dataset, the portion determined according to pre-defined capacities of a set of containers including individual containers, the set of containers being assigned to process the dataset;
- distributing, in parallel, batches of the input data to the set of containers, the batches assigned to the individual containers according to configured distribution rules;
- retrieving data processing status reports from the individual containers while processing their respective assigned batches;
- and
- distributing additional input data to the individual containers based on the retrieved processing status reports.
A computer system for controlled parallel distribution of a data workload via container orchestration, the computer system comprising:
- a processor set;
- and
- a computer readable storage medium;
- wherein:
- the processor set is structured, located, connected, and/or programmed to run program instructions stored on the computer readable storage medium;
- and
- the program instructions which, when executed by the processor set, cause the processor set to perform a process by:
- reading input data from a dataset in storage, the input data being a portion of the dataset, the portion determined according to pre-defined capacities of a set of containers including individual containers, the set of containers being assigned to process the dataset;
- distributing, in parallel, batches of the input data to the set of containers, the batches assigned to the individual containers according to configured distribution rules;
- retrieving data processing status reports from the individual containers while processing their respective assigned batches;
- and
- distributing additional input data to the individual containers based on the retrieved processing status reports.
Dependencies are inferred from the claim text.