Back to results
US 2012/0259875 A1Published application
Key-break and record-loop processing in parallel data transformation
AssigneeInternational Business Machines Corporation
InventorsGEOFFREY M. MCCLEAN, XIAOYAN PU
Life of this patent
Estimating the term…
Abstract
Embodiments of the invention provide a method and apparatus for providing additional functionality to a data processing program. This is achieved by various means, including preprocessing records in a data volume, designating certain records with a key-break, and creating an aggregation structure that user programs may use to store previously-processed records from the data volume.
Claims18
A method of processing data in a pipelined processing system, comprising:
- preprocessing a plurality of records, comprising:
- for each record of the plurality of records:
- determining whether the record is a last record in a group, and
- upon determining the record is the last record in the group, setting a key-break flag for the record, wherein the key-break flag designates the record as the last record in the group;
- reading, in a first stage of the pipelined processing system, a first record from the plurality of records;
- determining, in a second stage of the pipelined processing system, whether the key-break flag has been set for the first record, indicating that the first record is the last record in the group;
- and
- upon determining that the key-break flag is set, performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record.
5 more elements
A computer-readable storage medium containing a program which, when executed, performs an operation, comprising:
- preprocessing a plurality of records, comprising:
- for each record of the plurality of records:
- determining whether the record is a last record in a group, and
- upon determining the record is the last record in the group, setting a key-break flag for the record, wherein the key-break flag designates the record as the last record in the group;
- reading, in a first stage of the pipelined processing system, a first record from the plurality of records;
- determining, in a second stage of the pipelined processing system, whether the key-break flag has been set for the first record, indicating that the first record is the last record in the group;
- and
- upon determining that the key-break flag is set, performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record.
5 more elements
A system, comprising:
- a computer processor;
- and
- a memory containing a program that, when executed on the computer processor, performs an operation for processing data in a pipelined processing system, comprising:
- preprocessing a plurality of records, comprising:
- for each record of the plurality of records:
- determining whether the record is a last record in a group, and
- upon determining the record is the last record in the group, setting a key-break flag for the record, wherein the key-break flag designates the record as the last record in the group;
- reading, in a first stage of the pipelined processing system, a first record from the plurality of records;
- determining, in a second stage of the pipelined processing system, whether the key-break flag has been set for the first record, indicating that the first record is the last record in the group;
- and
- upon determining that the key-break flag is set, performing, in the second stage of the pipelined processing system, a transformation operation on the first record, to create a transformed record.
8 more elements
Dependencies are inferred from the claim text.