Consistency checking for distributed analytical database systems

A system comprising:

• a processor;
• and
• a memory that stores program code structured to cause the processor to:
• an event record obtainer configured to obtain an ordered sequence of event records, each event record in the ordered sequence including an operation type identifying a type of operation represented by the event record, a set of input data file identifiers identifying data files acted on by the particular operation, and a set of output data file identifiers identifying data files generated by the particular operation;
• a data file set determiner configured to perform the following for each first event record in a plurality of first event records in the ordered sequence, in an order specified by the ordered sequence:
• maintain, in a valid data file set, data file identifiers corresponding to data files that should be visible until the currently processed first event record;
• maintain, in an invalid data file set, data file identifiers corresponding to data files that should not be visible;
• determine whether the operation associated with the first event record was successfully committed;
• and
• in response to determining that the operation associated with the first event record was successfully committed, move any data file identifier in the set of input data file identifiers of the first event record from the valid data file set to the invalid data file set;
• and
• a consistency checker configured to perform the following for a second event record in the ordered sequence that follows at least one of the first event records in the ordered sequence:
• determine that a data inconsistency exists based on at least one of the valid data file set or the invalid data file set, and at least one of the set of input data file identifiers associated with the second event record or the set of output data file identifiers associated with the second event record.

A computer-readable storage medium comprising program instructions that, when executed by a processor, cause the processor to:

• receive an ordered sequence of event records, each event record in the ordered sequence including an operation type identifying a type of operation represented by the event record, a set of input data file identifiers identifying data files acted on by the particular operation, and a set of output data file identifiers identifying data files generated by the particular operation;
• process each first event record in a plurality of first event records in the ordered sequence, in an order specified by the ordered sequence, by:
• maintain, in a valid data file set, data file identifiers corresponding to data files that should be visible until the currently processed first event record;
• maintain, in an invalid data file set, data file identifiers corresponding to data files that should not be visible;
• determine whether the operation associated with the first event record was successfully committed;
• and
• in response to determining that the operation associated with the first event record was successfully committed, move any data file identifier in the set of input data file identifiers of the first event record from the valid data file set to the invalid data file set;
• and
• perform for a second event record in the ordered sequence that follows at least one of the first event records in the ordered sequence:
• determine that a data inconsistency exists based on at least one of the valid data file set or the invalid data file set, and at least one of the set of input data file identifiers associated with the second event record or the set of output data file identifiers associated with the second event record.