Generic parity solution for highly dynamic sources

A system, comprising:

• one or more processors;
• and
• a memory comprising instructions that, when executed by the one or more processors, cause the system to:
• receive, by a first data consuming component in a first data pipeline a plurality of database change records from a publication service, the plurality of database change records indicating changes to an underlying data source;
• update, by the first data consuming component, a secondary data store to include the plurality of database change records;
• select a sample subset of the plurality of database change records based on one or more conditions;
• and
• for each respective database change record in the sample subset that has already been consumed by a second data consuming component in a second data pipeline that received one or more database change records from the publication service and updated a primary data store based on the one or more database change records:
• compare the respective database change record to first data from the primary data store that corresponds to the respective database change record;
• determine an inconsistency between the primary data store and a particular database change record based on the comparing;
• in response to determining the inconsistency, determine whether the second data consuming component has consumed a database change record with a higher offset value than the first data consuming component;
• in response to determining the second data consuming component has consumed a database change record with a higher offset value than the first data consuming component, delay a parity determination for the particular database change record in the sample subset until the first data consuming component has consumed the database change record with the higher offset value;
• and
• in response to determining the second data consuming component has consumed no database change record with a higher offset value than the first data consuming component, determine a parity failure,
• wherein a notification is generated based on determination of the parity failure.

A non-transitory computer readable medium comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to:

• receive, by a first data consuming component in a first data pipeline, a plurality of database change records from a publication service, the plurality of database change records indicating changes to an underlying data source;
• update, by the first data consuming component, a secondary data store based on the plurality of database change records;
• select a sample subset of the plurality of database change records based on one or more conditions;
• and
• for each respective database change record in the sample subset that has already been consumed by a second data consuming component in a second data pipeline that received one or more database change records from the publication service and updated a primary data store based on the one or more database change records:
• compare the respective database change record to first data from the primary data store that corresponds to the respective database change record;
• determine an inconsistency between the primary data store and a particular database change record based on the comparing;
• in response to determining the inconsistency, determine whether the second data consuming component has consumed a database change record with a higher offset value than the first data consuming component;
• in response to determining the second data consuming component has consumed a database change record with a higher offset value than the first data consuming component, delay a parity determination for the particular database change record in the sample subset until the first data consuming component has consumed the database change record with the higher offset value;
• and
• in response to determining the second data consuming component has consumed no database change record with a higher offset value than the first data consuming component, determine a parity failure,
• wherein a notification is generated based on determination of the parity failure.