Systems and methods for ensuring quality of search system data

A non-transitory computer readable storage medium including instructions that, when executed by a processor, cause the processor to perform operations for verifying integrity of search system data, the operations comprising:

• accessing a first snapshot and a second snapshot of search system data, wherein the search system data in each snapshot comprises one or more search indexes and transformed data searchable by the one or more search indexes, and the search system data is generated for each snapshot from data ingested by a search system in response to one or more events performed by a commerce platform system, wherein the first snapshot is a validated snapshot for which a validation process has been performed ensuring reliability of data within the first snapshot, and the second snapshot is an unvalidated snapshot for which the validation process has not been performed;
• accessing an operation log of operations performed by the commerce platform system during a time period between when the first snapshot was generated and when the second snapshot was generated, the operation log including data to recreate each event of the one or more events processed by the commerce platform system during the time period between when the first snapshot was generated and when the second snapshot was generated;
• transforming first one or more search indexes and first transformed data from first search system data from the first snapshot by ingestion of the operations in the operation log at the search system and updating the first one or more search indexes and the first transformed data from the first search system data with the ingested operations in the operation log to generate a validated second search system data in a validated second snapshot;
• determining a reliability of second search system data within the second snapshot by comparing the second search system data from the second snapshot with the validated second search system data from the validated second snapshot;
• and
• in response to determining that the reliability does not indicate a match between the second search system data from the second snapshot and the validated second search system data from the validated second snapshot:
• updating a data quality status of the second search system data from the second snapshot indicating that a data quality of the second search system data from the second snapshot is not validated up to a time when the second snapshot was generated;
• tracing an inconsistent search system data entry in the second search system data from the second snapshot based on a first data identifier of a mismatched search system data entry found during comparison of the inconsistent search system data entry in the second snapshot with a corresponding validated search system data entry from the validated second snapshot having a second data identifier, the second data identifier being identical to the first data identifier, wherein the first data identifier and the second data identifier are used within the search system;
• requesting a correction of the mismatched search system data entry in the search system data that corresponds to the inconsistent search system data entry in the second search system data;
• updating and re-indexing the mismatched search system data entry in the second search system data by replacing the mismatched search system data entry with the validated search system data entry;
• in response to updating and re-indexing the mismatched search system data entry, updating the data quality status of the second search system data from the second snapshot indicating that the data quality of the second search system data from the second snapshot is validated up to the time when the second snapshot was generated;
• and
• limiting a search result up to the time when the second snapshot was generated based on the updated validated data quality.

A commerce platform system for verifying integrity of search system data, comprising:

• a memory;
• and
• a processor coupled with the memory configured to:
• access, at a search system, a first snapshot and a second snapshot of the search system data, wherein the search system data in each snapshot comprises one or more search indexes and transformed transaction data searchable by the one or more search indexes, and the search system data is generated for each snapshot from data ingested by the search system in response to transactions one or more events performed by the commerce platform system, wherein the first snapshot is a validated snapshot for which a validation process has been performed ensuring reliability of data within the first snapshot, and the second snapshot is an unvalidated snapshot for which the validation process has not been performed,
• access, at a processing system, an operation log of operations performed by the commerce platform system during a time period between when the first snapshot was generated and when the second snapshot was generated, the operation log including data to recreate each event of the one or more events processed by the commerce platform system during the time period between when the first snapshot was generated and when the second snapshot was generated,
• transform first one or more search indexes and first transformed data from first search system data from the first snapshot by ingestion of the operations in the operation log at the search system and update the first one or more search indexes and the first transformed data from the first search system data with the ingested operations in the operation log to generate a validated second search system data in a validated second snapshot,
• determine a reliability of second search system data within the second snapshot at a second time period by comparing the second search system data from the second snapshot with the validated second search system data from the validated second snapshot, and
• in response to determining that the reliability does not indicate a match between the second search system data from the second snapshot and the validated second search system data from the validated second snapshot:
• update a data quality status of the second search system data from the second snapshot indicating that a data quality of the second search system data from the second snapshot is not validated up to a time when the second snapshot was generated,
• trace an inconsistent search system data entry in the second search system data from the second snapshot based on a first data identifier of a mismatched search system data entry found during comparison of the inconsistent search system data entry in the second snapshot with a corresponding validated search system data entry from the validated second snapshot having a second data identifier, the second identifier being identical to the first data identifier, wherein the first data identifier and the second data identifier are used within the search system,
• request a correction of the mismatched search system data entry in the search system data that corresponds to the inconsistent search system data entry in the second search system data;
• update and re-index the mismatched search system data entry in the second search system data by replacing the mismatched search system data entry with the validated search system data entry;
• in response to updating and re-indexing the mismatched search system data entry, update the data quality status of the second search system data from the second snapshot indicating that the data quality of the second search system data from the second snapshot is validated up to the time when the second snapshot was generated;
• and
• limit a search result up to the time when the second snapshot was generated based on the updated validated data quality.