System and method for verifying the integrity and completeness of records

A computer program product comprising a computer storage medium having a plurality of executable instruction codes, for verifying the correctness of query results, comprising:

• a set of instruction codes in the computer storage medium, for receiving a query at a data repository;
• the data repository comprising a data retrieval system for executing the received query, wherein
• the data retrieval system comprises a query system and a trusted domain, wherein the trusted domain is a tamper-resistant secure sub-system,
• the trusted domain further comprises a storage system and a certification agent for providing verification information for retrieved data records, wherein:
• the storage system comprises index segments and data records, and
• the index segments comprise a metadata structure;
• a set of instructions for generating cryptographic signatures for use by the certification agent including a private key and a public key;
• a set of instruction codes for executing the received query using a query logic;
• a set of instruction codes for producing a query output when the cryptographic signatures are verified by the certification agent that includes a query result and the verification information associated with the query result, wherein
• the query result includes the received query and data records that satisfy the received query including data verification objects associated with the data records and, wherein
• the query result further includes index verification objects including index pages associated with the index segments and the received query, wherein the index verification objects comprise a secure one-way hash of one of the index pages and a timestamp signed by the certification agent;
• a set of instruction codes for transferring the query output to a verification system;
• a set of instruction codes for verifying the integrity and completeness of the query result using the verification information included in the query output;
• a set of instruction codes for verifying the integrity and completeness of the query result by executing an equivalent query logic on the query specified in the query output;
• a set of instruction codes for identifying migrated data and verifying the correctness and completeness of the migrated data by requiring additional verification including a signed Merkle tree root;
• and
• a set of instruction codes for generating Merkle tree authentication data structures for generating data verification objects to verify the integrity of the data records in the query result, by means of the index pages and the associated index verification objects to verify the completeness of the query result;
• and
• a set of instruction codes for implementing batch processing for batches of data records comprising:
• ingesting data records in a batch and continuing to ingest records until the batch is complete;
• generating a data verification object for the batch;
• generating an index for ingested data records accumulated in the batch;
• and
• generating an index verification object for the index for ingested data records accumulated in the batch.

A processor-implemented system for verifying the correctness of query results, comprising:

• a data repository including a computer;
• a query client system for receiving a query and retrieving data records from the data repository by sending data requests to the data repository, including receiving batches of the data records;
• a storage system for including an index of metadata structures including index segments for ingesting data objects and housing the data records and index segments in the data repository;
• a data retrieval system for executing the query received from the query client system to retrieve the data records from the storage system,
• the data retrieval system further including a certification agent residing in a trusted domain,
• the trusted domain serving as a tamper-resistant subsystem of the data repository,
• the certification agent comprising a secret private key, a published public key, and a logic for generating cryptographic signatures, the data retrieval system employing the certification agent for generating a data verification object for respective data objects ingested in the storage system and an index verification object for each index segment, wherein the index verification object comprises a secure one-way hash of one of the index segments and a timestamp signed by the certification agent, and the location of one of the index segments within the index verification object;
• the query client system for retrieving the data records retrieved by the data retrieval system and identifying index segments associated with the data records;
• the query client system for producing a query output that includes the received query, a query result and verification information associated with the query result wherein the query result includes the data records, the data verification objects, and the index verification objects that satisfy the received query;
• a data verification system including a verification agent for receiving the query output and verifying the integrity and completeness of the query result by employing the certification agent to verify the data records produced by the query execution match the query result included in the query output and verify the integrity of the data records using the data verification objects including the secret private key and the published public key and the index verification objects, and employing the certification agent to verify batches of objects, and identifying migrated data and verifying the correctness and completeness of the migrated data by requiring additional verification including a signed Merkle tree root.