Update and query of a large collection of files that represent a single dataset stored on a blob store

A method for updating data, comprising:

• receiving a table indication of a table, wherein the table comprises data and metadata, wherein the metadata includes a list of files, file partitioning information, and a transaction log comprising a plurality of transaction files, wherein each of the transaction files is associated with one or more files, wherein each of the plurality of transaction files is associated with a corresponding transaction log identifier;
• receiving a transaction indication to perform a transaction;
• determining a current position N in the transaction log, wherein the current position N is associated with a first transaction file that has a first transaction log identifier, wherein N is an integer;
• determining a current state of the metadata;
• determining a read set associated with the transaction, wherein the read set associated with the transaction is associated with a first set of files;
• attempting to write an update to the transaction log associated with a next position N+1, wherein the next position N+1 is associated with a second transaction file that has a second transaction log identifier, wherein the second transaction log identifier is a next sequence transaction log identifier after the first transaction log identifier;
• in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists:
• determining a set of updated files associated with the simultaneous transaction;
• determining whether the set of updated files associated with the simultaneous transaction overlap with the first set of files included in the read set associated with the transaction;
• and
• in response to a non-overlap determination that there is not an overlap between the read set associated with the transaction comprised of the first set of files and the set of updated files associated with the simultaneous transaction, attempting to write the update to the transaction log associated with a further position N+2.

A computer program product for updating data, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

• receiving a table indication of a table, wherein the table comprises data and metadata, wherein the metadata includes a list of files, file partitioning information, and a transaction log comprising a plurality of transaction files, wherein each of the transaction files is associated with one or more files, wherein each of the plurality of transaction files is associated with a corresponding transaction log identifier;
• receiving a transaction indication to perform a transaction;
• determining a current position N in the transaction log, wherein the current position N is associated with a first transaction file that has a first transaction log identifier, wherein N is an integer;
• determining a current state of the metadata;
• determining a read set associated with the transaction, wherein the read set associated with the transaction is associated with a first set of files;
• attempting to write an update to the transaction log associated with a next position N+1, wherein the next position N+1 is associated with a second transaction file that has a second transaction log identifier, wherein the second transaction log identifier is a next sequence transaction log identifier after the first transaction log identifier;
• in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists:
• determining a set of updated files associated with the simultaneous transaction;
• determining whether the set of updated files associated with the simultaneous transaction overlap with the first set of files included in the read set associated with the transaction;
• and
• in response to a non-overlap determination that there is not an overlap between the read set associated with the transaction comprised of the first set of files and the set of updated files associated with the simultaneous transaction, attempting to write the update to the transaction log associated with a further position N+2.