Messaging dedpulication in publish / subscribe system

A non-transitory computer-readable medium comprising stored instructions that when executed by one or more processors of one or more computing devices, cause the one or more computing devices to:

• fetch one or more sets of data records from a data source, each of the data records having a message identifier that uniquely identifies the data record;
• store the data records in respective rows of a data file, wherein the data file is associated with row-level metadata, the row-level metadata comprising, for each of the data records, the message identifier, a file path and a row number;
• determine whether one or more of the data records are duplicates based on respective message identifiers in the row-level metadata;
• generate, in response to determining that the one or more of the data records are duplicates, a second metadata comprising, for each duplicate data record, the file path and the row number associated with the duplicate data records record;
• group the second metadata based on file paths associated with the duplicate data records;
• generate a deletion vector using the grouped second metadata based on row numbers associated with the duplicate data records, wherein the deletion vector comprises a plurality of elements, each element corresponding to a data record indicating whether the corresponding data record in the data file is a duplicate;
• and
• skip the duplicate data records using the generated deletion vector when performing a query operation on the stored data records.

A system comprising:

• one or more computer processors;
• and
• one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the system to:
• fetch one or more sets of data records from a data source, each of the data records having a message identifier that uniquely identifies the data record;
• store the data records in respective rows of a data file, wherein the data file is associated with row-level metadata, the row-level metadata comprising, for each of the data records, the message identifier, a file path and a row number;
• determine whether one or more of the data records are duplicates based on respective message identifiers in the row-level metadata;
• generate, in response to determining that the one or more of the data records are duplicates, a second metadata comprising, for each duplicate data record, the file path and the row number associated with the duplicate data record;
• group the second metadata based on file paths associated with the duplicated duplicate data records;
• generate a deletion vector using the grouped second metadata based on row numbers associated with the duplicate data records, wherein the deletion vector comprises a plurality of elements, each element corresponding to a data record indicating whether the corresponding data record in the data file is a duplicate;
• and
• skip the duplicate data records using the generated deletion vector when performing a query operation on the stored data records.