Database with single copy records for query execution

A database system comprises:

• pluralities of processing core resources of a plurality of computing device clusters, wherein a computing device cluster of the plurality of computing device clusters includes a plurality of computing devices, wherein a computing device of the plurality of computing devices includes a plurality of nodes, and wherein a node of the plurality of nodes includes a plurality of the pluralities of processing core resources, wherein a set of computing devices of a first computing device cluster of the plurality of computing device clusters is operable to:
• receive data of a dataset from a data source, wherein the dataset includes a plurality of rows of columnar data, wherein the plurality of rows is associated with a plurality of row identifiers, and wherein a first row of the plurality of rows is associated with a first row identifier of the plurality of row identifiers;
• temporarily store the data to produce temporarily stored data, wherein a first data portion of the data is stored within a first computing device of the set of computing device, wherein a plurality of data sub-portions of the first data portion is stored within the plurality of nodes of the first computing device, wherein pluralities of data pages of the plurality of data sub-portions are stored by processing core resources of the pluralities of processing core resources associated with the plurality of nodes, and wherein a first data page of the pluralities of data pages is stored by a first processing core resource of the pluralities of processing core resources;
• create pluralities of page metadata for the pluralities of data pages, wherein a first page metadata of the pluralities of page metadata is regarding the first data page, and wherein the first page metadata includes data regarding the row identifiers of rows of the first data page;
• execute a deduplication process utilizing the pluralities of page metadata to determine whether the temporarily stored data includes a duplicate row of data;
• when the temporarily stored data includes a duplicated row of data:
• delete the duplicate row;
• and
• update a corresponding page metadata of the pluralities of page metadata;
• when the temporarily stored data does not include, or no longer includes, a duplicate row of data, process the temporarily stored data for long-term storage to produce a plurality of long-term storage data segments;
• and
• send the plurality of long-term storage data segments to a second computing device cluster of the plurality of computing device clusters.