Accumulating and flushing mutations in a column store
AssigneeCloudera, Inc.
InventorTodd Lipcon
Life of this patent
Estimating the term…
Abstract
Columnar storage provides many performance and space saving benefits for analytic workloads, but previous mechanisms for handling single row update transactions in column stores suffer from poor performance. A columnar data layout facilitates both low-latency random access capabilities together with high-throughput analytical access capabilities, simplifying Hadoop architectures for use cases involving real-time data. In disclosed embodiments, mutations within a single row are executed atomically across columns and do not necessarily include the entirety of a row. This allows for faster updates without the overhead of reading or rewriting larger columns.
Claims16
A method for a distributed database system, the method comprising:
- in response to an insert operation for a database table, storing a new row in a first in-memory store, wherein the first in-memory store is configured to flush to an on-disk store after a flushing criteria is reached;
- prior to the first in-memory store flushing the new row to the on-disk store, storing a separate mutation record linked to the new row that is stored in the first in-memory store, in response to a second mutation operation that is specified for the new row;
- and
- after the flushing criteria is reached, performing:
- modifying the new row stored in the first in-memory store based on applying the separate mutation record to the new row;
- and
- flushing (i) the modified new row to one or more base data files in the on-disk store, and (ii) the separate mutation record associated with the new row to one or more delta files in the on-disk store,
- wherein the first in-memory store and the on-disk store are configured for disjoint row storage such that a given row of the database table exists in one of a row set (RowSet) in the first in-memory store or a RowSet in the on-disk store.
A computing system comprising:
- one or more processors;
- and
- one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
- in response to a first insert operation for a database table, storing a new row in a first in-memory store, wherein the first in-memory store is configured to flush to an on-disk store after a flushing criteria is reached;
- prior to the first in-memory store flushing the new row to the on-disk store, storing a separate mutation record attached to the new row that is stored in the first in-memory store, in response to a second mutation operation that is specified for the new row;
- and
- after the flushing criteria is reached, performing:
- modifying the new row stored in the first in-memory store based on applying the separate mutation record to the new row;
- and
- flushing (i) the modified new row to one or more base data files in the on-disk store, and (ii) the separate mutation record associated with the new row to one or more delta files in the on-disk store,
- wherein the first in-memory store and the on-disk store are configured for disjoint row storage such that a given row of the database table exists in one of a row set (RowSet) in the first in-memory store or a RowSet in the on-disk store.
A non-transitory computer-readable medium storing instructions that, when executed by a computing system, cause the computing system to perform a process comprising:
- in response to a first insert operation for a database table, storing a new row in a first in-memory store, wherein the first in-memory store is configured to flush to an on-disk store after a flushing criteria is reached;
- prior to the first in-memory store flushing the new row to the on-disk store, storing a separate mutation record associated with the new row that is stored in the first in-memory store, in response to a second mutation operation that is specified for the new row;
- and
- after the flushing criteria, performing:
- modifying the new row stored in the first in-memory store based on applying the separate mutation record to the new row;
- and
- flushing (i) the modified new row to one or more base data files in the on-disk store, and (ii) flushing the separate mutation record associated with the new row to one or more delta files in the on-disk store,
- wherein the first in-memory store and the on-disk store are configured for disjoint row storage such that a given row of the database table exists in one of a row set (RowSet) in the first in-memory store or a RowSet in the on-disk store.
Dependencies are inferred from the claim text.