Back to results
US 10,761,756 B1Granted
Compressing data in line using weighted compression budgets
AssigneeEMC IP Holding Company LLC
InventorsIvan Bassov, Philippe Armangau, Yining Si
Life of this patent
Estimating the term…
Abstract
A technique for performing in-line compression includes receiving data into a data log that temporarily holds the data and aggregating the data into batches, where each batch includes multiple blocks of received data. For each batch of data, a storage system performs a compression operation, which proceeds block-by-block, compressing each block and comparing a total compressed size of all blocks compressed so far against a budget. The storage system increments the budget for successive blocks, such that a per-block budget is greater for a first block in the batch than it is for a last block in the batch, thus allowing earlier blocks to meet budget even if they are relatively incompressible.
Claims16
A method of storing data, the method comprising:
- receiving data into a data log of a data storage system, the data log providing temporary storage for the data in units of blocks;
- aggregating blocks of the data in the data log into multiple batches, each batch including multiple blocks;
- performing a compression operation on each batch, the compression operation including—
- processing a current block in the batch by (i) compressing the current block and (ii) performing a testing operation for the current block, the testing operation configured to (a) produce a first result in response to determining that an accumulated compressed size of all compressed blocks in the batch processed so far exceeds a compression budget and (b) produce a second result in response to determining that the accumulated compressed size of all compressed blocks in the batch processed so far does not exceed the compression budget;
- proceeding to a next block in the batch in response to the testing operation producing the second result for the current block;
- and
- increasing the compression budget when testing later blocks in the batch, such that a per-block value of the compression budget is greater when testing a first block in the batch than the per-block value of the compression budget is when testing a later block in the batch, the compression budget increasing in accordance with K*i+K−K*i/N, wherein K is a constant, i is an index of a current block being processed by the compression operation, and N is the number of blocks in the batch;
- and
- in response to the testing operation producing the second result for all blocks in a particular batch, storing the data of all blocks in the particular batch in compressed form in a set of non-volatile storage devices of the data storage system.
6 more elements
A data storage system, comprising control circuitry that includes a set of processing units coupled to memory, the control circuitry constructed and arranged to:
- receive data into a data log of a data storage system, the data log providing temporary storage for the data in units of blocks;
- aggregate blocks of the data in the data log into multiple batches, each batch including multiple blocks;
- perform a compression operation on each batch, the compression operation configured to—
- process a current block in the batch by (i) compressing the current block and (ii) performing a testing operation for the current block, the testing operation configured to (a) produce a first result in response to determining that an accumulated compressed size of all compressed blocks in the batch processed so far exceeds a compression budget and (b) produce a second result in response to determining that the accumulated compressed size of all compressed blocks in the batch processed so far does not exceed the compression budget;
- proceed to a next block in the batch in response to the testing operation producing the second result for the current block;
- and
- increase the compression budget when testing later blocks in the batch, such that a per-block value of the compression budget is greater when testing a first block in the batch than the per-block value of the compression budget is when testing a later block in the batch, the compression budget increased in accordance with K*i+K−K*i/N, wherein K is a constant, i is an index of a current block being processed by the compression operation, and N is the number of blocks in the batch;
- and
- in response to the testing operation producing the second result for all blocks in a particular batch, store the data of all blocks in the particular batch in compressed form in a set of non-volatile storage devices of the data storage system.
6 more elements
A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a data storage system, cause the control circuitry to perform a method for storing data, the method comprising:
- receiving data into a data log of a data storage system, the data log providing temporary storage for the data in units of blocks;
- aggregating blocks of the data in the data log into multiple batches, each batch including multiple blocks;
- performing a compression operation on each batch, the compression operation including—
- processing a current block in the batch by (i) compressing the current block and (ii) performing a testing operation for the current block, the testing operation configured to (a) produce a first result in response to determining that an accumulated compressed size of all compressed blocks in the batch processed so far exceeds a compression budget and (b) produce a second result in response to determining that the accumulated compressed size of all compressed blocks in the batch processed so far does not exceed the compression budget;
- proceeding to a next block in the batch in response to the testing operation producing the second result for the current block;
- and
- increasing the compression budget when testing later blocks in the batch, such that a per-block value of the compression budget is greater when testing a first block in the batch than the per-block value of the compression budget is when testing a later block in the batch, the compression budget increasing in accordance with K*i+K−K*i/N, wherein K is a constant, i is an index of a current block being processed by the compression operation, and N is the number of blocks in the batch;
- and
- in response to the testing operation producing the second result for all blocks in a particular batch, storing the data of all blocks in the particular batch in compressed form in a set of non-volatile storage devices of the data storage system.
6 more elements
Dependencies are inferred from the claim text.