Method for efficient derived data views using byte addressable persistent storage
AssigneeInternational Business Machines Corporation
InventorsDanny Harnik, Moshik Hershcovitch, Ronen Kat, Yaron Weinsberg
Life of this patent
Estimating the term…
Abstract
A system to reduce the amount of storage and memory used to maintain derived datasets is disclosed. The system operates by using pointers to the underlying data in persistent, byte-addressable storage media. The system additionally reduces the creation time of the views when storage class memory (SCM) is the underlying storage. Furthermore, the invention relates to a form of compression that is tailored to the use cases of big data analytics. The processes of this disclosure use random access to significantly improve performance.
Claims20
An apparatus comprising:
- a persistent, byte-addressable storage medium comprising storage-class memory, the storage medium storing a first data set;
- and
- a view generating controller operatively connected to the storage medium and configured to:
- access the first data set and generate a new view of the first dataset, the view comprising a first set of pointers to real data for which the view is generated in the first data set, the pointers based on offsets to the real data in the first dataset, and
- provide the first set of pointers to the real data as the new view.
A system comprising:
- a persistent, byte-addressable storage device comprising at least one data set;
- a view generating controller connected to the storage medium and configured to:
- access the at least one data set,
- perform data transformation on the at least one data set,
- generate a set of pointers referencing real data for which a view is generated in the at least one data set, and
- compose the set of pointers to the real data to form the view;
- and
- an analytics engine connected to the storage medium and configured to retrieve a data value by dereferencing a pointer in the set of pointers to obtain the real data.
A method comprising:
- extracting a data set from a persistent, byte-addressable storage medium;
- generating a set of pointers referencing real data for which a view is generated in the data set, wherein the set of pointers comprises a relative offset to the real data and a length of fields;
- creating a view comprising the set of pointers to the real data;
- and
- providing the view to an analytics engine.
Dependencies are inferred from the claim text.