# US 2024/0202166 A1 · Generating compressed column slabs for storage in a database system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0202166 A1 - Document ID: US20240202166A1 - Status: Published application - Kind: A1 - Application number: 18/520,702 - Filed: 2023-11-28 - Published: 2024-06-20 - Earliest claimed: 2022-12-15 - Assignees: Ocient Holdings LLC - Applicants: Ocient Holdings LLC - Inventors: Greg R. Dhuse; Wesley H. Darvin - Web page: https://patentgrep.com/patent/US20240202166A1 ## CPC - G06F 16/21: Design, administration or maintenance of databases - G06F 16/221: {Column-oriented storage; Management thereof} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A database system is operable to determine a dataset for storage that includes, for each row, a plurality of column values corresponding to a plurality of columns. A plurality of column slab data are generated from the dataset, where each of the plurality of column slab data includes column values for one of the plurality of columns. Each of a plurality of compression dictionaries are trained from a corresponding one of the plurality of column slab data. Each segment of the plurality of segments is generated based on writing a set of compressed column slabs to the each segment based on compressing each of a corresponding set of the plurality of column slab data as a set of variable-length compression frames written to the each segment based on applying a corresponding one of the plurality of compression dictionaries. ## Claims ### Claim 1 A method comprising: determining a dataset for storage that includes, for each of a plurality of rows of the dataset, a plurality of column values corresponding to a plurality of columns of the dataset; generating a plurality of column slab data from the dataset, wherein each of the plurality of column slab data includes column values for one of the plurality of columns from a proper subset of rows of the plurality of rows assigned to one of a plurality of segments; training each of a plurality of compression dictionaries from a corresponding one of the plurality of column slab data; and generating each segment of the plurality of segments based on writing a set of compressed column slabs to the each segment based on compressing each of a corresponding set of the plurality of column slab data as a set of variable-length compression frames written to the each segment based on applying a corresponding one of the plurality of compression dictionaries. ### Claim 2 (depends on 1) The method of claim 1, wherein the each of the plurality of compression dictionaries are trained from a proper subset of column values in the corresponding one of the plurality of column slab data, wherein the proper subset of column values includes one of: a first set of column values from a full set of column values in the corresponding one of the plurality of column slab data; or a randomly selected set of column values from the full set of column values in the corresponding one of the plurality of column slab data. ### Claim 3 (depends on 1) The method of claim 1, wherein each of the set of compressed column slabs is generated to include: a header; the corresponding one of the plurality of compression dictionaries; and compressed data generated based on a compressing one of the plurality of column slab data based on applying the corresponding one of the plurality of compression dictionaries. ### Claim 4 (depends on 1) The method of claim 1, wherein each of the set of variable-length compression frames includes a corresponding subset of a plurality of subsets of the proper subset of rows assigned to the each segment, wherein the plurality of subsets are mutually exclusive and collectively exhaustive with respect to the proper subset. ### Claim 5 (depends on 1) The method of claim 1, wherein generating the each of the plurality of segments is further based on writing a set of compression lookup structures corresponding to the set of compressed column slabs. ### Claim 6 (depends on 5) The method of claim 5, wherein the set of variable-length compression frames are written across a set of fixed-length blocks of the each segment, and wherein each compression lookup structure of the set of compression lookup structures indicates, for each of the set of fixed-length blocks of the corresponding compressed column slab in which a new frame of the set of variable-length compression frames starts: a frame identifier identifying the new frame; a row identifier for identifying a starting row of the new frame; and an offset identifying a starting location of the new frame within the each of the set of fixed-length blocks. ### Claim 7 (depends on 6) The method of claim 6, wherein at least one frame of the set of variable-length compression frames of the each compressed column slab spans more than two blocks of the set of fixed-length blocks, and wherein a corresponding compression lookup structure of the set of compression lookup structures indicates a corresponding at least one of the set of fixed-length blocks of the corresponding compressed column slab is entirely consumed by compressed data of a frame of the at least one frame that started in a prior one of the set of fixed-length blocks based on spanning more than two blocks. ### Claim 8 (depends on 1) The method of claim 1, further comprising; determining compression level parameter data based on the compression level parameter data being configured via user input; and determining dictionary size parameter data based on the dictionary size parameter data being configured via the user input; wherein the plurality of compression dictionaries are trained based on applying the compression level parameter data and the dictionary size parameter data. ### Claim 9 (depends on 1) The method of claim 1, wherein generating the each of the plurality of segments is further based on writing compression metadata to the each segment indicating segment compression data for the each segment; wherein a first corresponding set of the plurality of column slab data of a first segment of the plurality of segments are compressed in accordance with first segment compression data; and wherein a second corresponding set of the plurality of column slab data of a second segment of the plurality of segments are compressed in accordance with second segment compression data that is different from the first segment compression data. ### Claim 10 (depends on 9) The method of claim 9, wherein the first segment compression data is different from the second segment compression data based on at least one of: the first segment compression data denoting compression of a first subset of columns of the plurality of columns, the second segment compression data denoting compression of a second subset of columns of the plurality of columns, wherein the first subset has a non-null set difference with the second subset; the first segment compression data denoting compression of a first number of columns of the plurality of columns, the second segment compression data denoting compression of a second number of columns of the plurality of columns, wherein the first number is different from the second number; or the first segment compression data denoting compression of one of the plurality of columns in accordance with first compression parameters, and the second segment compression data denoting compression of the one of the plurality of columns in accordance with second compression parameters different from the first compression parameters. ### Claim 11 (depends on 1) The method of claim 1, wherein the compressing of each of the corresponding set of the plurality of column slab data is in accordance with a first compression type, and wherein determining the plurality of column slab data includes: generating a set of pre-compressed column data as a subset of the plurality of column slab data by applying a second compression type to column values of at least one of the plurality of columns for rows assigned to at least one segment of the plurality of segments; wherein a corresponding subset of the plurality of compression dictionaries are each trained from a corresponding one of the set of pre-compressed column data, and wherein the at least one of the of a plurality of segments are generated based on writing the set of compressed column slabs to the at least one segment based on further compressing each corresponding one of the set of pre-compressed column data in accordance with the first compression type as the set of variable-length compression frames written to the each segment based on applying the corresponding one of the plurality of compression dictionaries. ### Claim 12 (depends on 11) The method of claim 11, wherein the second compression type is a global dictionary compression type, wherein a same global compression dictionary is utilized to generate the set of pre-compressed column data for the at least one of the plurality of columns for all of the plurality of segments. ### Claim 13 (depends on 12) The method of claim 12, wherein the same global compression dictionary is utilized to generate the pre-compressed column data for multiple ones of the plurality of columns. ### Claim 14 (depends on 1) The method of claim 1, wherein a first set of column slab data is generated for a first column of the plurality of columns storing a first data type, wherein a second set of column slab data is generated for a first column of the plurality of columns storing a second data type, wherein the first data type and the second data type are different data types of a set of data types that includes: at least one fixed-length data type; at least one variable-length data type; at least one array data type; and at least one tuple data type. ### Claim 15 (depends on 1) The method of claim 1, further comprising: determining a query having query predicates indicating a first column compressed as compressed column slabs in a set of segments of the plurality of segments; for each segment of the set of segments, determining row data for rows satisfying the query predicates based on: determining a set of rows of the each segment for access; identifying ones of the set of variable-length compression frames of one compressed column slab written for the first column that include ones of the set of rows; and generating the row data based on reading only the ones of the set of variable-length compression frames identified to include the ones of the set of rows; and generating a query resultant for the query based on processing the row data for all segments of the set of segments. ### Claim 16 (depends on 15) The method of claim 15, further comprising, for the each segment: reproducing column values of the first column based on utilizing the compression dictionary to decompress at least one of the set of variable-length compression frames indicated in the row data generated for the each segment. ### Claim 17 (depends on 16) The method of claim 16, wherein reproducing the column values of the first column is based on decompressing only a portion of one variable-length compression frame of the set of variable-length compression frames, starting from a start of the variable-length compression frame and ending before an end of the variable-length compression frame, based on all ones of the set of rows compressed in the one variable-length compression frame being serially included within the portion of the one variable-length compression frame. ### Claim 18 (depends on 15) The method of claim 15, wherein identifying the ones of the set of variable-length compression frames of the compressed column slab that include ones of the set of rows is based on accessing a compression lookup structure for the compressed column slab mapping row identifiers of the set of rows to corresponding ones of the set of variable-length compression frames, and further mapping memory location data to corresponding ones of the set of variable-length compression frames. ### Claim 19 A method comprising: determining a query having query predicates indicating a first column compressed as compressed column slabs in a set of segments; for each segment of the set of segments, determining row data for rows satisfying the query predicates by processing a corresponding compressed column slab of the each segment corresponding to the first column based on: determining a set of rows of the each segment for access; identifying ones of a set of variable-length compression frames of the compressed column slab written for the first column that include ones of the set of rows based on accessing a lookup structure corresponding to the corresponding compressed column slab; and generating the row data based on reading only the ones of the set of variable-length compression frames identified to include the ones of the set of rows, where the row data is generated based on a compression dictionary corresponding to the compressed column slab; and generating a query resultant for the query based on processing the row data for all segments of the set of segments. ### Claim 20 A database system includes: at least one processor; and at least one memory that stores operational instructions that, when executed by the at least one processor, causes the database system to: determine a dataset for storage that includes, for each of a plurality of rows of the dataset, a plurality of column values corresponding to a plurality of columns of the dataset; generate a plurality of column slab data from the dataset, wherein each of the plurality of column slab data includes column values for one of the plurality of columns from a proper subset of rows of the plurality of rows assigned to one of a plurality of segments; train each of a plurality of compression dictionaries from a corresponding one of the plurality of column slab data; and generate each segment of the plurality of segments based on writing a set of compressed column slabs to the each segment based on compressing each of a corresponding set of the plurality of column slab data as a set of variable-length compression frames written to the each segment based on applying a corresponding one of the plurality of compression dictionaries.