Filtering records included in files of a data lakehouse platform based on applying a record identification pipeline

A method for execution by at least one processor of a data storage system, comprising:

• storing a plurality of records of a plurality of tables via a plurality of files in memory resources of an object storage system of the data storage system;
• storing configuration data indicating table metadata, mapping storage of the plurality of records of the plurality of tables via the plurality of files, via a metadata processing system of the data storage system;
• receiving a request from a data processing system indicating filtering parameter data in accordance with data storage communication protocol;
• generating, via the metadata processing system, a record identification pipeline for execution based on the filtering parameter data and the configuration data based on accessing the table metadata;
• generating a filtered row set identifying a proper subset of the plurality of records meeting the filtering parameter data based on executing the record identification pipeline via the metadata processing system by accessing at least one file of the plurality of files in the memory resources of the object storage system;
• and
• sending a response to the data processing system that indicates the filtered row set in accordance with the data storage communication protocol, wherein the data processing system generates a query resultant based on the filtered row set.

A non-transitory computer readable storage medium comprises:

• at least one memory section that stores operational instructions that, when executed by at least one processing module that includes a processor and a memory, causes the at least one processing module to perform operations that include:
• storing a plurality of records of a plurality of tables via a plurality of files in memory resources of an object storage system of a data storage system;
• storing configuration data indicating table metadata, mapping storage of the plurality of records of the plurality of tables via the plurality of files, via a metadata processing system of the data storage system;
• receiving a request from a data processing system indicating filtering parameter data in accordance with data storage communication protocol;
• generating, via the metadata processing system, a record identification pipeline for execution based on the filtering parameter data and the configuration data based on accessing the table metadata;
• generating a filtered row set identifying a proper subset of the plurality of records meeting the filtering parameter data based on executing the record identification pipeline via the metadata processing system by accessing at least one file of the plurality of files in the memory resources of the object storage system;
• and
• sending a response to the data processing system that indicates the filtered row set in accordance with the data storage communication protocol, wherein the data processing system generates a query resultant based on the filtered row set.