# US 2024/0256539 A1 · Static approach to lazy materialization in database scans using pushed filters

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0256539 A1
- Document ID: US20240256539A1
- Status: Published application
- Kind: A1
- Application number: 18/160,850
- Filed: 2023-01-27
- Published: 2024-08-01
- Earliest claimed: 2023-01-27
- Assignees: Databricks, Inc.
- Applicants: Databricks, Inc.
- Inventors: Shoumik Palkar; Alexander Behm; Mostafa Mokhtar; Sriram Krishnamurthy
- Web page: https://patentgrep.com/patent/US20240256539A1

## CPC

- G06F 16/24539: {using cached or materialised query results}
- G06F 16/221: {Column-oriented storage; Management thereof}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Disclosed herein is a method for determining whether to apply a lazy materialization technique to a query run. The method includes receiving a request to perform a new query in a columnar database containing a plurality of columns. A step in the method includes accessing a set of data in a column of the plurality of columns based on the query. The method includes generating an input to a machine-learned model comprising characteristics of the set of data in the column. From the machine-learned model, the method includes generating a likelihood value indicative of whether a filter of a first portion of the set of data in the column has greater efficiency than a download followed by a filter of the set of data in the column. The method further includes comparing the likelihood value to a threshold value. Based on the comparison, the method includes filtering the first portion of the set of data before downloading the set of data if the likelihood value is equal to or above the threshold value.

## Claims

### Claim 1

A method comprising:
receiving a request to perform a new query in a columnar database having a plurality of columns;
accessing a set of data based on the query, the set of data characterizing the query and the plurality of columns;
generating an input to a machine-learned model based on the set of data;
generating a likelihood value from the machine-learned model based on the input, the likelihood value indicating a likelihood of gaining a performance benefit by applying a lazy materialization technique to a first task of the query;
comparing the likelihood value to a threshold value; and
applying the lazy materialization technique to the first task based on the likelihood value meeting or exceeding a threshold value.

### Claim 2 (depends on 1)

The method of claim 1, wherein the machine-learned model is trained on a dataset of previous queries and a known run time for downloading columns of the previous queries.

### Claim 3 (depends on 1)

The method of claim 1, wherein the machine-learned model is trained on a dataset of data characteristics including at least one of: a number of files per task, a number of requests per row, a number of requests per column, and a size of each column.

### Claim 4 (depends on 1)

The method of claim 1, wherein the likelihood value corresponds to an estimated run time for applying the lazy materialization technique to the first task.

### Claim 5 (depends on 1)

The method of claim 1, wherein the likelihood value is determined, in part, based on a size of a dictionary encoding of the column.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
generating filter results for the first task;
applying an adaptive approach to lazy materialization comprising:
determining, based on the filter results for the first task, a likelihood value indicating a likelihood of gaining a performance benefit by applying the lazy materialization technique to a second task of the query; and,
applying the lazy materialization technique to the second task based on the likelihood value meeting or exceeding a threshold value.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
applying an eager approach to the second task of the query based on the likelihood value being below a threshold value.

### Claim 8

A system comprising:
at least one processor;
at least one memory comprising stored instructions, the instructions when executed cause the at least one processor to:
receive a request to perform a new query in a columnar database having a plurality of columns;
access a set of data in a column of the plurality of columns based on the query;
generate an input to a machine-learned model comprising characteristics of the set of data in the column;
generate a likelihood value from the machine-learned model, the likelihood value indicative of whether a filter of a first portion of the set of data in the column has greater efficiency than a download followed by a filter of the set of data in the column;
compare the likelihood value to a threshold value; and
filter the first portion of the set of data before downloading the set of data if the likelihood value is equal to or above the threshold value.

### Claim 9 (depends on 8)

The system of claim 8, wherein the machine-learned model is trained on a dataset of previous queries and a known run time for downloading columns of the previous queries.

### Claim 10 (depends on 8)

The system of claim 8, wherein the machine-learned model is trained on a dataset of data characteristics including at least one of: a number of files per task, a number of requests per row, a number of requests per column, and a size of each column.

### Claim 11 (depends on 8)

The system of claim 8, wherein the likelihood value corresponds to an estimated run time for applying the lazy materialization technique to the first task.

### Claim 12 (depends on 8)

The system of claim 8, wherein the likelihood value is determined, in part, based on a size of a dictionary encoding of the column.

### Claim 13 (depends on 8)

The system of claim 8, wherein the at least one memory further comprises instructions that when executed further cause the at least one processor to:
generate filter results for the first task;
apply an adaptive approach to lazy materialization comprising:
determine, based on the filter results for the first task, a likelihood value indicating a likelihood of gaining a performance benefit by applying the lazy materialization technique to a second task of the query; and,
apply the lazy materialization technique to the second task based on the likelihood value meeting or exceeding a threshold value.

### Claim 14 (depends on 13)

The system of claim 13, wherein the at least one memory further comprises instructions that when executed causes the at least one processor to:
apply an eager approach to the second task of the query based on the likelihood value being below a threshold value.

### Claim 15

A non-transitory computer readable medium comprising stored instructions encoded thereon that, when executed by a processor, cause at least one processor to:
receive a request to perform a new query in a columnar database having a plurality of columns;
access a set of data in a column of the plurality of columns based on the query;
generate an input to a machine-learned model comprising characteristics of the set of data in the column;
generate a likelihood value from the machine-learned model, the likelihood value indicative of whether a filter of a first portion of the set of data in the column has greater efficiency than a download followed by a filter of the set of data in the column;
compare the likelihood value to a threshold value; and
filter the first portion of the set of data before downloading the set of data if the likelihood value is equal to or above the threshold value.

### Claim 16 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the machine-learned model is trained on a dataset of previous queries and a known run time for downloading columns of the previous queries.

### Claim 17 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the machine-learned model is trained on a dataset of data characteristics including at least one of: a number of files per task, a number of requests per row, a number of requests per column, and a size of each column.

### Claim 18 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the likelihood value corresponds to an estimated run time based application of the lazy materialization technique to the first task.

### Claim 19 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the likelihood value is determined, in part, based on a size of a dictionary encoding of the column.

### Claim 20 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the stored instructions further comprises stored instructions that when executed further causes the at least one processor to:
generate filter results for the first task;
apply an adaptive approach to lazy materialization that further comprises stored instructions to cause the at least one processor to:
determine, based on the filter results for the first task, a likelihood value indicating a likelihood of gaining a performance benefit by applying the lazy materialization technique to a second task of the query; and,
apply the lazy materialization technique to the second task based on the likelihood value meeting or exceeding a threshold value.
