US 12,724,771 B2Granted

Systems and methods for data management and query optimization

AssigneeSentinel Labs Israel Ltd.

InventorSteve Newman

Life of this patent

Estimating the term…

Loading patent…

Abstract

A central node can: receive a query comprising at least one parameter comprising a time range of a dataset stored in a cloud storage system; transmit one or more of the query parameters comprising the time range to a metadata service; receive from the metadata service a list of files related to the query; and assign to each processing node of a plurality of processing nodes a subset of the files. Each processing node can: determine that the subset is not stored on a cache; retrieving the subset not stored on the cache from the cloud storage system; store the retrieved subset in a local memory; scan the subset stored in the local memory for data matching the at least one parameter to generate a subset of query results; and concurrently copy using a separate thread from the scanning, the subset stored in the local memory to the cache.

Claims20

Dependent claims

A computer-implemented method for optimizing query processing in a cloud-based storage system, the computer-implemented method comprising:

receiving, by a computing system from a query source, a query of a dataset stored in a database storage system, wherein the query comprises one or more query parameters, wherein the one or more query parameters comprises a time range;
transmitting, by the computing system to the cloud-based storage system, the time range;
receiving, by the computing system, a list comprising a plurality of files, wherein each file of the plurality of files is selected from the time range;
distributing, by the computing system to a plurality of processing nodes, the plurality of files, wherein each processing node of the plurality of processing nodes is assigned a corresponding subset of the plurality of files, and wherein each processing node of the plurality of processing nodes comprises a local storage cache;
determining, by each of the plurality of processing nodes, whether the corresponding subset of the plurality of files is stored on the local storage cache;
if the corresponding subset of the plurality of files is not stored on the local storage cache:
retrieving, by each of the plurality of processing nodes, the corresponding subset of the plurality of files not stored on the local storage cache;
storing, by each of the plurality of processing nodes, the retrieved corresponding subset of the plurality of files in a memory corresponding to each of the plurality of processing nodes;
scanning, by each of the plurality of processing nodes, the corresponding subset of the plurality of files stored in the memory for data matching one or more query parameters to generate a subset of query results;
copying, concurrently with the scanning and using a separate processing thread from the scanning, the corresponding subset of the plurality of files stored in the memory to the local storage cache;
determining that the scanning is complete;
determining if the copying is complete;
and
if the copying is not complete:
abandoning the copying;
and
discarding the subset of the plurality of files from the memory;
if the corresponding subset of the plurality of files is stored on the local storage cache:
scanning, by each of the plurality of processing nodes, the corresponding subset of the plurality of files stored on the local storage cache for data matching the one or more query parameters to generate the subset of query results;
transmitting, from each of the plurality of processing nodes to the computing system, the subset of query results;
aggregating, by the computing system, each subset of query results from each of the plurality of processing nodes to generate a final query result;
and
transmitting, by the computing system to the query source, the final query result,
wherein the computing system and each of the plurality of processing nodes comprise a processor and computer-readable memory.

21 more elements

A computer system for optimizing query processing in a cloud database storage system, the system comprising:

a computing system comprising a processor and a plurality of processing nodes;
one or more non-transitory computer storage media in communication with the processor and/or the plurality of processing nodes and configured to store computer-executable instructions that when executed by the processor and/or the plurality of processing nodes, cause the processor and/or the plurality of processing nodes to:
receive, by the computing system from a query source, a query of a dataset stored in a database storage system, wherein the query comprises one or more query parameters, wherein the one or more query parameters comprises a time range;
transmit, by the computing system to the cloud-based storage system, the time range;
receive, by the computing system, a list comprising a plurality of files, wherein each file of the plurality of files is selected from the time range;
distribute, by the computing system to a plurality of processing nodes, the plurality of files, wherein each processing node of the plurality of processing nodes is assigned a corresponding subset of the plurality of files, and wherein each processing node of the plurality of processing nodes comprises a local storage cache;
determine, by each of the plurality of processing nodes, whether the corresponding subset of the plurality of files is stored on the local storage cache;
if the corresponding subset of the plurality of files is not stored on the local storage cache:
retrieve, by each of the plurality of processing nodes, the corresponding subset of the plurality of files not stored on the local storage cache;
store, by each of the plurality of processing nodes, the retrieved corresponding subset of the plurality of files in a memory corresponding to each of the plurality of processing nodes;
scan, by each of the plurality of processing nodes, the corresponding subset of the plurality of files stored in the memory for data matching one or more query parameters to generate a subset of query results;
copy, concurrently with the scanning and using a separate processing thread from the scanning, the corresponding subset of the plurality of files stored in the memory to the local storage cache;
determine that the scanning is complete;
determine if the copying is complete;
and
if the copying is not complete:
abandon the copying;
and
discard the subset of the plurality of files from the memory;
if the corresponding subset of the plurality of files is stored on the local storage cache:
scan, by each of the plurality of processing nodes, the corresponding subset of the plurality of files stored on the local storage cache for data matching the one or more query parameters to generate the subset of query results;
transmit, from each of the plurality of processing nodes to the computing system, the subset of query results;
aggregate, by the computing system, each subset of query results from each of the plurality of processing nodes to generate a final query result;
and
transmit, by the computing system to the query source, the final query result.

22 more elements

A non-transitory computer storage media in communication with the processor and/or the plurality of processing nodes and configured to store computer-executable instructions that when executed by the processor and/or the plurality of processing nodes, cause the processor and/or the plurality of processing nodes to:

receive, by the computing system from a query source, a query of a dataset stored in a database storage system, wherein the query comprises one or more query parameters, wherein the one or more query parameters comprises a time range;
transmit, by the computing system to the cloud-based storage system, the time range;
receive, by the computing system, a list comprising a plurality of files, wherein each file of the plurality of files is selected from the time range;
distribute, by the computing system to a plurality of processing nodes, the plurality of files, wherein each processing node of the plurality of processing nodes is assigned a corresponding subset of the plurality of files, and wherein each processing node of the plurality of processing nodes comprises a local storage cache;
determine, by each of the plurality of processing nodes, whether the corresponding subset of the plurality of files is stored on the local storage cache;
if the corresponding subset of the plurality of files is not stored on the local storage cache:
retrieve, by each of the plurality of processing nodes, the corresponding subset of the plurality of files not stored on the local storage cache;
store, by each of the plurality of processing nodes, the retrieved corresponding subset of the plurality of files in a memory corresponding to each of the plurality of processing nodes;
scan, by each of the plurality of processing nodes, the corresponding subset of the plurality of files stored in the memory for data matching one or more query parameters to generate a subset of query results;
copy, concurrently with the scanning and using a separate processing thread from the scanning, the corresponding subset of the plurality of files stored in the memory to the local storage cache;
determine that the scanning is complete;
determine if the copying is complete;
and
if the copying is not complete:
abandon the copying;
and
discard the subset of the plurality of files from the memory;
if the corresponding subset of the plurality of files is stored on the local storage cache:
scan, by each of the plurality of processing nodes, the corresponding subset of the plurality of files stored on the local storage cache for data matching the one or more query parameters to generate the subset of query results;
transmit, from each of the plurality of processing nodes to the computing system, the subset of query results;
aggregate, by the computing system, each subset of query results from each of the plurality of processing nodes to generate a final query result;
and
transmit, by the computing system to the query source, the final query result.

20 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/970,661Filed 5 Dec 2024 · continuation
Priority: 14 Jan 2022
Published: US 2025/0173335 A129 May 2025
Granted: 1 Sep 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification5 codes

G06F 16/24539
Query rewriting; Transformation: using cached or materialised query results
G06F 16/24552
Query execution: database cache management
G06F 16/24561
Of query operations: intermediate data storage techniques for performance improvement
G06F 16/256
Integrating or interfacing systems involving database management systems: in federated or virtual databases

Family

Application63/299,569Us provisional application · filed 14 Jan 2022
Application18/154,559Parent of this continuation · filed 13 Jan 2023
US 2025/0173335 A1Pre-grant publication · published 29 May 2025
US 12,189,623 B2Parent of this continuation
US 2023/0229658 A1Published application
US 12,724,771 B2This patent

From the US filing history. Foreign family members aren’t listed.