# US 11,030,241 B2 · Query usage based organization for very large databases > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,030,241 B2 - Document ID: US11030241B2 - Status: Granted - Kind: B2 - Application number: 15/452,072 - Filed: 2017-03-07 - Published: 2018-09-13 - Granted: 2021-06-08 - Earliest claimed: 2017-03-07 - Assignees: Imperva, Inc. - Applicants: jSonar Inc. - Inventors: Ron Ben-Natan - Web page: https://patentgrep.com/patent/US11030241B2 ## CPC - G06F 16/285: {Clustering or classification} - G06F 16/81: Indexing, e.g. XML tags; Data structures therefor; Storage structures - G06F 9/45504: {Abstract machines for program code execution, e.g. Java virtual machine [JVM], interpreters or emulators} - G06F 16/24544: {Join order optimisation} - G06F 16/258: {Data format conversion from or to a database} - G06F 16/278: {Data partitioning, e.g. horizontal or vertical partitioning} - G06F 16/84: Mapping; Conversion - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A query server identifies data collections of interest in a cloud store, and categorizes the collections based on an intended usage. Depending on the intended usage, the categorized data may be cataloged, indexed, or undergo a full intake into a column store. In a database of large data collections, some collections may experience sparse or indefinite usage. Cataloging or indexing position the collections for subsequent query access, but defers the computational burden. The full intake performs a columnar shredding of the collection for facilitating eminent and regular query access. Upon invocation of query activity, an instantiation of virtual machines provided by the cloud store vendor implements query logic, such that the VMs launch in conjunction with the cloud store having the collections. Collections therefore incur processing based on their expected usage-full intake for high query traffic collections, and reduced cataloging for maintaining accessibility of collections of indefinite query interest. ## Claims ### Claim 1 A method for defining large databases, comprising: receiving an identification of a collection of data in a cloud store, the cloud store accessible via a public access network for storing data in a location independent manner; determining, based on an expected usage of the collection, whether to intake the collection into an object store, the object store having a predefined format under the control of the user; selecting, if the collection remains in the cloud store, a catalog organization for the collection, the catalog organization defining a structure for accessing the data in the manner called for by the expected usage, the expected usage defining a computational intensity and storage demand of a query directed to the collection; and generating the catalog organization by associating elements of the collection with a created entry in the catalog organization, wherein selecting the catalog organization includes: determining an intended query usage and: generating a name and source, if the query usage is not foreseeable; generating an index on a subset of fields, if a keyword lookup query is expected; and generating a columnar organization of the data, if data analytics are expected. ### Claim 2 (depends on 1) The method of claim 1, further comprising: performing, if the expected usage involves a transformation of the collection from the cloud store, the intake on the collection, the intake transferring the identified collection from the cloud store to the object store, the object store having a columnar format. ### Claim 3 (depends on 1) The method of claim 1 wherein a workload defines the expected use, the workload based on a likelihood of an eminent query of the collection and whether the query includes a lookup or online analytical processing (OLAP). ### Claim 4 (depends on 1) The method of claim 1 wherein the expected usage indicates an indefinite need for use of the collection and the intake defines acceptable delays in successive query responses based on the collection, and the catalog organization includes a file name and source location of the collection. ### Claim 5 (depends on 1) The method of claim 1 wherein the expected usage is for individual field value based lookup and the catalog organization includes an index creation on the individual fields available for query. ### Claim 6 (depends on 1) The method of claim 1 wherein the expected usage is for analytical queries and the catalog organization includes a columnar arrangement of named fields in the collection. ### Claim 7 (depends on 1) The method of claim 1 wherein the catalog organization further includes a defer flag indicative of whether to defer creation of the catalog organization until receipt of a query request. ### Claim 8 (depends on 1) The method of claim 1 further comprising: receiving a query request directed to the collection, the collection residing on the cloud store; partitioning the collection into a plurality of partitions, each partition containing a subset of elements from the collection; allocating a plurality of virtual machines (VMs) in a virtualization environment, each VM assigned to a partition of the plurality of partitions; and performing at least a portion of the query on each VM until limited by dependencies between the partitions. ### Claim 9 (depends on 8) The method of claim 8, further comprising: receiving a query and an identifier of a collection; determining the catalog organization of the collection; and transmitting the query, an indication of the catalog organization, and a set of instructions for performing the query to each of the plurality of VMs for performing the query. ### Claim 10 (depends on 9) The method of claim 9 wherein the collection further comprises a plurality of collections, and performing the query includes performing a join on at least two of the collections. ### Claim 11 (depends on 8) The method of claim 8 wherein each VM has a computing speed and a bandwidth, further comprising determining the plurality of VMs based on a cost of each VM and a number of VMs for accommodating the partitions. ### Claim 12 (depends on 1) The method of claim 1 wherein the columnar format stores values based on named fields.