Back to results
US 9,317,553 B2Granted
Declarative partitioning for data collection queries
AssigneeMicrosoft Technology Licensing, LLC
InventorsPuneet Bhatia, Spencer Alan Bishop
Life of this patent
Estimating the term…
Abstract
Aspects of the subject matter described herein relate to data collection. In aspects, when data is desired from one or more data sources, a size and/or other data regarding the data is obtained. Then, a data structure is generated that indicates multiple chunks of the data to obtain. Using the data structure and a query template, a query is generated to obtain a chunk of data. The obtained chunk of data is stored in a local store. If more chunks need to be obtained, first a wait for a period is performed before submitting another query for another chunk of data.
Claims20
A method implemented at least in part by a computer, the method comprising:
- discovering one or more target data sources from which to obtain data;
- generating a data structure that indicates a plurality of chunks of the data to obtain from the one or more target data sources;
- generating a plurality of queries from a template, the plurality of queries including values derived from the data structure to indicate the plurality of respective chunks of the data;
- submitting a first query of the plurality of queries to one or more of the one or more target data sources;
- receiving a response to the first query, the response including a respective chunk of the plurality of chunks;
- storing the chunk in a local data store;
- determining a plurality of periods to wait, each period corresponding to a respective query of the plurality of queries, the determining comprising:
- obtaining a formula from configuration data and using the formula to compute at least one of the plurality of periods;
- in response to the first query completing, waiting until the period that corresponds to the first query expires to submit another query of the plurality of queries, the waiting comprising:
- enabling one or more other queries that are not included in the plurality of queries to be processed;
- and
- performing the following operations for each subsequent query of the plurality of queries until the plurality of chunks is obtained:
- after the period that corresponds to the previous query has expired, submitting the query to obtain a respective chunk of the data from the one or more target data sources;
- receiving a response to the query, the response including a respective chunk of the plurality of chunks;
- storing the chunk in the local data store;
- and
- in response to the query completing, waiting until the period that corresponds to the query expires to submit another query of the plurality of queries, the waiting comprising:
- enabling one or more other queries that are not included in the plurality of queries to be processed,
- each period of the plurality of periods being determined prior to waiting until the respective period expires.
16 more elements
In a computing environment, a system, comprising:
- one or more processors;
- a preprocessor engine implemented using at least one of the one or more processors, the preprocessor engine configured to generate a data structure that indicates a plurality of chunks of data to obtain from one or more target data sources;
- a query manager implemented using at least one of the one or more processors, the query manager configured to generate a plurality of queries from a template, the plurality of queries including values derived from the data structure to indicate the plurality of chunks of the data, the query manager further configured to submit the plurality of queries to the one or more target data sources, a size of each chunk of the plurality of chunks is based on at least one of a total size of the data which is greater than the size of each chunk or one or more performance characteristics of the one or more target data sources to which the plurality of queries is submitted;
- a local store comprising a computer storage medium, the local store having storage elements to store the plurality of chunks received in response to the plurality of queries;
- and
- a persistence manager implemented using at least one of the one or more processors, the persistence manager operable to save the plurality of chunks to the local store.
3 more elements
A computer storage medium having computer-executable instructions, which when executed perform actions, comprising:
- parsing a document that includes declarative statements that indicate data to obtain from target data sources independent of steps for obtaining the data, the document also including a template for a query to use to obtain the data, the document also indicating a period to wait between queries to the target data sources;
- querying the target data sources to discover a size of the data;
- in response to querying the target data sources to discover the size of the data, generating a data structure that indicates chunks of the data to obtain from one or more of the target data sources, a size of the chunks is based on at least one of the size of the data which is greater than the size of the chunks or one or more performance characteristics of at least one of the target data sources;
- generating the query using the template and the data structure;
- submitting the query to the at least one of the target data sources;
- and
- throttling submitting another next query based on the period to wait.
4 more elements
Dependencies are inferred from the claim text.