Back to results
US 9,805,177 B1Granted
Processing large data sets from heterogeneous data sources using federated computing resources
AssigneeAmazon Technologies, Inc.
InventorsDiwakar Chakravarthy, Justin Canfield Crites, Sainath Chowdary Mallidi, Charles Porter Schermerhorn, Patrick Jing Ye
Life of this patent
Estimating the term…
Abstract
Disclosed are various embodiments of systems, methods, and computer programs that can facilitate processing of large data sets from data sources. Examples of the disclosure can allow a client to perform data processing tasks in computing resources that are external to a data source from which the data is retrieved rather than relying upon the computing resources of the data source.
Claims18
A method comprising:
- obtaining a registration of a data source, the registration of the data source comprising metadata specifying how to access the data source;
- obtaining a request to generate resultant data based upon data stored in the data source, the request comprising a query into the data source and identifying a data processing task associated with the data;
- generating a command to access the data source on behalf of a client;
- interpreting the query associated with the request, wherein the query comprises the data processing task;
- identifying a data set from the data source associated with the query;
- transmitting a request to retrieve the data set from the data source instead of the query;
- retrieving data from the data source based at least in part upon the command;
- allocating at least one computing instance in a computing environment external to the data source for the data processing task associated with the data;
- executing the data processing task using the at least one computing instance in the computing environment external to the data source, wherein the query is executed using the at least one computing instance in the computing environment external to the data source instead of using the data source;
- and
- generating the resultant data based at least in part upon the data processing task and the data.
8 more elements
A system comprising:
- at least one computing device;
- and
- a data processing application executable by the at least one computing device, the data processing application, when executed, configured to cause the at least one computing device to at least:
- generate a computing instance in the at least one computing device, the computing instance comprising a virtual machine instance;
- identify a query comprising a data processing task on data stored in a plurality of data sources;
- interpret the query;
- identify a data set from the data sources associated with the query;
- transmit a request to retrieve the data set from the data sources instead of the query;
- generate a respective command to retrieve the data from each of the plurality of data sources;
- initiate loading of the data from each of the plurality of data sources into the computing instance;
- and
- execute the data processing task in the computing instance, wherein the query is executed using the at least one computing instance in the computing instance external to the data source instead of using the data sources.
9 more elements
A non-transitory computer-readable medium embodying a program executable on at least one computing device, the program, when executed, causing the at least one computing device to at least:
- generate a computing instance in the at least one computing device, the computing instance comprising a virtual machine instance;
- identify a query comprising a data processing task on data stored in a plurality of data sources;
- interpret the query;
- identify a data set from the data sources associated with the query;
- transmit a request to retrieve the data set from the data sources instead of the query;
- generate a respective command to retrieve the data from each of the plurality of data sources;
- initiate loading of the data from each of the plurality of data sources into the computing instance;
- and
- execute the data processing task in the computing instance, wherein executing the query using the at least one computing instance in the computing instance external to the data source instead of using the data sources.
6 more elements
Dependencies are inferred from the claim text.