Data loading engine for complex data loading
AssigneePaypal, Inc.
InventorsPrabin Patodia, Rajendra Bhat
Life of this patent
Estimating the term…
Abstract
The disclosed computer-implemented method may relate to a data loading process that involves retrieving data objects and using data producers and data consumers. The method may include determining a dependency hierarchy for the data objects, the data producers, and the data consumers and generating a graph based on the dependency hierarchy. The graph may include various levels, each having nodes representing the data objects, data producers, and/or data consumers, and links between nodes of different levels may represent data routes. The method may further include executing the data loading process using the data routes from the graph to produce a final set of data objects. Various other methods, systems, and computer-readable media are also disclosed.
Claims20
A system comprising:
- a processor;
- and
- a non-transitory computer-readable medium having stored thereon instructions that are executable by the processor to cause the system to perform operations comprising:
- receiving a request for a plurality of data items;
- identifying one or more data stores related to the request;
- grouping data retrievals for the request based on common data stores;
- identifying at least one data transformer related to the request;
- grouping data transformations for the request based on common data transformers;
- building a graph based on the grouped data retrievals and the grouped data transformations;
- retrieving the plurality of data items based on the graph;
- and
- updating the graph based on a performance associated with the processor.
A non-transitory computer-readable medium having stored thereon instructions that are executable by a processor of a computing system to cause the computing system to perform operations comprising:
- receiving a request for a plurality of data items that includes retrieving one or more data items from one or more data sources and transforming at least one of the one or more data items using at least one data transformer;
- determining which of the one or more data items are to be transformed using the at least one data transformer;
- creating a graph based on batch retrieval of data items from common data sources and batch transformation of data items having common transformations organized into stages based on expected processor utilization;
- executing the graph by executing successive stages to output the requested plurality of data items;
- and
- dynamically reorganizing the stages based on updated processor utilization.
A computer-implemented method comprising:
- determining, for a data loading process that includes retrieving a plurality of data objects using at least one data producer and at least one data consumer to produce a final set of data objects, a dependency hierarchy for the plurality of data objects, the at least one data producer, and the at least one data consumer;
- generating, based on the dependency hierarchy, a graph including a plurality of levels between a root node and a final node corresponding to the final set of data objects, wherein each of the plurality of levels includes one or more nodes representing one or more of the plurality of data objects, the at least one data producer and the at least one data consumer, and links from nodes of one of the plurality of levels to a next level of the plurality of levels represent data routes;
- executing the data loading process based on the data routes defined by the plurality of levels to produce the final set of data objects;
- and
- dynamically updating the graph in response to a performance of the executed data loading process.
Dependencies are inferred from the claim text.