Dataflow optimization for extractions from a data repository
Assigneesalesforce.com, inc.
InventorsZuye Zheng, Amruta Moktali, Suyog Anil Deshpande
Life of this patent
Estimating the term…
Abstract
Dataflow optimization is described for extractions from a data depository. In one example an object-relationship graph of a data extraction definition is traversed in a first pass. The object-relationship graph has a node for each object. The steps from each node of the graph in the first pass is determined. It is determined if any of the determined steps are repeated. A single instance of each repeated step is placed before other steps in a query plan. Data is then extracted in a second pass from the object-oriented database system according to the object-relationship graph by performing the repeated steps first and then reusing the repeated steps in other determined steps.
Claims17
A method performed by a system, having at least a processor and a memory therein to execute instructions, wherein the method comprises:
- operating an object-oriented database system communicatively interfaced with the system and having a plurality of objects stored therein;
- retrieving metadata from the object-oriented database system, wherein the retrieved metadata defines an extraction having a plurality of steps to extract data from the object-oriented database system, including one or more repeated steps;
- performing a first pass to traverse an object-relationship graph according to the defined extraction, wherein the object-relationship graph includes a node for every object represented within the object-relationship graph;
- determining steps from each node of the object-relationship graph in the first pass;
- identifying one or more repeated steps within the defined extraction;
- modifying the defined extraction to reuse the data extracted from earlier instances of repeated steps by placing a single instance of each repeated step before other steps and then reusing the data from the repeated steps in later repetitions of the repeated steps;
- and
- performing a second pass to extract the data from the object-oriented database system during the second pass according to the extraction defined by the object-relationship graph by performing the repeated steps first to generate the data and then reusing the data extracted from earlier instances of the repeated steps in the other determined steps.
Non-transitory computer readable storage media having instructions stored thereupon that, when executed by a processor of a system, the instructions cause the system to perform operations comprising:
- operating an object-oriented database system communicatively interfaced with the system and having a plurality of objects stored therein;
- retrieving metadata from the object-oriented database system, wherein the retrieved metadata defines an extraction having a plurality of steps to extract data from the object-oriented database system, including one or more repeated steps;
- performing a first pass to traverse an object-relationship graph according to the extraction, wherein the object-relationship graph includes a node for every object represented within the object-relationship graph;
- determining steps from each node of the object-relationship graph in the first pass;
- identifying one or more repeated steps within the defined extraction;
- modifying the defined extraction to reuse the data extracted from earlier instances of repeated steps by placing a single instance of each repeated step before other steps and then reusing the data from the repeated steps in later repetitions of the repeated steps;
- and
- performing a second pass to extract the data from the object-oriented database system during the second pass according to the extraction defined by the object-relationship graph by performing the repeated steps first to generate the data and then reusing the data extracted from earlier instances of the repeated steps in the other determined steps.
A system comprising:
- a processor and a memory to execute instructions at the system;
- a communications interface to an object-oriented database system having a plurality of objects stored therein;
- wherein the system is to retrieve metadata from the object-oriented database system, wherein the retrieved metadata defines an extraction having a plurality of steps to extract data from the object-oriented database system, including one or more repeated steps;
- wherein the system is to further perform a first pass to traverse an object-relationship graph according to the defined extraction, wherein the object-relationship graph includes a node for every object represented within the object-relationship graph;
- wherein the system is to determine steps from each node of the object-relationship graph in the first pass;
- wherein the system is to identify one or more repeated steps within the defined extraction;
- wherein the system is to further modify the defined extraction to reuse the data extracted from earlier instances of repeated steps by placing a single instance of each repeated step before other steps and then reusing the data from the repeated steps in later repetitions of the repeated steps;
- and
- wherein the system is to further perform a second pass to extract the data from the object-oriented database system during the second pass according to the extraction defined by the object-relationship graph by performing the repeated steps first to generate the data and then reusing the data extracted from earlier instances of the repeated steps in the other determined steps.
Dependencies are inferred from the claim text.