# US 2018/0322178 A1 · Pseudo-synchronous processing by an analytic query and build cluster > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0322178 A1 - Document ID: US20180322178A1 - Status: Published application - Kind: A1 - Application number: 15/589,728 - Filed: 2017-05-08 - Published: 2018-11-08 - Earliest claimed: 2017-05-08 - Assignees: salesforce.com, inc. - Applicants: salesforce.com, inc. - Inventors: Alex GITELMAN; Ruisheng SHI - Web page: https://patentgrep.com/patent/US20180322178A1 ## CPC - G06F 16/254: {Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses} - G06F 16/2379: {Updates performed during online database operations; commit processing} - G06F 16/24542: {Plan optimisation} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The technology disclosed relates to creating and frequently updating multiple online analytic processing (OLAF) analytic databases from an online transaction processing (OLTP) transaction updatable system that includes transaction commit, rollback, and field level security capabilities. It also relates to transparently decoupling extraction from rebuilding of frequently updated OLAP analytic databases from the OLTP transaction updatable system. ## Claims ### Claim 1 A method of controlling deployment of units of work to an analytic query and build cluster (referred to as analytics cluster), following dataflow build specifications defined in the units of work, creating multiple non-updatable online analytic processing (abbreviated OLAP) databases, using hardware resources of the analytics cluster that are shared between building databases and responding to user queries, the method including: using an execution plan controller executing on hardware separate from the analytics cluster, deploying the units of work in a current step to the analytics cluster according to an execution plan, including: responsive to the execution plan, the execution plan controller grouping the units of work into steps for execution and submitting the units of work for a particular step to the analytics cluster, without prerequisites for order of processing among the units of work in the particular step, and further responsive to the execution plan, the execution plan controller creating bulk updatable-only builder-input-datasets and creating, from the bulk updatable-only builder-input-datasets, multiple non-updatable OLAP analytic databases; and using the execution plan controller, monitoring completion by the analytics cluster of all units of work in the current step of a step list in the execution plan before deploying to the analytics cluster additional units of work at a next step in the step list. ### Claim 2 (depends on 1) The method of claim 1, wherein the analytics cluster shares limited hardware resources among multiple users and multiple non-updatable OLAP analytic databases, and guards user query response experience by controlling its own ordering of simultaneous units of work and resource allocation between responding to queries and executing the units of work deployed by the execution plan controller. ### Claim 3 (depends on 1) The method of claim 1, wherein at least a plurality of the ordered units of work have prerequisites among units of work from entry points to at least one exit point in a directed acyclic graph. ### Claim 4 (depends on 3) The method of claim 3, wherein the prerequisites among the units of work are arranged in chains from entry to exit points of the chains, the chains have indexes that count the number of units of work from entry to exit points of the chains, and each unit of work in a particular chain is in a separate step from other units of work in the particular chain. ### Claim 5 (depends on 4) The method of claim 4, wherein a plurality of the chains have indexes greater than or equal to four. ### Claim 6 (depends on 4) The method of claim 4, wherein a plurality of the chains have indexes greater than or equal to five. ### Claim 7 (depends on 3) The method of claim 3, wherein units of work include: extraction of object sets from an OLTP transaction updatable system to incrementally replicate database objects and fields updated since a prior extraction, from the OLTP transaction updatable system, to multiple bulk updatable-only builder-input-datasets for respective object sets; and extractions of the object sets are entry points to the directed acyclic graph. ### Claim 8 (depends on 7) The method of claim 7, wherein units of work also include: extraction of object sets from an OLTP transaction updatable system to fully replicate database objects and fields if bulk updatable-only builder-input-datasets for respective object sets have not been created in a prior extraction. ### Claim 9 (depends on 7) The method of claim 7, wherein units of work also include: merging and organizing the multiple bulk updatable-only builder-input-datasets for the respective object sets from a replicated data cache to produce the multiple non-updatable OLAP analytic databases. ### Claim 10 (depends on 7) The method of claim 7, wherein units of work also include: merging and organizing the multiple bulk updatable-only builder-input-datasets for the respective object sets to produce the multiple intermediate non-updatable OLAP analytic databases; and merging and organizing the multiple intermediate non-updatable OLAP analytic databases to produce at least one non-updatable OLAP analytic database. ### Claim 11 (depends on 3) The method of claim 3, wherein the units of work with OLAP dataflow build specifications for replication dataflows are deployed to the analytics cluster before the units of work with OLAP dataflow build specifications for user dataflows are deployed to the analytics cluster. ### Claim 12 (depends on 3) The method of claim 3, wherein the ordering of the ordered units of work is defined based on user input through a user interface. ### Claim 13 (depends on 3) The method of claim 3, wherein the ordering of the ordered units of work is defined based on user input received at an application programming interface (API) end point. ### Claim 14 (depends on 1) The method of claim 1, wherein the execution plan controller creates a snapshot of the execution plan before deploying the ordered units of work to the analytics cluster. ### Claim 15 (depends on 14) The method of claim 14, wherein the execution plan controller, in response to detecting a change to the snapshotted execution plan after initiating deployment of the ordered units of work to the analytics cluster in accordance with the snapshotted execution plan: maintains the deployment ordering in accordance with the snapshotted execution plan. ### Claim 16 (depends on 1) The method of claim 1, wherein the execution plan is generated based on properties that identifies scheduling and execution attributes of the OLAP dataflow build specifications. ### Claim 17 (depends on 1) The method of claim 1, wherein the execution plan identifies prerequisites data processing operations to be completed before initiating the deployment of the OLAP dataflow build specifications to the analytics cluster. ### Claim 18 (depends on 17) The method of claim 17, wherein the prerequisite data processing includes generating a CSV file from an external data source to be identified as input source in at least one of the OLAP dataflow build specifications deployed to the analytics cluster. ### Claim 19 (depends on 1) The method of claim 1, further including an execution plan tracker that tracks states of the OLAP dataflow build specifications identified in the execution plan by: using a request ID to identify undispatched dataflow build specifications that have not yet been deployed to the analytics cluster; using an instance ID to identify dispatched dataflow build specifications that have been deployed to the analytics cluster but not yet completely processed by the analytics cluster; and using a complete ID to identify dataflow build specifications that have been completely processed by the analytics cluster. ### Claim 20 (depends on 19) The method of claim 19, wherein the execution plan tracker implements a failure recovery scheme which, in response to detecting a failed execution of a first OLAP dataflow build specification, responds by reprocessing the first OLAP dataflow build specification. ### Claim 21 A system including one or more processors coupled to memory, the memory loaded with computer instructions to control deployment of units of work to an analytic query and build cluster (referred to as analytics cluster), following dataflow build specifications defined in the units of work, creating multiple non-updatable online analytic processing (abbreviated OLAP) databases, using hardware resources of the analytics cluster that are shared between building databases and responding to user queries, the instructions, when executed on the processors, implement actions comprising: using an execution plan controller executing on hardware separate from the analytics cluster, deploying the units of work in a current step to the analytics cluster according to an execution plan, including: responsive to the execution plan, the execution plan controller grouping the units of work into steps for execution and submitting the units of work for a particular step to the analytics cluster, without prerequisites for order of processing among the units of work in the particular step, and further responsive to the execution plan, the execution plan controller creating bulk updatable-only builder-input-datasets and creating, from the bulk updatable-only builder-input-datasets, multiple non-updatable OLAP analytic databases; and using the execution plan controller, monitoring completion by the analytics cluster of all units of work in the current step of a step list in the execution plan before deploying to the analytics cluster additional units of work at a next step in the step list. ### Claim 22 A non-transitory computer readable storage medium impressed with computer program instructions to control deployment of units of work to an analytic query and build cluster (referred to as analytics cluster), following dataflow build specifications defined in the units of work, creating multiple non-updatable online analytic processing (abbreviated OLAP) databases, using hardware resources of the analytics cluster that are shared between building databases and responding to user queries, the instructions, when executed on a processor, implement a method comprising: using an execution plan controller executing on hardware separate from the analytics cluster, deploying the units of work in a current step to the analytics cluster according to an execution plan, including: responsive to the execution plan, the execution plan controller grouping the units of work into steps for execution and submitting the units of work for a particular step to the analytics cluster, without prerequisites for order of processing among the units of work in the particular step, and further responsive to the execution plan, the execution plan controller creating bulk updatable-only builder-input-datasets and creating, from the bulk updatable-only builder-input-datasets, multiple non-updatable OLAP analytic databases; and using the execution plan controller, monitoring completion by the analytics cluster of all units of work in the current step of a step list in the execution plan before deploying to the analytics cluster additional units of work at a next step in the step list.