Back to results
US 12,743,325 B1Granted
Continuous integration and updating a live extract, transform, and load (ETL) workflow based on determined code change
AssigneeAmazon Technologies, Inc.
InventorsGanesh Vel Thangavelu, Satish Gurav, Sandeep Kumar Lakshmichand Jain, Shruti Sanjay Shinde
Life of this patent
Estimating the term…
Abstract
Provided is a system for facilitating continuous integration and continuous deployment of data warehousing workflows in a cloud network environment. A workflow integration and deployment service may communicate with a code repository service to be notified of code changes, obtain code changes, generate updated workflows, test updated workflows, and deploy updated workflows. The workflow integration and deployment service may maintain workflow generation and testing rules to generate, test, and deploy workflows according to user preferences.
Claims20
A cloud provider system comprising:
- one or more processors;
- a code repository service, executed by the one or more processors, storing a plurality of user tasks, wherein each task of the plurality of user tasks when executed by the one or more processors is configured to perform an operation as part of an extract, transform, and load (ETL) workflow for extracting data from one or more data sources, transforming the data, and loading the transformed data onto one or more destinations;
- and
- a workflow integration and deployment service comprising a workflow manager with one or more workflow testing rules and in network communication with the code repository service, wherein the workflow integration and deployment service when executed by the one or more processors is configured to at least:
- receive, from the code repository service, an indication that a code change has been made to a task associated with a user account;
- determine that the task is part of a live ETL workflow that has been deployed based on the ETL workflow on a first set of computing resources assigned to the user account and configured to receive input data from a source associated with the user account, transform the input data, and output the transformed input data to a destination associated with the user account;
- determine that the code change made to the task is to be propagated to the live ETL workflow such that any additional data subsequently received by the live ETL workflow is transformed according to the code change made to the task;
- identify a set of dependencies associated with the task based at least in part on the code change made to the task, wherein the set of dependencies associated with the task includes code dependency information usable to determine a position of the task within an updated ETL workflow, and wherein the set of dependencies includes at least time dependency information usable to determine a time or time range at which the task is to be executed and cluster dependency information usable to determine a cluster of compute capacity in which the task is to be executed;
- generate, using the workflow manager and based on the code dependency information, the time dependency information, and the cluster dependency information, the updated ETL workflow that includes the code change made to the task and satisfies the set of dependencies associated with the task;
- test, using the workflow manager, the updated ETL workflow according to the one or more workflow testing rules associated with the user account to lower error rates;
- update and deploy the live ETL workflow according to the updated ETL workflow based on the testing;
- and
- automatically execute the updated live ETL workflow to cause the updated live ETL workflow to receive additional input data from the source, transform the additional input data according to the code change made to the task, and output the transformed additional input data to the destination.
10 more elements
A computer-implemented method comprising:
- receiving, from a code repository service executed by one or more processors, an indication that a code change has been made to a task associated with a user account that is part of a live extract, transform, and load (ETL) workflow that has been deployed on a first set of computing resources assigned to the user account for extracting data from one or more data sources, transforming the data, and loading the transformed data onto one or more destinations, wherein the live ETL workflow comprises a plurality of tasks that when executed by the one or more processors are each configured to perform an operation as part of the live ETL workflow and is configured to receive input data from a source associated with the user account, transform the input data, and output the transformed input data to a destination associated with the user account;
- determining that the code change made to the task is to be propagated to the live ETL workflow such that any additional data subsequently received by the live ETL workflow is transformed according to the code change made to the task;
- identifying a set of dependencies associated with the task based at least in part on the code change made to the task, wherein the set of dependencies associated with the task include code dependency information usable to determine a position of the task within an updated ETL workflow, and wherein the set of dependencies includes at least time dependency information usable to determine a time or time range at which the task is to be executed and cluster dependency information usable to determine a cluster of compute capacity in which the task is to be executed;
- generating, by a workflow manager with one or more workflow testing rules of a workflow integration and deployment service executed by the one or more processors, the updated ETL workflow based at least on the code dependency information, the time dependency information, and the cluster dependency information, wherein the updated ETL workflow includes the code change made to the task and satisfies the set of dependencies associated with the task;
- testing, by the workflow manager, the updated ETL workflow according to the one or more workflow testing rules associated with the user account to lower error rates;
- updating and deploying the live ETL workflow according to the updated ETL workflow based on the testing;
- and
- automatically executing the updated live ETL workflow such that the updated live ETL workflow is configured to receive additional input data from the source, transform the additional input data according to the code change made to the task, and output the transformed additional input data to the destination.
5 more elements
A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a computing system within a cloud provider network, cause the computing system to perform operations comprising:
- receiving, from a code repository service executed by the one or more processors, an indication that a code change has been made to a task associated with a user account that is part of a live extract, transform, and load (ETL) workflow that has been deployed on a first set of computing resources assigned to the user account for performing data transformations, wherein the live ETL workflow comprises a plurality of tasks that when executed by the one or more processors are each configured to perform an operation as part of the live ETL workflow and is configured to receive input data from a source associated with the user account, transform the input data, and output the transformed input data to a destination associated with the user account;
- determining that the code change made to the task is to be propagated to the live ETL workflow such that any additional data subsequently received by the live ETL workflow is transformed according to the code change made to the task;
- identifying a set of dependencies associated with the task based at least in part on the code change made to the task, wherein the set of dependencies associated with the task include code dependency information usable to determine a position of the task within an updated ETL workflow, and wherein the set of dependencies includes at least time dependency information usable to determine a time or time range at which the task is to be executed and cluster dependency information usable to determine a cluster of compute capacity in which the task is to be executed;
- generating, by a workflow manager with one or more workflow testing rules of a workflow integration and deployment service executed by the one or more processors, the updated ETL workflow based at least on the code dependency information, the time dependency information, and the cluster dependency information, wherein the updated ETL workflow includes the code change made to the task and satisfies the set of dependencies associated with the task;
- testing, by the workflow manager, the updated ETL workflow according to the one or more workflow testing rules associated with the user account to lower error rates;
- updating and deploying the live ETL workflow according to the updated ETL workflow based on the testing;
- and
- automatically executing the updated live ETL workflow such that the updated live ETL workflow is configured to receive additional input data from the source, transform the additional input data according to the code change made to the task, and output the transformed additional input data to the destination.
5 more elements
Dependencies are inferred from the claim text.