US 10,503,748 B1Granted

ELT management service with publish/subscribe architecture

AssigneeAmazon Technologies, Inc.

InventorPeter Farrow

Life of this patent

Estimating the term…

Loading patent…

Abstract

An extract, load, and transform (ELT) management service provides functionality for managing ELT operations, including publish/subscribe functionality. The ELT management service maintains metadata describing publications, such as database tables or files stored by a storage service operating in a service provider network, and publication certificates, which describe the creation and change history for the publications. The metadata can be searched to identify publications that rely on other publications that have changed since the publications were generated. The identified publications can then be automatically republished by executing logical pipelines that specify activities that can be executed to republish the publications.

Claims20

Dependent claims

A system, comprising:

one or more processors;
and
at least one non-transitory computer-readable storage medium having instructions stored thereupon which are executable by the one or more processors and which, when executed, cause the one or more processors to perform acts comprising:
storing metadata that describes a plurality of publications and publication certificates associated with the plurality of publications, the publication certificates comprising data describing creation and change history of the plurality of publications, the plurality of publications comprising database tables on database clusters provided by a data warehouse service executing in a service provider network and files stored by a storage service executing in the service provider network, the database clusters comprising interconnected databases that are connected via a network in a distributed computing environment;
performing a query against the metadata to generate a list of publications, wherein the list of publications derives publications from one or more other publications that have changed since the publications in the list of publications were last generated;
and triggering execution of one or more pipelines to republish the publications identified in the list of publications by executing one or more extract, transform, or load activities, wherein the one or more pipelines comprise one or more logical activities that execute against a cluster group included in one or more of the database clusters, the one or more logical activities comprising at least one of the one or more extract, transform, or load activities.

3 more elements

A non-transitory computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by a computer, cause the computer to:

store metadata describing a plurality of publications and publication certificates associated with the plurality of publications, the publication certificates comprising data describing creation and change history of the plurality of publications, the plurality of publications comprising a database table stored on a database cluster, the database cluster comprising interconnected databases that are connected via a network in a distributed computing environment;
perform a query against the metadata to generate a list of the plurality of publications, wherein the plurality of publications in the list is derived from one or more other publications that have changed since the plurality of publications in the list were last generated;
and
cause one or more pipelines to be executed against a cluster group to republish the plurality of publications identified in the list, wherein the cluster group is included in the database cluster, and wherein the one or more pipelines comprise one or more logical activities that execute against the cluster group, the one or more logical activities comprising one or more extract, transform, or load activities.

1 more element

A computer-implemented method, comprising:

storing metadata describing a plurality of publications and publication certificates associated with the plurality of publications, the publication certificates comprising data describing creation and change history of the plurality of publications, the plurality of publications comprising a database table stored on a database cluster, the database cluster comprising interconnected databases that are connected via a network in a distributed computing environment;
causing a query to be performed against the metadata to generate a list of the plurality of publications, wherein the plurality of publications in the list is derived from one or more other publications that have changed since the plurality of publications in the list were last generated;
and
causing one or more pipelines to be executed against a cluster group to republish the plurality of publications identified in the list, wherein the cluster group is included in the database cluster, and wherein the one or more pipelines comprise one or more logical activities that execute against the cluster group, the one or more logical activities comprising one or more extract, transform, or load activities.

1 more element

Dependencies are inferred from the claim text.