US 11,886,429 B2Granted

Persistent metadata catalog

AssigneeAmazon Technologies, Inc.

InventorsGeorge Steven McPherson, Yevgeniy Mikhaylyuta, Tobin Daniel Baker, Richard Jeffrey Cole

Life of this patent

Estimating the term…

Loading patent…

Abstract

A metadata catalog may store metadata about data sources. The metadata may define table, column, partition and/or data formats, as well as title information, textual descriptions, modification information, data schemas and access information, such as location and connection information. A metadata catalog service may provide a shared, persistent, central repository for the metadata. The metadata of the catalog may be published via multiple access mechanisms and programmatically obtained and processed by other programs via the metadata catalog service. Metadata consumers may utilize the metadata catalog service to perform various metadata tasks, such as metadata retrieval, creation, addition, modification and/or deletion. The metadata catalog service may be directly integrated into existing data analytic tools as a persistent and durable metadata store allowing users to utilize the broad set of data processing tools.

Claims20

Dependent claims

A system, comprising:

one or more computing devices configured to implement:
a data catalog comprising a central repository accessible to one or more data producers and one or more data consumers, the data catalog configured to:
store metadata regarding a plurality of data sources configured to store data, wherein the metadata includes a description, a location, a schema, and access information for accessing the data stored at respective ones of the plurality of data sources;
receive, from a data producer of the one or more data producers, a request to add new metadata of a new data source to the plurality of data sources of the data catalog;
add the new metadata for the new data source to the data catalog according to the request to add the new metadata, wherein the new metadata comprises information indicating a description, a location, and a schema for data stored at the new data source;
receive, from a data consumer of the one or more data consumers, a request to discover at least one data source of the plurality of data sources;
in response to the request to discover the at least one data source, provide the metadata for the at least one data source from the data catalog to the data consumer, wherein the data consumer is configured to connect to the at least one data source according to the provided metadata, and wherein the metadata provided from the data catalog indicates a location of a loadable software module configured to execute at the data consumer and access the at least one data source on behalf of the data consumer;
and
provide the loadable software module to the data consumer.

6 more elements

A method, comprising:

performing, with one or more computing devices:
storing, to a data catalog comprising a central repository accessible to one or more data producers and one or more data consumers, metadata regarding a plurality of data sources configured to store data, wherein the metadata includes a description, a location, a schema, and access information for accessing the data stored at respective ones of the plurality of data sources;
receiving, from a data producer of the one or more data producers, a request to add new metadata of a new data source to the plurality of data sources of the data catalog;
adding the new metadata for the new data source to the data catalog according to the request to add the new metadata, wherein the new metadata comprises information indicating a description, a location, and a schema for data stored at the new data source;
receiving, from a data consumer of the one or more data consumers, a request to discover at least one data source of the plurality of data sources;
in response to the request to discover the at least one data source, providing the metadata for the at least one data source from the data catalog to the data consumer, wherein the data consumer is configured to connect to the at least one data source according to the provided metadata, and wherein the metadata provided from the data catalog indicates a location of a loadable software module configured to execute at the data consumer and access the at least one data source on behalf of the data consumer;
and
providing the loadable software module to the data consumer.

5 more elements

One or more computer-readable storage media storing instructions that, when executed on or across one or more processors, cause the one or more processors to implement a data catalog configured to:

store, to the data catalog comprising a central repository accessible to one or more data producers and one or more data consumers, metadata regarding a plurality of data sources configured to store data, wherein the metadata includes a description, a location, a schema, and access information for accessing the data stored at respective ones of the plurality of data sources;
in response to a request from a data producer of the one or more data producers to add new metadata of a new data source to the plurality of data sources of the data catalog, add the new metadata for the new data source to the data catalog according to the request to add the new metadata, wherein the new metadata comprises information indicating a description, a location, and a schema for data stored at the new data source;
in response to the request from a data consumer of the one or more data consumers to discover at least one data source of the plurality of data sources, provide the metadata for the at least one data source from the data catalog to the data consumer, wherein the data consumer is configured to connect to the at least one data source according to the provided metadata, and wherein the metadata provided from the catalog service indicates a location of a loadable software module configured to execute at the data consumer and access the at least one data source on behalf of the data consumer;
and,
provide the loadable software module to the data consumer.

2 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/104,392Filed 25 Nov 2020 · continuation
Priority: 20 Jun 2014
Published: US 2021/0173828 A110 Jun 2021
Granted: 30 Jan 2024
Term ends: Not estimatedThe source records don’t support an estimate.

Classification5 codes

G06F 16/21
Of structured data, e.g. relational data: design, administration or maintenance of databases
G06F 16/24
Of structured data, e.g. relational data: querying
G06F 16/25
Of structured data, e.g. relational data: integrating or interfacing systems involving database management systems
G06F 21/6218
Protecting access to data via a platform, e.g. using keys or access control rules: to a system of files or objects, e.g. local or distributed file…