# US 2021/0173828 A1 · Persistent metadata catalog

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0173828 A1
- Document ID: US20210173828A1
- Status: Published application
- Kind: A1
- Application number: 17/104,392
- Filed: 2020-11-25
- Published: 2021-06-10
- Earliest claimed: 2014-06-20
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: George Steven McPherson; Yevgeniy Mikhaylyuta; Tobin Daniel Baker; Richard Jeffrey Cole
- Web page: https://patentgrep.com/patent/US20210173828A1

## CPC

- G06F 16/21: Design, administration or maintenance of databases
- G06F 16/24: Querying
- G06F 16/25: Integrating or interfacing systems involving database management systems
- G06F 21/6218: {to a system of files or objects, e.g. local or distributed file system or database}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A metadata catalog may store metadata about data sources. The metadata may define table, column, partition and/or data formats, as well as title information, textual descriptions, modification information, data schemas and access information, such as location and connection information. A metadata catalog service may provide a shared, persistent, central repository for the metadata. The metadata of the catalog may be published via multiple access mechanisms and programmatically obtained and processed by other programs via the metadata catalog service. Metadata consumers may utilize the metadata catalog service to perform various metadata tasks, such as metadata retrieval, creation, addition, modification and/or deletion. The metadata catalog service may be directly integrated into existing data analytic tools as a persistent and durable metadata store allowing users to utilize the broad set of data processing tools.

## Claims

### Claim 1

A system, comprising:
one or more computers configured to implement:
a persistent metadata store comprising metadata information regarding a plurality of data sources; and
a metadata catalog service, wherein the metadata catalog service is accessible via an application programming interface that provides discovery of, and access to, the metadata information of the metadata store;
wherein the metadata catalog service is configured to:
maintain, in the metadata store, the metadata information regarding the plurality of data sources, wherein each data source is external to the metadata store and the metadata catalog service;
receive, via the application programming interface, requests from a plurality of different metadata producers to add metadata information regarding respective ones of the data sources;
store, in response to the requests to add metadata information, the metadata information regarding the respective ones of the data sources to the metadata store;
receive, via the application programming interface, requests from a plurality of different metadata consumers for metadata information regarding respective ones of the data sources; and
provide, in response to a respective request for metadata information, the respective requested metadata information from the metadata store to the respective metadata consumer, wherein metadata information regarding different ones of the data sources is provided to different metadata consumers based on the respective metadata consumer's request.

### Claim 2 (depends on 1)

The system of claim 1, wherein the persistent metadata store and the metadata catalog service are part of a provider network configured to host a plurality of computer servers implementing a plurality of network-based services, and wherein the metadata catalog service is configured to:
provide access to the metadata store to the plurality of different metadata producers, wherein individual ones of the plurality of different metadata producers are part of the provider network, and wherein other ones of the plurality of different metadata producers are not part of the provider network;
provide access to the metadata store to the plurality of different metadata consumers, wherein individual ones of the plurality of different metadata consumers are part of the provider network, and wherein other ones of the plurality of different metadata consumers are not part of the provider network; and
wherein individual ones of the plurality of data sources are part of the provider network, and wherein other ones of the plurality of data sources are not part of the provider network.

### Claim 3 (depends on 1)

The system of claim 1, wherein to provide the requested respective metadata information, the metadata catalog service is further configured to determine that the respective metadata consumer has permission to access the respective metadata information based on permission data regarding the respective data source.

### Claim 4 (depends on 1)

The system of claim 1, wherein to store the metadata information, the metadata catalog service is further configured to determine that the respective metadata producer has permission to add the respective metadata information to the metadata store based on permission data regarding the respective data source.

### Claim 5 (depends on 1)

The system of claim 1, further comprising:
an artifact repository comprising a plurality of binary artifacts configured to provide programmatic access to data on respective ones of the plurality of data sources;
wherein the metadata catalog service is further configured to:
receive, via the application programming interface, requests from individual ones of the plurality of different metadata consumers for individual ones of the plurality of binary artifacts; and
provide, in response to a respective request for a binary artifact, the respective one of the plurality of binary artifacts from the artifact repository to the respective metadata consumer, wherein different binary artifacts are provided to different metadata consumers based on the respective metadata consumer's request.

### Claim 6

A method, comprising:
performing, by one or more computers:
providing network-based access, via a network-based interface of a metadata catalog service, to metadata information regarding data on a plurality of data sources that are distinct from, and external to, the metadata catalog service, and wherein the metadata information is accessible, via the network-based interface, to a plurality of different metadata consumers that are distinct from, and external to, the metadata catalog service;
receiving, from a requesting one of the plurality of metadata consumers via the network-based interface of the metadata catalog service, a request for metadata information regarding data on a particular one of the plurality of data sources;
determining, according to permission data for the particular data source, that the requesting metadata consumer has permission to access the metadata information regarding data on a particular one of the plurality of data sources; and
providing, in response to determining that the metadata consumer does have permission to access the metadata information, the metadata information regarding data on the particular data source to the requesting metadata consumer.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
receiving, from another metadata consumer via the network-based interface of the metadata catalog service, a request for the metadata information regarding data on the particular data source;
determining, according to permission data for the particular one of the plurality of data sources, that the other metadata consumer does not have permission to access the metadata information regarding data on the particular data source; and
not providing the metadata information regarding data on the particular data source to the other metadata consumer in response to determining that the other metadata consumer does not have permission to access the metadata information regarding data on the particular data source.

### Claim 8 (depends on 6)

The method of claim 6, wherein said determining comprises validating authentication information received from the requesting consumer.

### Claim 9 (depends on 6)

The method of claim 6, wherein the metadata catalog service module is configured to provide metadata information to individual ones of the plurality of metadata consumers, such that at least one metadata consumer is not provided metadata information that is provided to another of the metadata consumers.

### Claim 10 (depends on 6)

The method of claim 6,
wherein said receiving a request for metadata information is performed via one or more of;
an application programming interface of the network-based interface;
a command line interface of the network-based interface; or
a web interface of the network-based interface;
wherein the request is initiated by either a human or a computer agent.

### Claim 11 (depends on 6)

The method of claim 6,
wherein the metadata information regarding data on the particular data source comprises one or more of:
schema data regarding the data;
description data regarding the data;
location data regarding the particular data source;
connection data regarding the particular data source;
information regarding one or more binary artifacts for accessing the data; or
transformation information regarding manipulating the data.

### Claim 12 (depends on 6)

The method of claim 6, wherein said providing comprises decrypting the requested metadata information based, in part, on a decryption key provided by the requesting metadata consumer.

### Claim 13

A non-transitory, computer-accessible storage medium comprising program instructions executable on one or more computers to implement:
providing network-based access, via a network-based interface of a metadata catalog service, to metadata information regarding a plurality of different data sources distinct from, and external to, the metadata catalog service, wherein the metadata information comprises information about data on the plurality of data sources, and wherein the metadata information is accessible, via the network-based interface, to a plurality of metadata producers distinct from, and external to, the metadata catalog service;
receiving, from one of the plurality of metadata producers via the network-based interface of the metadata catalog service, a request to store metadata information regarding the data on one of the data sources; and
storing, in response to the request to store metadata information, the metadata information regarding the data on the one of the data sources in a persistent metadata catalog configured to persistently store metadata information.

### Claim 14 (depends on 13)

The non-transitory, computer-accessible storage medium of claim 13, wherein the request to store metadata information comprises schema information regarding the data on the data source.

### Claim 15 (depends on 13)

The non-transitory, computer-accessible storage medium of claim 13, wherein the program instructions are further executable on the one or more computers to implement:
receiving a request from another of the plurality of metadata producers to remove metadata information for a particular one of the plurality of data sources; and
removing the metadata information regarding the particular data source from the metadata store in response to receiving the request to remove the metadata information.

### Claim 16 (depends on 13)

The non-transitory, computer-accessible storage medium of claim 13, wherein the program instructions are further executable on the one or more computers to implement:
receiving a request from another of the plurality of metadata producers to modify metadata information for a particular one of the plurality of data sources; and
modifying the metadata information for the particular data source according to modification information provided in the request to modify the metadata information in response to receiving the request to modify the metadata information.

### Claim 17 (depends on 16)

The non-transitory, computer-accessible storage medium of claim 16, wherein the modification information provided in the request to modify specifies one or more of:
schema information for the particular data source;
location information for the particular data source;
connection information for the particular data source;
description information for the particular data source;
an encryption key regarding either the particular data source or the metadata information for the particular data source;
notification pool information regarding the particular data source;
configuration information regarding accessing the particular data source; or
configuration information for an application regarding the particular data source.

### Claim 18 (depends on 13)

The non-transitory, computer-accessible storage medium of claim 13, wherein the request to store metadata information comprises a request for store metadata information regarding a binary artifact corresponding to the data on the data source.

### Claim 19 (depends on 18)

The non-transitory, computer-accessible storage medium of claim 18, wherein the binary artifact comprises one or more of:
a client software development kit configured to access data from the data source;
scripts configured to access data from the data source;
information specifying one or more keys values configured for accessing particular data on the data source;
information specifying one or more queries for accessing particular data on the data source;
a deserializer object configured to load data from the data source;
a serializer object configured to store data to the data source; or
a binary module executable to access the data source.

### Claim 20 (depends on 13)

The non-transitory, computer-accessible storage medium of claim 13, wherein the request to store metadata information comprises a request to encrypt the metadata information, wherein the program instructions are further executable on the one or more computers to implement:
encrypting the metadata information prior to said storing the metadata information in the persistent metadata catalog.
