# US 2018/0025024 A1 · Pluggable storage system for parallel query engines

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0025024 A1
- Document ID: US20180025024A1
- Status: Published application
- Kind: A1
- Application number: 15/714,651
- Filed: 2017-09-25
- Published: 2018-01-25
- Earliest claimed: 2013-02-25
- Assignees: EMC IP Holding Company LLC
- Applicants: EMC IP Holding Company LLC
- Inventors: Sameer Tiwari; Milind Arun Bhandarkar; Bhooshan Deepak Mogal
- Web page: https://patentgrep.com/patent/US20180025024A1

## CPC

- G06F 16/24542: {Plan optimisation}
- G06F 16/10: File systems; File servers
- G06F 16/11: File system administration, e.g. details of archiving or snapshots (error detection or correction of the data by redundancy in operations G06F11/14)
- G06F 16/148: {File search processing}
- G06F 16/182: Distributed file systems
- G06F 16/1858: {Parallel file systems, i.e. file systems supporting multiple processors}
- G06F 16/24524: {Access plan code generation and invalidation; Reuse of access plans}
- G06F 16/2453: Query optimisation
- G06F 16/24532: {of parallel queries}
- G06F 16/2455: Query execution
- G06F 16/2471: {Distributed queries}
- G06F 16/27: Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F 16/907: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- H04L 65/60: Network streaming of media packets
- H04L 67/1097: for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
- G06F 16/113: {Details of archiving (lifecycle management in storage systems G06F3/0649; point-in-time backing up or restoration of persistent data G06F11/1446)}
- G06F 16/217: {Database tuning (G06F16/2282 takes precedence; database performance monitoring G06F11/3409)}
- G06F 16/245: Query processing
- G06F 16/43: Querying
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A method, article of manufacture, and apparatus for managing data. In some embodiments, this includes, receiving a query from a client, based on the received query, analyzing a catalog for location information, based on the analysis, determining a first storage system, an associated first file system and an associated first protocol, using the associated first protocol to communicate with the first storage system, and performing at least a portion of the query on the first storage system.

## Claims

### Claim 1

A method for managing data, comprising:
receiving, by a universal namenode, a query from a client;
based at least in part on the received query, accessing a catalog service and searching a catalog provided by the catalog service for location information of one or more files responsive to the query;
based at least in part on a result of the search of the catalog, determining to move at least one of the plurality of files from a second storage system to a first storage system, and determining to communicate with the first storage system in connection with the one or more files responsive to the queries;
performing at least a portion of the query on the first storage system; and
providing, to the client, results of the query such that in the event that various portions of the results correspond to query results stored on a set of a plurality of storage systems, the results of the query are presented in a unified view across the set of the plurality of storage systems and appear, from a perspective of the client, to exist from a single namespace, wherein the plurality of storage systems comprises at least the first storage system and the second storage system.

### Claim 2 (depends on 1)

The method of claim 1, further comprising determining a first protocol for communication with the storage system, wherein the universal namenode communicates with the first storage system using the first protocol.

### Claim 3 (depends on 1)

The method of claim 1, wherein the catalog stores a mapping of a plurality of files stored on a plurality of storage systems to a location at which the plurality of files are respectively stored on the corresponding plurality of storage systems.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first protocol for communication with the first storage system is determined, based at least in part, on a first file system associated with the first storage system.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
determining whether to move a first file that is stored on the first storage system to another storage system based at least in part on a frequency with which the first file is accessed;
in the event that the first file is determined to be moved from the first storage system to the other storage system, moving the first file to the other storage system; and
updating location information associated with the first file stored in the catalog to point from a location at which the first file was stored in the first storage system to a location at which the first file is stored in the other storage system, wherein, from the perspective of the client, the a location of the first file is the same in relation to the singe namespace regardless of whether the first file is moved from the first storage system to the other storage system.

### Claim 6 (depends on 5)

The method of claim 5, wherein the updating of the location information comprises updating the mapping of the plurality of files stored on the plurality of storage systems to the location at which the plurality of files so as to update a record corresponding to the first file.

### Claim 7 (depends on 1)

The method of claim 1, wherein the query received from the client is a query to identify one or more files stored across a distributed storage system comprising the plurality of storage systems.

### Claim 8 (depends on 1)

The method of claim 1, wherein the determining to move the at least one of the plurality of files from the second storage system to the first storage system based at least in part on the search of the catalog comprises: determining to move the at least one of the plurality of files in response to the receiving the query from the client.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
tracking, by a universal job tracker of the universal namenode, a status of one or more portions of the query being respectively performed by corresponding ones of the plurality of storage systems.

### Claim 10 (depends on 1)

The method of claim 1, further comprising the first storage system reporting back the results of at least a portion of the query.

### Claim 11 (depends on 1)

The method of claim 1, further comprising adding a new storage system.

### Claim 12 (depends on 11)

The method of claim 11, further comprising determining a new protocol for communication with the new storage system.

### Claim 13 (depends on 12)

The method of claim 12, wherein a new file system associated with the new storage system is the same as a first file system associated with the first storage system.

### Claim 14 (depends on 12)

The method of claim 12, wherein a new file system associated with the new storage system is not the same as a first file system associated with the first storage system.

### Claim 15 (depends on 12)

The method of claim 12, further comprising updating the catalog based on data stored in the new storage system.

### Claim 16 (depends on 1)

The method of claim 1, further comprising determining to communicate with the second storage system in connection with the one or more files responsive to the queries, and determining a second protocol for communication with the second storage system.

### Claim 17

A system for managing data, comprising one or more processors configured to:
receive a query from a client;
based at least in part on the received query, accessing a catalog service and searching a catalog provided by the catalog service for location information of one or more files responsive to the query;
based at least in part on a result of the search of the catalog, determine to move at least one of the plurality of files from a second storage system to a first storage system, and determine to communicate with the first storage system in connection with the one or more files responsive to the queries;
perform at least a portion of the query on the first storage system; and
provide, to the client, results of the query such that in the event that various portions of the results correspond to query results stored on a set of a plurality of storage systems, the results of the query are presented in a unified view across the set of the plurality of storage systems and appear, from a perspective of the client, to exist from a single namespace, wherein the plurality of storage systems comprises at least the first storage system and the second storage system.

### Claim 18

A computer program product for processing data, comprising a non-transitory computer readable medium having program instructions embodied therein for:
receiving a query from a client;
based at least in part on the received query, accessing a catalog service and searching a catalog provided by the catalog service for location information of one or more files responsive to the query;
based at least in part on a result of the search of the catalog, determining to move at least one of the plurality of files from a second storage system to a first storage system, and determining to communicate with the first storage system in connection with the one or more files responsive to the queries;
performing at least a portion of the query on the first storage system; and
providing, to the client, results of the query such that in the event that various portions of the results correspond to query results stored on a set of a plurality of storage systems, the results of the query are presented in a unified view across the set of the plurality of storage systems and appear, from a perspective of the client, to exist from a single namespace, wherein the plurality of storage systems comprises at least the first storage system and the second storage system.
