US 2014/0351300 A1Published application

Method and system for data transfer between compute clusters and file system

AssigneeDataDirect Networks, Inc.

InventorsPAVAN KUMAR UPPU, JASON MICAH COPE, PAUL NOWOCZYNSKI, MICHAEL PISZCZEK

Life of this patent

Estimating the term…

Loading patent…

Abstract

A data migrating system and method are provided in which a Burst Buffer Network Aggregator (BBNA) process is configured either on the File Servers or on the File System's dedicated I/O nodes to coalesce data fragments stored in participating Burst Buffer nodes under the direction of a primary BB node appointed by a data generating entity prior to transfer of the full data stripe into the File System. The “write” request in the form of a full data stripe is distributed into a plurality of data fragments among participating BB nodes along with corresponding metadata. The primary BB node gathers the metadata from the participating BB nodes, sends the metadata list to the BBNA unit, responsive to which the BBNA unit allocates a buffer sufficient to store the full data stripe, and transfers data fragments from participating BB nodes into the full data stripe buffer, thereby coalescing the data fragments into the full data stripe, which is subsequently transferred from the buffer in the BBNA unit into the File System.

Claims23

Dependent claims

A method for data migration between data generating entities and a File System, comprising the steps of:

(a) operatively coupling a Burst Buffer (BB) tier between at least one data generating entity and a File System, and configuring said BB tier with a plurality of Burst Buffer (BB) nodes, each operating with data input speed exceeding the data input speed of said File System;
(b) configuring said File System with at least one Burst Buffer Network Aggregator (BBNA) unit;
(c) sending a first data “write” request from said at least one data generating entity to said BB tier;
(d) distributing said data into data fragments, and storing each data fragment and metadata associated therewith in a respective participating BB node from said plurality of BB nodes in said BB tier;
(e) assigning a primary BB node from said participating BB nodes for servicing said first data “write” request;
(f) gathering, by said primary BB node, metadata for said data fragments residing in said participating BB nodes, and generating a metadata list;
(g) sending a command along with said metadata list from said primary BB node to said at least one BBNA unit;
(h) subsequently, allocating, by said at least one BBNA unit, a full data stripe buffer sufficient to accommodate said first data “write” request;
(i) transferring said data fragments from said participating BB nodes into said full data stripe buffer in accordance with said metadata list, thereby coalescing said data fragments into a full data stripe in said full data stripe buffer;
and
(j) transferring said full data stripe from said full data stripe buffer of said at least one BBNA unit into said File System.

8 more elements

A system for data migration between data generating entities and a File System, comprising:

a Burst Buffer (BB) tier operatively coupled between at least one data generating entity and the File System, wherein said Burst Buffer tier includes a plurality of BB nodes having data input speed exceeding the data input speed of said File System;
a plurality of File Servers operatively coupled to said BB tier;
at least one Burst Buffer Network Aggregator (BBNA) unit configured at said File System in operative coupling to said plurality of BB nodes, said at least one BBNA unit being configured to coalesce data fragments stored in participating BB nodes from said plurality thereof into a full data stripe under the direction of at least one of said participating BB nodes prior to transfer said full data stripe into said File System.

Dependencies are inferred from the claim text.