Method and system for parallelizing data copy in a distributed file system
AssigneeVMware, Inc.
InventorsChristopher FROST, Jinyuan LI, Mayank RAWAT, Murali VILAYANNUR
Life of this patent
Estimating the term…
Abstract
Multiple servers sharing a distributed file system are used to perform copies of regions of a source file in parallel from a source storage unit to corresponding temporary files at a destination storage unit. These temporary files are then merged or combined into a single file at the destination storage unit in a way that preserves the inode structure and attributes of the source file. A substantial speedup is obtained by copying regions of the file in parallel.
Claims20
A method for increasing the speed of data replication by parallelizing data copy using a coordinating server that is connected to one or more other servers, the method comprising:
- partitioning in place a source file stored in a shared clustered file system of the coordinating server and the other servers into multiple regions including at least first and second regions;
- creating first and second temporary files in the shared clustered file system;
- copying the first region of the source file to the first temporary file, wherein the copying of the first region is performed by the coordinating server;
- directing one of the other servers to copy the second region of the source file to the second temporary file, wherein the copying of the first region is performed during a first time period and the copying of the second region is performed during a second time period, and wherein the first time period and the second time period partially overlap;
- and
- merging the temporary files into a single destination file in the shared clustered file system in a way that preserves a file descriptor data structure and attributes of the source file.
A non-transitory computer-readable storage medium for storing instructions that, when executed by a coordinating server that is connected to one or more other servers, causes the coordinating server to perform a method for increasing the speed of data replication by parallelizing data copy, the method comprising:
- partitioning in place a source file stored in a shared clustered file system of the coordinating server and the other servers into multiple regions including at least first and second regions;
- creating first and second temporary files in the shared clustered file system;
- copying the first region of the source file to the first temporary file;
- directing one of the other servers to copy the second region of the source file to the second temporary file, wherein the copying of the first region is performed during a first time period and the copying of the second region is performed during a second time period, and wherein the first time period and the second time period partially overlap;
- and
- merging the temporary files into a single destination file in the shared clustered file system in a way that preserves a file descriptor data structure and attributes of the source file.
A computer system comprising:
- a cluster of servers, wherein each server comprises a CPU and a system memory, and wherein one of the servers is a coordinating server;
- and
- a shared file system for the cluster of servers,
- wherein the coordinating server is configured to partition in place a source file in the shared file system into multiple regions, create a first temporary file in the shared file system, and copy a first region of the source file to the first temporary file, and
- wherein other servers in the cluster are each configured to create a temporary file in the shared file system and copy subsequent regions of the source file to the temporary file so created, wherein the copying of the first region is performed during a first time period, and wherein the first time period partially overlaps with a second time period during which the copying of one or more of the subsequent regions is performed.
Dependencies are inferred from the claim text.