File transfer to a distributed file system

A method for transferring a file from a client device to a distributed file system, the method comprising:

• segmenting a file, to be received from the client device, into a plurality of virtual segments, to be stored onto a plurality of data nodes of the distributed file system, wherein at least one virtual segment from amongst the plurality of virtual segments is stored onto each data node from amongst a plurality of data nodes, wherein each of the plurality of virtual segments is a logical portion of the file;
• receiving a request for a plurality of secure connections from the plurality of data nodes of the distributed file system, wherein number of secure connection requests received is equal to actual number of data nodes of the distributed file system, wherein the actual number of data nodes is total number of data nodes used for transferring the file from a client device to the distributed file system, wherein the actual number of data nodes to be utilized is determined based on total number of virtual segments and number of virtual segments per data node,
• wherein the total number of virtual segments is determined based on file size and size of each of the plurality of virtual segments,
• and wherein number of virtual segments per data node is determined based on the total number of virtual segments and a maximum number of data nodes for transferring each of the plurality of virtual segments to the distributed file system;
• transferring the determined number of virtual segments per data node through a corresponding secure connection from the plurality of connections in parallel, wherein each of the determined number of virtual segments is transferred to each of the data node through respective file transfer protocol connection;
• detecting a network failure in the file transfer protocol connection while transferring the determined number of virtual segments;
• retransmitting only the determined number of virtual segments to the distributed file system by re-establishing the failed file transfer protocol connection;
• downloading the virtual segments, and updating name of each virtual segment and location of the virtual segment indicating the data node among the plurality data nodes in a name node of the distributed file system;
• and
• invoking an Application programming interface (API) to logically merge the virtual segments into a single file and also modifying entries in the name node such the plurality of data nodes related to each of the plurality of virtual segments points to the single file.

A non-transitory computer-readable medium having embodied thereon a computer program for executing a method comprising:

• segmenting a file, to be received from the client device, into a plurality of virtual segments, to be stored onto a plurality of data nodes of the distributed file system, wherein at least one virtual segment from amongst the plurality of virtual segments is stored onto each data node from amongst a plurality of data nodes, wherein each of the plurality of virtual segments is a logical portion of the file;
• receiving a request for a plurality of secure connections from the plurality of data nodes of the distributed file system, wherein number of secure connection requests received is equal to actual number of data nodes of the distributed file system, wherein the actual number of data nodes is total number of data nodes used for transferring the file from a client device to the distributed file system, wherein the actual number of data nodes to be utilized is determined based on total number of virtual segments and number of virtual segments per data node,
• wherein the total number of virtual segments is determined based on file size and size of each of the plurality of virtual segments,
• and wherein number of virtual segments per data node is determined based on the total number of virtual segments and a maximum number of data nodes for transferring each of the plurality of virtual segments to the distributed file system;
• transferring the determined number of virtual segments per data node through a corresponding secure connection from the plurality of connections in parallel, wherein each of the determined number of virtual segments is transferred to each of the data node through respective file transfer protocol connection;
• detecting a network failure in the file transfer protocol connection while transferring the determined number of virtual segments;
• and
• retransmitting only the determined number of virtual segments to the distributed file system by re-establishing the failed file transfer protocol connection.