Method and system for generating backup of a large file and optimizing resource utilization

A production host for hosting virtual machines, comprising:

• a persistent storage storing:
• a data node monitoring information repository;
• a data limit information repository;
• a bandwidth limit information repository;
• and
• a primary data node monitoring information;
• an application agent programmed to:
• receive a backup request specifying a file, wherein the file is stored in a shared storage;
• and
• in response to the backup request:
• retrieve, by a primary data node, properties of the file;
• determine, by the primary data node, resource availability for plurality of secondary data nodes, wherein the primary data node and the plurality of secondary data nodes is operatively connected to the shared storage and a backup storage;
• logically divide, by the primary data node, the file into a plurality of segments using at least the properties of the file;
• and
• coordinate, by the primary data node, the backup of the file using the plurality of segments and the resource availability of the plurality of secondary data nodes,
• wherein coordinating the backup of the file using the plurality of segments and the resource availability of the plurality of secondary data nodes comprises:
• allocating a first set of segments from the plurality of segments to a first secondary data node of the plurality of secondary data nodes;
• allocating a second set of segments from the plurality of segments to a second secondary data node of the plurality of secondary data nodes;
• receiving, by the primary data node, a first completion notification from the first secondary data node and a second completion notification from the second secondary data node;
• generate, in response to receiving the first completion notification and the second completion notification, aggregated metadata specifying storage locations of all the plurality of segments in the backup storage;
• and
• sending, by the primary data node, the aggregated metadata to the backup storage,
• wherein the first secondary data node backs up the first set of segments,
• wherein the second secondary data node backs up the second set of segments, and
• wherein the file comprises the first set of segments and the second set of segments.

A non-transitory computer readable medium (CRM) comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for hosting virtual machines, the method comprising:

• receiving a backup request specifying a file, wherein the file is stored in a shared storage;
• and
• in response to the backup request:
• retrieving, by a primary data node, the properties of the file from a snapshot;
• determining, by the primary data node, querying resource availability for a plurality of a secondary data nodes, by the primary data node, wherein the primary data node and the plurality of secondary data nodes is operatively connected to the shared storage and a backup storage;
• logically dividing, by the primary data node, the file into a plurality of segments, by the primary data node using at least the properties of the file;
• coordinating, by the primary data node, the backup of the file using the plurality of segments and the resource availability of the plurality of secondary data nodes, wherein coordinating the backup of the file using the plurality of segments and the resource availability of the plurality of secondary data nodes comprises:
• allocating a first set of segments from the plurality of segments to a first secondary data node of the plurality of secondary data nodes;
• allocating a second set of segments from the plurality of segments to a second secondary data node of the plurality of secondary data nodes;
• receiving, by the primary data node, a first completion notification from the first secondary data node and a second completion notification from the second secondary data node;
• generate, in response to receiving the first completion notification and the second completion notification, aggregated metadata specifying storage locations of all the plurality of segments in the backup storage;
• and
• sending, by the primary data node, the aggregated metadata to the backup storage,
• wherein the first secondary data node backs up the first set of segments,
• wherein the second secondary data node backs up the second set of segments, and
• wherein the file comprises the first set of segments and the second set of segments.