Method and apparatus for transferring archival data among an arbitrarily large number of computer devices in a networked computer environment

In a networked computer system having a communication network interconnecting one or more primary storage devices, one or more secondary storage devices and a plurality of computer devices, each computer device having one or more disks, a method for storing data from one or more computer devices, said method comprising the steps of:

• indicating when a transfer of data from a primary storage device to a secondary storage device should occur;
• and
• for each disk,
• creating index entries for data on a computer device, each index entry specifying a corresponding portion of the data,
• sending a copy of data and associated index entries from the computer device to the secondary storage device using the primary storage device without storing the data or index entries on the primary storage device,
• storing the data in a backup data file on the secondary storage device, the backup data file corresponding to the disk from which the data was transferred,
• forming an index from the index entries such that each index entry specifies a location within the backup data file where data associated with that index entry is located, the index corresponding to the disk from which the data was transferred,
• storing the index for a disk on the secondary storage device,
• sending an index for the disk from the secondary storage device to the primary storage device and storing the index on the primary storage device,
• indicating to the computer device which files on the disk are identified in the index on the primary storage device and when those files were last modified,
• determining which files on the disk have been changed or created since the last modified dates of the files indicated by primary storage device,
• sending an index entry for each file on the disk from the computer device to the primary storage device, each index entry indicating the location of the data corresponding to said each index entry,
• sending data for the files which have been changed or created from the computer device to the primary storage device,
• if there is not an index and a backup data file corresponding to the disk on the primary storage device, forming an index containing the index entries from the computer device and forming a backup data file containing the data from the computer device, the formed index and backup data file corresponding to the disk,
• if there is an index and a backup data file corresponding to the disk, then forming a new index containing the index entries from the computer device, forming a new backup data file, checking each index entry in the new index to see if the data associated with that entry has been changed and, if it has, then incorporating the associated data in the backup data file received from the computer device into the new backup data file, and if it has not been changed, then incorporating the associated data in the backup data file on the primary storage device into the new backup data file, and
• transferring in response to said indicating step one or more backup data files and index corresponding to the disk from the primary storage device to the secondary storage device, replacing the index on the secondary storage device with the index received from the primary storage device, forming a new backup data file, checking each index entry in the new index to see if the data corresponding to that entry has been changed and, if it has, then incorporating the corresponding data in the backup data file received from the primary storage device into the formed new backup data file, and if it has not been changed, then incorporating the corresponding data in the backup data file on the secondary storage device into the formed new backup data file.

An apparatus for storing archival data from one or more computer devices, each computer device having at least one disk, said apparatus comprising:

• a primary storage device;
• a secondary storage device;
• means for interconnecting said primary storage device, said secondary storage device and the computer devices;
• transfer indicating-means for indicating a transfer from said primary storage device to said secondary storage device;
• a full index having a plurality of index entries, each index entry specifying corresponding data;
• a backup initialization having
• means for sending a copy of data and corresponding index entries for a disk from the computer device to the primary storage device,
• means for passing the data and corresponding index entries through the primary storage device to the secondary storage device without storing the data or index entries on the primary storage device,
• means for storing the data in a backup data file on the secondary storage device,
• means for forming an index from the index entries such that each index entry specifies a location within the backup data file where data associated with that index entry is located;
• and
• a backup cycle having
• a first state having
• means for sending an index from the secondary storage device to the primary storage device,
• computer device indicating-means for indicating to the computer device which files are identified in the full index on the primary storage device and the last modified date specified in the index entry for each file,
• means for determining which files on the computer device have been changed or created since the last modified date indicated in the index entry for that file in the full index on the primary device,
• file change indicating means for indicating that the file has been changed or created,
• means for sending an index entry for each file on the disk and data for the files which have changed from the computer device to the primary storage device,
• means for discarding the full index on the primary storage device,
• means for forming a new index containing the index entries from the computer device, and
• means for forming a backup data file containing the data,
• a second state having
• computer device indicating means for indicating to the computer device which files are identified in the full index on the primary storage device and the last modified date specified in the index entry for each file,
• means for determining which files on the computer device have been changed or created since the last modified date indicated in the index entry for that file in the full index on the primary device,
• file change indicating means for indicating that the file has been changed or created,
• means for sending an index entry for each file on the disk and data for the files which have changed from the computer device to the primary storage device,
• means for forming a new index containing the index entries from the computer device and a new backup data file by checking each index entry received from the computer device to see if the data associated with that entry has been changed and, if it has, then incorporating the associated data in the backup data file received from the computer device into the new backup data file, and if it has not been changed, then incorporating the associated data in the backup data file on the primary storage device into the new backup data file;
• and
• a third state having
• means for responding to said indicating means by transferring one or more backup data files and index associated with the disk from the primary storage device to the secondary storage device,
• means for replacing the index on the secondary storage device with the index received from the primary storage device, and
• means for forming a new backup data file by checking each index entry in the new index to see if the data associated with that entry has been changed and, if it has, then incorporating the associated data in the backup data file received from the primary storage device into a new backup data file, and if it has not been changed, then incorporating the associated data in the backup data file on the secondary storage device into a new backup data file.

In a networked computer system having a communication network interconnecting one or more primary storage devices, one or more secondary storage devices and a plurality of computer devices, each computer device having one or more storage resources, a method for transferring data from one or more of the storage resources, said method comprising the steps of:

• initiating and controlling two or more concurrent processes by each active primary storage device, wherein each process initiates a connection between a computer device and a secondary storage device;
• indicating when a transfer of data from the primary storage device to the secondary storage device should occur;
• transferring data from the primary storage device to the secondary storage device in response to such indicating;
• forming for a storage resource of a computer device a new index and a corresponding data file on a secondary storage device by comparing and merging data from a primary storage device index and data files with previously-existing index and data file corresponding to the storage resource, said previously-existing index and data file being stored on the secondary storage device;
• checking whether the primary storage device has an index that defines the data from the storage resource;
• if the primary storage device does not have the index, checking to see if the secondary storage device has such an index;
• if the secondary storage device has a version of the index,
• sending a copy of the secondary storage device version of the index to the computer device via the primary storage device,
• if the secondary storage device does not have such an index,
• copying data and corresponding information from the storage resource of the computer device to the secondary storage device,
• storing the data in a data file on the secondary storage device or external storage devices connected to the secondary storage device, creating an index for accessing the data in the backup data file, and
• storing a copy of the index from the computer device on the secondary storage device;
• determining which data on the storage resource to update;
• creating an index entry for the determined data;
• sending index entries and a copy of the determined data to the primary storage device;
• and
• forming a new index and a new corresponding data file on the primary storage device by comparing and merging newly-acquired determined data from the computer device contained in the most recent index and data files with the previously existing index and data files already stored on the primary storage device.

In a networked computer system having a communication network interconnecting a plurality of devices, the devices including one or more primary storage devices, one or more secondary storage devices and a plurality of computer devices, each computer device having one or more storage resources, a method for transferring among the devices data from the storage resources, said plurality of devices capable of creating a plurality of parallel processes, said method comprising the steps of:

• assigning a secondary storage device to a storage resource;
• performing a backup initialization operation for the storage resource from which data is being transferred, the backup initialization operation forming a first full index and a first backup data file for each storage resource being stored on a secondary storage device;
• repeatedly performing backup cycles for the storage resource from which data is being transferred, during each backup cycle,
• determining from which files and directories on the storage resource data should be transferred;
• transferring from the computer device of the storage resource a second full index for the storage resource, the second full index having an entry for each file and directory on the storage resource, and a second backup data file for the storage resource, the second backup data file including data for the determined storage resource's files and directories, and
• storing the second full index and the second backup data file on the primary storage device, incorporating the second full index and the second backup data file with a previously-stored second full index and a previously-stored second backup data file, if any exist, on the primary storage device, the backup cycles being repeatedly performed until a transfer of data from the primary storage device is indicated;
• indicating when a transfer of data from the primary storage device to a secondary storage device should occur;
• transferring data from the primary storage device to the secondary storage device in response to such indicating, the data comprising with respect to a particular storage resource, the second full index and the second backup data file stored on the primary storage device;
• incorporating the data transferred from the primary storage device into the first full index and the first backup data file stored on the secondary storage device to produce a new first full index having an index entry for each file and directory on the storage resource and a new first backup data file having data for each file and directory, the data for a particular index entry being from the second backup data file if the second backup data file contains data for that particular index entry and being from the first backup data file if the second backup data file does not contain data for that particular index entry;
• and
• storing the new first full index and the new first backup data file on the secondary storage device.