Modular systems and methods for managing data storage operations

A modular system for managing data storage operations in a network environment, the system comprising:

• a plurality of storage devices, wherein a first storage device is a magnetic disk and a second storage device is a tape drive;
• a plurality of first modules, each first module operating on a particular one of a plurality of first computing devices and configured to identify data from the particular first computing device to be associated with one or more storage operations, wherein the data comprises files that are organized into data bundles;
• a plurality of second modules operating on a plurality of second computing devices and communicatively coupled to the plurality of first modules and to the plurality of storage devices via a network, the plurality of second modules being configured to receive the identified data from the plurality of first modules and to direct a physical transfer of the identified data to the plurality of storage devices;
• and
• a manager module communicatively coupled to the plurality of first and second modules, the manager module being configured to initiate a first storage operation based on a schedule associated with one or more storage policies, wherein the manager module identifies at least one of the second modules to perform the storage of at least a first data bundle,
• wherein the manager module is further configured to dynamically alter the schedule based on usage of the network and to select at least one first module from the plurality of first modules and at least one second module from the plurality of second modules to perform the first storage operation, said selection of the at least one second module being based on at least loads of the plurality of second modules,
• wherein the at least one second module is configured to store the first data bundle by spanning the files in the first data bundle on at least the first storage device comprising the magnetic disk and the second storage device comprising the tape drive,
• wherein the at least one second module creates and stores a first index, the first index is stored in association with the at least one second module, the first index maintaining at least a physical location of the files in the first data bundle, wherein the physical location in the first index details the physical addresses of the files spanning the first storage device comprising the magnetic disk, and the second storage device comprising the tape drive,
• wherein the at least one second module is further configured to create and forward a second index to the manager module, wherein the second index is stored in association with the manager module, and comprises information that associates the first data bundle with the at least one second module, while retaining the physical addresses of the files in the first index such that the second index associated with the manager module comprises less information regarding the physical location of the files than the first index associated with the at least one second module,
• wherein when retrieval of at least a portion of the first data bundle is requested from a client, the manager module uses the second index stored in association therewith, to identify that the at least one second module is associated with the first data bundle, and forwards the request to the at least one second module,
• the manager module further configured to forward to the at least one second module, communications information by which the at least one second module can establish a communications link with the client,
• the at least one second module configured to use the physical addresses in the second index stored in association with the second module, to retrieve the requested portion of first data bundle spanning the first storage device comprising the magnetic disk and the second storage device comprising the tape drive, and
• the at least one second module further configured to provide the requested portion of the first data bundle to the client via the communications link indentified by the communications information provided by the manager module.