Performing data storage operations with a cloud environment, including containerized deduplication, data pruning, and data transfer

A non-transitory computer-readable medium storing instructions that when executed by a processor perform a method of deduplicating multiple data objects, wherein the method is performed by one or more computing systems, each computing system including a processor and memory, the method comprising:

• receiving an indication to perform a storage operation to store data to at least one cloud storage location;
• receiving a set of data objects involved in the storage operation;
• determining a size for a container file to utilize when deduplicating the set of objects,
• wherein the container file size is determined at least in part on the latency, bandwidth, or both, associated with a network connection to between the one or more computing systems and the at least one cloud storage location;
• deduplicating, with the computing device, at least some of the data objects in the set in order to create one or more container files containing deduplicated data objects, wherein at least one of the container files has the determined size,
• wherein, for at least some of the data objects in the set, by the one or more computing systems:
• determining if an instance of the data object has already been stored at the at least one cloud storage location;
• if an instance of the data object has already been stored, then:
• determining the location of the instance of the data object;
• and
• storing a reference to the location of the instance of the data object in a first file, wherein the first file stores multiple references, each reference referring to a location of an instance of a data object that has already been stored at the at least one cloud storage location;
• and wherein a reference may comprise a universal resource locator or logical address to the cloud storage location;
• and
• if an instance of the data object has not already been stored, then storing the data object in the created one or more container files, wherein the one or more container files store only a single instance of each data object;
• and
• instructing the storage of the first file and the one or more container files at the cloud storage location.

A non-transitory computer-readable medium having instructions for a method of storing a secondary copy of an original data set on a cloud storage site, wherein executing the instructions with a processor of a computing device causes the processor to perform the method, comprising:

• receiving, with the computing device, a primary copy of an original data set;
• updating, with the computing device, a content index to reflect at least some of data content in the original data set;
• identifying, with the computing device, a target cloud storage site on which to store a secondary copy of the original data set,
• wherein a network connection is to be established between the target cloud storage site and a media file system agent, and
• wherein the established network connection has an associated latency and bandwidth;
• determining, with the computing device, a size for a container file to utilize when deduplicating the primary copy of the original data set,
• wherein the container file size is determined at least in part on the latency, bandwidth, or both, associated with the network connection to be established;
• deduplicating, with the computing device, at least some of the data content in the primary copy in order to create one or more container files containing deduplicated data, wherein at least one of the container files has the determined size;
• establishing, with the computing device, the network connection between the target cloud storage site and the media file system agent;
• and
• transferring, with the computing device, the one or more container files to the target cloud storage site.