Dynamic throughput ingestion of backup sources

#Embodiments of the present disclosure generally relate to a backup system, and more specifically to dynamically adjusting a rate of ingesting backup data based on resource utilization at a source system being backed up by the backup system.

#Backing up a computer system generally includes copying the contents of the computer system to an external storage system. A basic backup operation may generate a bit-by-bit duplicate of a source system. In some cases, backup operations may compress a backup and/or compare previously backed up data to the data to be backed up, which generally uses additional resources (e.g., processing time). Thus, backing up computer systems is generally a resource-intensive operation.

#Backup systems are generally configured to perform backups based on a set schedule and using a pre-set ingestion rate. These schedules may be configured by a system administrator to begin backup operations when it is assumed that the system is operating at a low load. For example, for an online shopping system, backup operations could be scheduled for a weekday morning period during which the source system typically is lightly loaded.

#So that the manner in which the above recited features of the present disclosure can be understood in detail, a more particular description of the disclosure, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only exemplary embodiments and are therefore not to be considered limiting of its scope, may admit to other equally effective embodiments.

#FIG. 1 illustrates an example of a networked computing environment, according to one embodiment.

#FIG. 2 illustrates a block diagram of an example backup engine, according to one embodiment.

#To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further recitation.

#Embodiments presented herein describe techniques for dynamically managing an ingestion rate for capturing data from a source system based on resource utilization at the source system. Doing so allows a backup system to perform backup operations on a source system based on resources available at the source system without disrupting other computing demands placed on the source system (e.g., the source system's primary workload). Decreasing the ingestion rate allows a backup system to decrease backup resource utilization at a source system and thus allow the source system to allocate additional resources to the source system's primary workload. Conversely, increasing the ingestion rate when the source system is being backed up allows the backup system to use additional or excess capacity to complete backup operations faster.

#In one embodiment, backup systems monitor resource utilization at each source system (e.g., web servers, database servers, development servers, etc.) to be backed up. Resource utilization metrics include CPU usage, memory usage, network throughput, and other relevant metrics. As backup operations are performed, the resource utilization monitors in a backup system can determine whether to increase an ingestion rate (in response to determining that the source system has resources that are currently idle but could be used to accelerate backup operations) or decrease the ingestion rate (in response to determining that activity at the source system has spiked).