# US 11,442,647 B2 · Lossless data restore using multiple levels of lightweight snapshots > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,442,647 B2 - Document ID: US11442647B2 - Status: Granted - Kind: B2 - Application number: 16/200,470 - Filed: 2018-11-26 - Published: 2019-11-14 - Granted: 2022-09-13 - Earliest claimed: 2017-11-27 - Assignees: Nutanix, Inc. - Applicants: Nutanix, Inc. - Inventors: Parthasarathy Ramachandran; Bharat Kumar Beedu; Monoreet Mutsuddi; Vanita Prabhu; Mayur Vijay Sadavarte - Web page: https://patentgrep.com/patent/US11442647B2 ## CPC - G06F 3/0647: {Migration mechanisms} - G06F 11/1469: {Backup restoration techniques} - G06F 3/0619: {in relation to data integrity, e.g. data losses, bit errors} - G06F 3/065: {Replication mechanisms} - G06F 3/0659: {Command handling arrangements, e.g. command buffers, queues, command scheduling} - G06F 3/0664: {at device level, e.g. emulation of a storage device or system} - G06F 3/067: {Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]} - G06F 3/0673: {Single storage device} - G06F 11/1448: {Management of the data involved in backup or backup restore} - G06F 2201/815: Virtual - G06F 2201/82: Solving problems relating to consistency - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Systems for high performance restore of data to storage devices. A method embodiment commences upon identifying a plurality of virtual disks to be grouped together into one or more consistency sets. Storage I/O commands for the plurality of virtual disks of the consistency sets are captured into multiple levels of backup data. On a time schedule, multiple levels of backup data for the virtual disks are cascaded by processing data from one or more higher granularity levels of backup data to one or more lower granularity levels of backup data. A restore operation can access the multiple levels of backup data to construct a restore set that is consistent to a designated point in time or to a designated state. Multiple staging areas can be maintained using lightweight snapshot data structures that each comprise a series of captured I/Os to be replayed over other datasets to generate a restore set. ## Claims ### Claim 1 A method comprising: capturing storage I/O commands for a virtual disk; managing multiple levels of backup data for the virtual disk by cascading storage I/O commands from a higher granularity level of backup data to a lower granularity level of backup data, wherein a plurality of snapshots at a first granularity level comprising a plurality of sets of storage I/O commands are cascaded into a snapshot comprising another set of storage I/O commands at a second granularity level that has a granularity that is less than the first granularity level and a storage I/O command of at least one snapshot of the plurality of snapshots at the first granularity level that is superseded by another storage I/O command is omitted from the snapshot at the second granularity level; invoking restoration of the virtual disk to a designated point in time or to a designated state; and accessing a selected level of the multiple levels of the backup data to restore the virtual disk to a state corresponding to the designated point in time or to the designated state. ### Claim 2 (depends on 1) The method of claim 1, wherein the capturing of the storage I/O commands for the virtual disk comprises storing log entries into a command stream, the storage I/O commands for the virtual disk comprise at least one of, a write command to write to a storage area or a write command to add to a storage area, and the virtual disk comprises a plurality of virtual disks that are members of the same consistency group. ### Claim 3 (depends on 1) The method of claim 1, wherein the plurality of snapshots at a first granularity level are deleted after being cascaded into the snapshot at a second level of granularity. ### Claim 4 (depends on 1) The method of claim 1, wherein a portion of multiple snapshots of the plurality of snapshots at the first granularity level superseded by other storage I/O commands are omitted from the snapshot at the second granularity level, and the portion of the multiple snapshots are unnecessary to represent the storage I/O commands when they are not needed for a representation of a result of the storage I/O commands at an end of a time period for the snapshot at a second level of granularity. ### Claim 5 (depends on 1) The method of claim 1, further comprising applying two or more successively smaller checkpoint records to a backup set. ### Claim 6 (depends on 1) The method of claim 1, wherein at least some I/O commands from an I/O buffer are included in a lightweight snapshot. ### Claim 7 (depends on 1) The method of claim 1, wherein at least some I/O commands from a lightweight snapshot are applied to at least one checkpoint record. ### Claim 8 (depends on 1) The method of claim 1, wherein the multiple levels of backup data comprise at least two of, an individual storage I/O command, or a pointer to storage I/O command, or a first lightweight snapshot that bounds a first group of storage I/O commands over a first time period, or a second lightweight snapshot that bounds a second group of storage I/O commands over a second time period, or a full snapshot that comprises previously stored backup data. ### Claim 9 A non-transitory computer readable medium having stored thereon a sequence of instructions which, when stored in memory and executed by a processor causes a set of acts comprising: capturing storage I/O commands for a virtual disk; managing multiple levels of backup data for the virtual disk by cascading storage I/O commands from a higher granularity level of backup data to a lower granularity level of backup data, wherein a plurality of snapshots at a first granularity level comprising a plurality of sets of storage I/O commands are cascaded into a snapshot comprising another set of storage I/O commands at a second granularity level that has a granularity that is less than the first granularity level and a storage I/O command of at least one snapshot of the plurality of snapshots at the first granularity level that is superseded by another storage I/O command is omitted from the snapshot at the second granularity level; invoking restoration of the virtual disk to a designated point in time or to a designated state; and accessing a selected level of the multiple levels of the backup data to restore the virtual disk to a state corresponding to the designated point in time or to the designated state. ### Claim 10 (depends on 9) The computer readable medium of claim 9, wherein the multiple levels of backup data comprise at least two of an individual storage I/O command, or a pointer to a storage I/O command, or a first lightweight snapshot that bounds a first group of storage I/O commands over a first time period, or a second lightweight snapshot that bounds a second group of storage I/O commands over a second time period, or a full snapshot that comprises previously stored backup data. ### Claim 11 (depends on 9) The computer readable medium of claim 9, wherein the capturing of the storage I/O commands for the plurality of the virtual disks comprises storing log entries into a command stream, the storage I/O commands for the virtual disk comprise at least one of, a write command to write to a storage area or a write command to add to a storage area, and the virtual disk comprises a plurality of virtual disks that are members of the same consistency group. ### Claim 12 (depends on 9) The computer readable medium of claim 9, wherein the plurality of snapshots at a first granularity level are deleted after being cascaded into the snapshot at a second level of granularity. ### Claim 13 (depends on 9) The computer readable medium of claim 9, wherein a portion of multiple snapshots of the plurality of snapshots at the first granularity level superseded by other storage I/O commands are omitted from the snapshot at the second granularity level, and the portion of the multiple snapshots are unnecessary to represent the storage I/O commands when they are not needed for a representation of a result of the storage I/O commands at an end of a time period for the snapshot at a second level of granularity. ### Claim 14 (depends on 9) The computer readable medium of claim 9, the set of acts further comprising applying two or more successively smaller checkpoint records to a backup set. ### Claim 15 (depends on 9) The computer readable medium of claim 9, wherein at least some I/O commands from an I/O buffer are included in a lightweight snapshot. ### Claim 16 (depends on 9) The computer readable medium of claim 9, wherein at least some I/O commands from a lightweight snapshot are applied to at least one checkpoint record. ### Claim 17 A system comprising: a storage medium having stored thereon a sequence of instructions; and a processor that execute the sequence of instructions to cause a set of acts comprising, capturing storage I/O commands for a plurality of virtual disks that comprise a consistency group; managing multiple levels of backup data for the virtual disk by cascading storage I/O commands from a higher granularity level of backup data to a lower granularity level of backup data, wherein a plurality of snapshots at a first granularity level comprising a plurality of sets of storage I/O commands are cascaded into a snapshot comprising another set of storage I/O commands at a second granularity level that has a granularity that is less than the first granularity level and a storage I/O command of at least one snapshot of the plurality of snapshots at the first granularity level that is superseded by another storage I/O command is omitted from the snapshot at the second granularity level; invoking restoration of the plurality of virtual disks that comprise the consistency group to a designated point in time or to a designated state; and accessing a selected level of the multiple levels of the backup data to restore the plurality of virtual disks to a state corresponding to the designated point in time or to the designated state. ### Claim 18 (depends on 17) The system of claim 17, wherein the plurality of snapshots at a first granularity level are deleted after being cascaded into the snapshot at a second level of granularity. ### Claim 19 (depends on 17) The system of claim 17, wherein a portion of multiple snapshots of the plurality of snapshots at the first granularity level superseded by other storage I/O commands are from the snapshot at the second granularity level, and the portion of the multiple snapshots are unnecessary to represent the storage I/O commands when they are not needed for a representation of a result of the storage I/O commands at an end of a time period for the snapshot at a second level of granularity. ### Claim 20 (depends on 17) The system of claim 17, wherein the capturing of the storage I/O commands for the plurality of the virtual disks comprises storing log entries into a command stream, the storage I/O commands for the plurality of virtual disks comprise at least one of, a write command to write to a storage area or a write command to add to a storage area, and the virtual disk comprises a plurality of virtual disks that are members of the same consistency group. ### Claim 21 (depends on 17) The system of claim 17, the set of acts further comprising applying two or more successively smaller checkpoint records to a backup set. ### Claim 22 (depends on 17) The system of claim 17, wherein at least some I/O commands from an I/O buffer are included in a lightweight snapshot. ### Claim 23 (depends on 17) The system of claim 17, wherein at least some I/O commands from a lightweight snapshot are applied to at least one checkpoint record. ### Claim 24 (depends on 17) The system of claim 17, wherein the multiple levels of backup data comprise at least two of an individual storage I/O command, or a pointer to a storage I/O command, or a first lightweight snapshot that bounds a first group of storage I/O commands over a first time period, or a second lightweight snapshot that bounds a second group of storage I/O commands over a second time period, or a full snapshot that comprises previously stored backup data.