System and methods for an overlay disk and cache using portable flash memory

A system for storing data comprising:

• a host computer having a virtual machine running thereon;
• a removable flash memory device operably connected to the host computer, said memory device configured to store changes to a state of a virtual disk image corresponding to the virtual machine, wherein the flash memory device comprises:
• a write log located in the flash memory device structured to contain a sector number and a pointer to a sector buffer;
• at least a cached portion of a fingerprint array located in the flash memory device structured to contain a fingerprint for each chunk on the virtual disk image;
• a chunk cache located in the flash memory device structured to contain a list of chunks resident on the flash memory device;
• and
• a fingerprint array that contains a fingerprint for each memory chunk,
• wherein said host computer further comprises instructions executable on said host computer for performing the following:
• storing changes to a state of the virtual disk image on a removable flash memory device;
• wherein the act of storing changes comprises buffering at least one write to the virtual disk image using the flash memory device, and caching at least one read from the virtual disk image using the flash memory device;
• and
• wherein the read from the virtual disk comprises:
• first inspecting the write-log for a sector number of a sector needed and returning the sector from the flash memory device if found;
• looking up, in the fingerprint array stored on the flash memory device, a chunk-fingerprint corresponding to the sector needed if the sector needed was not found inspecting the write log;
• looking up a chunk corresponding to the chunk fingerprint in the chunk cache and returning the sector from the flash memory device if the chunk is present on the flash memory device;
• looking up the chunk on a host system disk if the chunk was not present on the flash memory device and then returning the sector needed if present on the host system disk;
• and
• fetching the chunk from a remote disk if the chunk was not present on the host system disk and returning the sector needed from the remote disk;
• and
• caching the chunk on the flash memory device;
• and
• further comprising flushing at least a portion of the buffered written data to a remote differencing disk.

A system for managing data comprising:

• means for storing data as a virtual disk image, said means including a static disk (S) and a differencing disk (D) that are stored in a server running a remote virtual disk service;
• and
• means for storing changes to a state of the virtual disk image on a flash memory device in communication with the means for storing data as a virtual disk image, said means for storing comprising a write log for storing an entry for each sector written, a chunk cache list that lists memory chunks currently residing in the portable memory device and a fingerprint array that contains a fingerprint for each memory chunk,
• means for buffering at least one read from the virtual disk image using the portable memory device;
• means for flushing a memory state of the particular virtual machine associated with the virtual disk image to the portable memory device;
• means for associating the memory state with a particular position on the write log on the portable memory device;
• means for buffering at least one write to the virtual disk image using the flash memory device, and means for caching at least one read from the virtual disk image using the flash memory device within the means for storing changes,
• the system further comprising:
• means, in communication with the means for storing changes, for first inspecting a write-log for a sector number of a sector needed and returning the sector from the flash memory device if found;
• means, in communication with the means for storing changes, for looking up, in a fingerprint array stored on the flash memory device, a chunk fingerprint corresponding to the sector needed if the sector needed was not found inspecting the write log;
• means, in communication with the means for storing changes, for looking up a chunk corresponding to the chunk fingerprint in a chunk cache and returning the sector from the flash memory device if the chunk is present on the flash memory device;
• means, in communication with the means for storing changes, for looking up the chunk on a host system disk if the chunk was not present on the flash memory device and then returning the sector needed if present on the host system disk;
• means, in communication with the means for storing changes, for fetching the chunk from a remote disk if the chunk was not present on the host system disk and returning the sector needed from the remote disk;
• and
• means, in communication with the means for storing changes, for caching the chunk on the flash memory device.

A method for retrieving data comprising:

• inserting a portable memory device into a computing device having a virtual machine monitor running on said computing device, said portable memory device comprising a write log for storing an entry for each sector written, a chunk cache that lists memory chunks currently residing in the portable memory device, and a fingerprint array that contains a fingerprint for each memory chunk;
• retrieving a memory state, via the portable memory device, from a previously stored state;
• utilizing the retrieved memory state to create a virtual disk image;
• and
• providing a computing environment to a user based on the utilization of the retrieved memory state,
• wherein the act of utilizing the retrieved memory state to create a virtual disk image comprises:
• buffering at least one write to the virtual disk image using the portable memory device;
• and
• caching at least one read from the virtual disk image using the portable memory device,
• wherein the read from the virtual disk comprises:
• first inspecting the write-log for a sector number of a sector needed and returning the sector from the portable memory device if found;
• looking up, in the fingerprint array stored on the portable memory device, a chunk fingerprint corresponding to the sector needed if the sector needed was not found inspecting the write log;
• looking up a chunk corresponding to the chunk fingerprint in the chunk cache and returning the sector from the portable memory device if the chunk is present on the portable memory device;
• looking up the chunk on a host system disk if the chunk was not present on the portable memory device and then returning the sector needed if present on the host system disk;
• fetching the chunk from a remote disk if the chunk was not present on the host system disk and returning the sector needed from the remote disk;
• and
• caching the chunk on the portable memory device;
• and
• flushing at least a portion of the buffered written data to a remote differencing disk.

A method for managing data comprising:

• caching information from data stored on a networked disk to a portable memory device, the port able memory device comprising a write log for storing an entry for each sector written, a chunk cache that lists memory chunks currently residing in the portable memory device, and a fingerprint array that contains a fingerprint for each memory chunk;
• buffering a write to the networked disk on the portable memory device;
• caching a read from the networked disk to the potable memory device;
• performing a read from the networked disk wherein the read comprises:
• first inspecting the write-log for a sector number of a sector needed and returning the sector from the portable memory device if found;
• looking up, in the fingerprint array stored on the portable memory device, a chunk fingerprint corresponding to the sector needed if the sector needed was not found inspecting the write log;
• looking up a chunk corresponding to the chunk fingerprint in a chunk cache and returning the sector from the portable memory device if the chunk is present on the portable memory device;
• looking up the chunk on a host system disk if the chunk was not present on the portable memory device and then returning the sector needed if present on the host system disk;
• fetching the chunk from a remote disk if the chunk was not present on the host system disk and returning the sector needed from the remote disk;
• and
• caching the chunk on the portable memory device;
• and
• flushing at least a portion of the buffered written data to a remote differencing disk.

A method for reading and writing disk sectors comprising:

• storing changes to a state of a virtual disk image on a removable memory device, the removable memory comprising a write-log for storing an entry for each sector written, a chunk cache that lists memory chunks currently residing in the portable memory device, and a fingerprint array that contains a fingerprint for each memory chunk;
• wherein the act of storing changes comprises buffering at least one write to the virtual disk image using the removable memory device, and caching at least one read from the virtual disk image using the removable memory device;
• and
• wherein the read from the virtual disk comprises:
• first inspecting a write-log for a sector number of a sector needed and returning the sector from a portable memory device if found;
• looking up, in a fingerprint array stored on the portable memory device, a chunk fingerprint corresponding to the sector needed if the sector needed was not found inspecting the write log;
• looking up a chunk corresponding to the chunk fingerprint in a chunk cache and returning the sector from the portable memory device if the chunk is present on the portable memory device;
• looking up the chunk on a host system disk if the chunk was not present on the portable memory device and then returning the sector needed if present on the host system disk;
• fetching the chunk from a remote disk if the chunk was not present on the host system disk and returning the sector needed from the remote disk;
• and
• caching the chunk on the portable memory device;
• and
• further comprising flushing at least a portion of the buffered written data to a remote differencing disk.

A system for reading and writing disk sectors comprising a subsystem in communication with a local operating system of a computer wherein the subsystem comprises:

• means for storing changes to a state of a virtual disk image on a portable memory device, said portable memory device comprising a write log for storing an entry for each sector written, a chunk cache list that lists memory chunks currently residing in the portable memory device and a fingerprint array that contains a fingerprint for each memory chunk;
• means for buffering at least one read from the virtual disk image using the portable memory device;
• means for first inspecting a write-log for a sector number of a sector needed and returning the sector from a portable memory device if found;
• means for looking up, in a fingerprint array stored on the portable memory device, a chunk fingerprint corresponding to the sector needed if the sector needed was not found inspecting the write log;
• means for looking up a chunk corresponding to the chunk fingerprint in a chunk cache and returning the sector from the portable memory device if the chunk is present on the portable memory device;
• means for looking up the chunk on a host system disk if the chunk was not present on the portable memory device and then returning the sector needed if present on the host system disk;
• means for fetching the chunk from a remote disk if the chunk was not present on the host system disk and returning the sector needed from the remote disk;
• means for caching the chunk on the portable memory device;
• and
• means for flushing a memory state of the particular virtual machine associated with the virtual disk image to the portable memory device.

A system for storing data comprising:

• a host computer in communication with a networked disk;
• a portable memory device operably connected to the host computer, said memory device configured to store changes to a state of a disk image corresponding to the networked disk,
• wherein the portable memory device comprises:
• a write log located in the portable memory device structured to contain a sector number and a pointer to a sector buffer;
• at least a cached portion of a fingerprint array located in the portable memory device structured to contain a fingerprint for each chunk on the virtual disk image, and a chunk cache located in the portable memory device structured to contain a list of chunks resident on the portable memory device,
• wherein said host computer further comprises instructions executable on said host computer for performing the following:
• storing changes to a state of the virtual disk image on a removable flash memory device;
• wherein the act of storing changes comprises buffering at least one write to the virtual disk image using the flash memory device, and caching at least one read from the virtual disk image using the flash memory device;
• and
• wherein the read from the virtual disk comprises:
• first inspecting the write-log for a sector number of a sector needed and returning the sector from the flash memory device if found;
• looking up, in the fingerprint array stored on the flash memory device,
• a chunk-fingerprint corresponding to the sector needed if the sector needed was not found inspecting the write log;
• looking up a chunk corresponding to the chunk fingerprint in the chunk cache and returning the sector from the flash memory device if the chunk is present on the flash memory device;
• looking up the chunk on a host system disk if the chunk was not present on the flash memory device and then returning the sector needed if present on the host system disk;
• and
• fetching the chunk from a remote disk if the chunk was not present on the host system disk and returning the sector needed from the remote disk;
• and caching the chunk on the flash memory device,
• further comprising flushing at least a portion of the buffered written data to a remote differencing disk.