Efficient Storage Architecture for High Speed Packet Capture

A system comprising:

• a network interface module;
• volatile memory configured to temporarily store data packets received by way of the network interface module;
• high-speed non-volatile memory;
• an interface connecting the system to low-speed non-volatile memory, wherein the high-speed non-volatile memory has lower write latency and less storage capacity than the low-speed non-volatile memory;
• a first set of processors configured to perform a first set of operations that involve:
• (i) reading the data packets from the volatile memory, (ii) arranging the data packets into chunks thereof, each chunk containing a respective plurality of the data packets, and (iii) writing the chunks to the high-speed non-volatile memory;
• and
• a second set of processors configured to perform a second set of operations in parallel to the first set of operations, wherein the second set of operations involve:
• (i) reading the chunks from the high-speed non-volatile memory, (ii) compressing the chunks, (iii) arranging the chunks into blocks thereof, each block containing a respective plurality of the chunks, and (iv) writing, by way of the interface, the blocks to the low-speed non-volatile memory.