Encoding and transmitting stream data without prior knowledge of data size

A system comprising at least one storage device and at least one hardware processor configured to:

• ingest raw stream data by a client application;
• generate a chunk header that includes a batch size chunk payload size for each of a plurality of chunks, wherein generating the chunk header includes determining a total batch size and setting the batch size chunk payload size based on the total batch size;
• write the raw stream data into each chunk of the plurality of chunks, wherein the raw stream data fills a first portion of each chunk, the first portion corresponding to an offset related to the batch size chunk payload size;
• generate a chunk footer for each chunk that delineates an amount of the raw stream data written into each chunk;
• in response to a size of each chunk ready for sending being less than the batch size chunk payload size, pad each chunk with zeroes to fill each chunk to the batch size chunk payload size;
• generate a data size integer based on the size of each chunk and the padding;
• and
• use the data size integer as each chunk footer;
• and
• send each chunk to a controller for storage without buffering at the client application and based on a time associated with a dynamic ingestion of the raw stream data, wherein each chunk is sent regardless of an amount of raw stream data written into each chunk.

A non-transitory computer readable medium with program instructions stored thereon to perform the following acts:

• ingest raw stream data by a client application;
• generate a chunk header that includes a batch size chunk payload size for each of a plurality of chunks, wherein generating the chunk header includes determining a total batch size and setting the batch size chunk payload size based on the total batch size;
• write the raw stream data into each chunk of the plurality of chunks, wherein the raw stream data fills a first portion of each chunk, the first portion corresponding to an offset related to the batch size chunk payload size;
• generate a chunk footer for each chunk that delineates an amount of the raw stream data written into each chunk;
• in response to a size of each chunk ready for sending being less than the batch size chunk payload size, padding each chunk with zeroes to fill each chunk to the batch size chunk payload size;
• generating a data size integer based on the size of each chunk and the padding;
• and
• using the data size integer as each chunk footer;
• and
• send each chunk to a controller for storage without buffering at the client application and based on a time associated with a dynamic ingestion of the raw stream data, wherein each chunk is sent regardless of an amount of raw stream data written into each chunk.