Systems, methods, and data structures for serial data collection and compression

A method of compressing a plurality of messages, the method comprising:

• for each of the plurality of messages, determining a message group among a plurality of message groups, based on a message identification number included in metadata of the message, wherein each of a plurality of in-memory storage areas corresponds to one of the plurality of message groups and comprises a first data structure and a second data structure;
• for each of the plurality of messages, recording in the first data structure for the corresponding message group, a time of receipt of the message in association with data contained in the message, the first data structure arranged according to time of receipt;
• for each of the plurality of messages, recording, in the second data structure for the corresponding message group data contained in the message in association with a time of receipt of the message, the second data structure arranged according to the data;
• for each message group, when a number of messages in the first data structure and the second structure exceeds a prescribed threshold or when a prescribed amount of time has elapsed, evaluating a first layout of the first data structure and a second layout of the second data structure for determining which one of the first layout and the second layout is a more compact layout for the message group;
• for each message group, selecting, based on the evaluation, the more compact layout for storing received messages in the message group;
• and
• generating a compressed message storage record comprising a plurality of message group fields each corresponding to one of the message groups and each formatted with the more compact layout selected for the message group.