Managing storage of individually accessible data units

A non-transitory computer-readable medium storing a computer program for managing data, the computer program including instructions for causing a computer to:

• receive at least one group of individually accessible data units over an input device or port, each data unit identified by a key value, with key values of received data units being determined such that the key value identifying a given first data unit that is received before a given second data unit occurs earlier in a sort order than the key value identifying the given second data unit;
• and
• process the received data units for storage in a data storage system, the processing including
• storing a plurality of blocks of data, one or more of the blocks being generated by combining a plurality of the received data units;
• providing an index that includes an entry for each of the blocks, wherein one or more of the entries enable location, based on a provided key value, of a-block that includes a data unit corresponding to the provided key value;
• and
• generating one or more screening data structures associated with the stored blocks for determining, based on a given key value and one or more of the screening data structures, whether to search the stored blocks for a data unit that corresponds to the given key value;
• wherein generating the one or more screening data structures is based on a user-defined probability that a screening data structure correctly or incorrectly identifies a stored block as the location of a data unit.

A system comprising:

• one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations including:
• receiving at least one group of individually accessible data units over an input device or port, each data unit identified by a key value, with key values of received data units being determined such that the key value identifying a given first data unit that is received before a given second data unit occurs earlier in a sort order than the key value identifying the given second data unit;
• and
• processing, by at least one processor, the received data units for storage in a data storage system, the processing including
• storing a plurality of blocks of data, one or more of the blocks being generated by combining a plurality of the received data units;
• providing an index that includes an entry for each of the blocks, wherein one or more of the entries enable location, based on a provided key value, of a block that includes a data unit corresponding to the provided key value;
• and
• generating one or more screening data structures associated with the stored blocks, the one or more screening data structures for determining, based on a given key value and one or more of the screening data structures, whether to search the stored blocks for a data unit that corresponds to the given key value;
• wherein generating the one or more screening data structures is based on a user-defined probability that a screening data structure correctly or incorrectly identifies a stored block as the location of a data unit.