Enabling use of analytic functions for distributed storage system data

A system, comprising:

• one or more processors;
• a software agent, wherein
• the software agent is configured to generate an index for a data stream,
• the data stream comprises a plurality of objects,
• the index is configured to indicate a respective location of each of the plurality of objects, and
• the software agent is configured to be executed using the one or more processors;
• and
• a plurality of nodes, wherein
• the plurality of nodes are configured to
• store the data stream,
• perform a plurality of mapping functions, wherein
• the plurality of mapping functions are configured to be performed substantially concurrently,
• each of the plurality of mapping functions are configured to access, based on the index, a respective object of the plurality of objects,
• the performing the each of the plurality of mapping functions is configured to
• access a respective object of the plurality of objects, wherein
• the accessing is based on the index, and
• generate a respective first output of a plurality of first outputs, wherein
• the respective first output comprises the respective object;
• and
• perform one or more reduce functions, wherein
• the one or more reduce functions are configured to use the plurality of first outputs.

A computer program product comprising:

• a plurality of instructions, comprising
• a first set of instructions, executable on a computer system, configured to generate an index, wherein
• the index is generated for a data stream comprising a plurality of objects, and
• the index indicates a location of each of the plurality of objects,
• a second set of instructions, executable on the computer system, configured to perform a plurality of first functions, wherein
• the plurality of first functions are performed substantially concurrently,
• each of the plurality of first functions accesses, based on the index, a respective object of the plurality of objects,
• the performing the plurality of first functions generates a plurality of first outputs, and
• the performing the each of the plurality of first functions comprises
• accessing a respective object of the plurality of objects, wherein the accessing is based on the index, and
• generating a respective first output of the plurality of first outputs, wherein
• the respective first output comprises the respective object,
• a third set of instructions, executable on the computer system, configured to, perform one or more second functions, wherein
• the one or more second functions use the plurality of first outputs;
• and
• a non-transitory computer-readable storage medium, wherein the instructions are encoded in the computer-readable storage medium.