Stream processing diagnostics

A system comprising:

• one or more processors;
• and
• memory storing instructions that, when executed by the one or more processors, cause the system to:
• receive records of activity within a stream-processing system over a set of event streams, wherein each event stream in the set of event streams contains events related to a corresponding job in the stream-processing system;
• index data in the records under a set of keys comprising a first key related to jobs in the stream-processing system and a second key related to errors in the stream-processing system, wherein each value of the second key is generated based on an error type of an error and a stack trace associated with the error;
• when a first record received over the set of event streams produces a first key value in the set of keys that is not found in the indexed data, store, in the indexed data, a mapping of the first key value to fields that comprise an earliest timestamp that is set to a first timestamp in the first record and a latest timestamp that is also set to the first timestamp in the first record;
• and
• output the indexed data.

A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, the method comprising:

• receiving records of activity within a stream-processing system over a set of event streams;
• indexing data in the records under a set of keys comprising a first key related to jobs in the stream-processing system, a second key related to errors in the stream-processing system, and a third key related to containers in the stream-processing system, wherein a container runs one or more tasks that process partitions of one or more input streams to produce output that is used to populate one or more output streams;
• wherein indexing the data in the records comprises:
• when a first record received over the set of event streams produces a first key value in the set of keys that is not found in the indexed data, storing, in the indexed data, a mapping of the first key value to fields that comprise an earliest timestamp that is set to a first timestamp in the first record and a latest timestamp that is also set to the first timestamp in the first record;
• and
• for each subsequent record received over the set of event streams that produces a second key value that is identical to the first key value,
• setting a latest timestamp in the fields to a second timestamp in the subsequent record;
• outputting the indexed data.