# US 11,308,092 B2 · Stream processing diagnostics > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,092 B2 - Document ID: US11308092B2 - Status: Granted - Kind: B2 - Application number: 16/437,217 - Filed: 2019-06-11 - Published: 2020-12-17 - Granted: 2022-04-19 - Earliest claimed: 2019-06-11 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Abhishek Shivanna; Kartik Paramasivam; Ray Manpreet Singh Matharu; Samarth Shetty; Srinivasulu Punuru; Yi Pan; Prateek Maheshwari - Web page: https://patentgrep.com/patent/US11308092B2 ## CPC - G06F 16/24568: {Data stream processing; Continuous queries} - G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors - G06F 16/2264: {Multidimensional index structures} - G06F 16/2322: {using timestamps} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The disclosed embodiments provide a system for processing data. During operation, the system receives records of activity within a stream-processing system over a set of event streams, wherein each event stream in the set of event streams contains events related to a corresponding job in the stream-processing system. Next, the system indexes data in the records under a set of keys that include a first key related to jobs in the stream-processing system and a second key related to errors in the stream-processing system. The system then outputs the indexed data for use in analyzing the execution of the stream-processing system. ## Claims ### Claim 1 A method, comprising: receiving, by one or more computer systems, records of activity within a stream-processing system over a set of event streams; indexing, by the one or more computer systems, data in the records under a set of keys comprising a first key related to jobs in the stream-processing system and a second key related to errors in the stream-processing system, wherein indexing the data in the records comprises: when a first record received over the set of event streams produces a first key value in the set of keys that is not found in the indexed data, storing, in the indexed data, a mapping of the first key value to fields that comprise an earliest timestamp that is set to a first timestamp in the first record and a latest timestamp that is also set to the first timestamp in the first record; and for each subsequent record received over the set of event streams that produces a second key value that is identical to the first key value, setting the latest timestamp in the fields to a second timestamp in the subsequent record; and outputting the indexed data. ### Claim 2 (depends on 1) The method of claim 1, wherein indexing the data in the records under the set of keys further comprises: generating a value of the first key based on a job identifier (ID) of a job in the stream-processing system and an instance ID of an instance of the job. ### Claim 3 (depends on 2) The method of claim 2, wherein indexing the data in the records under the set of keys further comprises: mapping the value of the first key to the earliest timestamp, the latest timestamp, a health metric for the job, container IDs for containers associated with the job, and job attributes of the job. ### Claim 4 (depends on 2) The method of claim 2, wherein indexing the data in the records under the set of keys further comprises: generating the value of the first key based on a number of attempts associated with the instance of the job. ### Claim 5 (depends on 4) The method of claim 4, wherein indexing the data in the records under the set of keys further comprises: generating the values of the first key based a container name for a container associated with the job, an incarnation number of the container, and a container ID for the container. ### Claim 6 (depends on 1) The method of claim 1, wherein indexing the data in the records under the set of keys further comprises: generating a value of the second key based on a stack trace and an error type of an error in the stream-processing system. ### Claim 7 (depends on 6) The method of claim 6, wherein indexing the data in the records under the set of keys further comprises: generating the value of the second key based on a job in the stream-processing system and a container associated with the job. ### Claim 8 (depends on 7) The method of claim 7, wherein indexing the data in the records under the set of keys further comprises: mapping the value of the second key to timestamps representing occurrences of the error for the job and the container. ### Claim 9 (depends on 6) The method of claim 6, wherein indexing the data in the records under the set of keys further comprises: mapping the value of the second key to the earliest timestamp, the latest timestamp, and error attributes of the error. ### Claim 10 (depends on 1) The method of claim 1, wherein indexing the data in the records under the set of keys further comprises: generating a value of a third key in the set of keys based on a container ID of a container in the stream-processing system; and mapping the value of the third key to the earliest timestamp, the latest timestamp, a health metric, and task attributes of tasks in the container. ### Claim 11 (depends on 1) The method of claim 1, wherein indexing the data in the records under the set of keys further comprises: deleting a subset of the indexed data with a value of the latest timestamp that is older than a threshold. ### Claim 12 (depends on 1) The method of claim 1, wherein outputting the indexed data comprises: identifying a subset of the indexed data with a value of the latest timestamp that is greater than a last write time of the indexed data to a data store; and writing the subset of indexed data to the data store. ### Claim 13 (depends on 1) The method of claim 1, wherein outputting the indexed data comprises: displaying, in a user interface, an attribute associated with the set of keys and one or more errors associated with the attribute. ### Claim 14 (depends on 1) The method of claim 1, wherein each event stream in the set of event streams contains events related to a corresponding job in the stream-processing system. ### Claim 15 A system comprising: one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the system to: receive records of activity within a stream-processing system over a set of event streams, wherein each event stream in the set of event streams contains events related to a corresponding job in the stream-processing system; index data in the records under a set of keys comprising a first key related to jobs in the stream-processing system and a second key related to errors in the stream-processing system, wherein each value of the second key is generated based on an error type of an error and a stack trace associated with the error; when a first record received over the set of event streams produces a first key value in the set of keys that is not found in the indexed data, store, in the indexed data, a mapping of the first key value to fields that comprise an earliest timestamp that is set to a first timestamp in the first record and a latest timestamp that is also set to the first timestamp in the first record; and output the indexed data. ### Claim 16 (depends on 15) The system of claim 15, wherein indexing the data in the records comprises: for each subsequent record received over the set of event streams that produces a second key value that is identical to the first key value, setting the latest timestamp in the fields to a second timestamp in the subsequent record. ### Claim 17 (depends on 15) The system of claim 15, wherein indexing the data in the records under the set of keys comprises: generating a first value of the first key based on at least one of a job ID of a job in the stream-processing system, an instance ID of an instance of the job, and a number of attempts associated with the instance; and generating a second value of the second key based on a stack trace and an error type of the error in the stream-processing system. ### Claim 18 (depends on 17) The system of claim 17, wherein indexing the data in the records under the set of keys further comprises: mapping the first value of the first key to a health metric for the job, container IDs for containers associated with the job, and job attributes of the job; and mapping the value of the second key to at least one of timestamps representing occurrences of the error and error attributes of the error. ### Claim 19 (depends on 15) The system of claim 15, wherein the records of activity comprise at least one of: a job name; a job identifier (ID); a container name; a container ID; a source version; a host name; a timestamp; an offset; or an error. ### Claim 20 A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, the method comprising: receiving records of activity within a stream-processing system over a set of event streams; indexing data in the records under a set of keys comprising a first key related to jobs in the stream-processing system, a second key related to errors in the stream-processing system, and a third key related to containers in the stream-processing system, wherein a container runs one or more tasks that process partitions of one or more input streams to produce output that is used to populate one or more output streams; wherein indexing the data in the records comprises: when a first record received over the set of event streams produces a first key value in the set of keys that is not found in the indexed data, storing, in the indexed data, a mapping of the first key value to fields that comprise an earliest timestamp that is set to a first timestamp in the first record and a latest timestamp that is also set to the first timestamp in the first record; and for each subsequent record received over the set of event streams that produces a second key value that is identical to the first key value, setting a latest timestamp in the fields to a second timestamp in the subsequent record; outputting the indexed data.