# US 9,679,016 B1 · Checkpointing in distributed streaming platform for real-time applications > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,679,016 B1 - Document ID: US9679016B1 - Status: Granted - Kind: B1 - Application number: 15/077,895 - Filed: 2016-03-22 - Granted: 2017-06-13 - Earliest claimed: 2013-03-11 - Assignees: DataTorrent, Inc. - Applicants: DataTorrent, Inc. - Inventors: Amol J. Kekre; Phu D. Hoang; Chetan Narsude; Thomas Weise - Web page: https://patentgrep.com/patent/US9679016B1 ## CPC - H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) - G06F 9/541: {via adapters, e.g. between incompatible applications} - G06F 11/1402: {Saving, restoring, recovering or retrying} - G06F 16/24568: {Data stream processing; Continuous queries} - G06F 16/248: Presentation of query results - G06F 16/252: {between a Database Management System and a front-end application} - G06F 16/258: {Data format conversion from or to a database} - G06F 16/951: Indexing; Web crawling techniques - H04L 65/60: Network streaming of media packets - H04L 65/765: {intermediate} - H04L 67/55: Push-based network services - G06F 11/00: Error detection; Error correction; Monitoring (error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Software receives a data stream for an application running on a distributed streaming platform over a networked cluster of servers. The software converts the data into a plurality of data tuples structured according to a schema. The software repeatedly emits a plurality of the data tuples as a streaming window, which is separated from other streaming windows by a leading control tuple associated with an ordinal identifier for the streaming window. The streaming window is a sequential sequence of tuples that is associated with a recovery policy. Then the software emits a checkpointing tuple after a plurality of streaming windows. The checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance. Each of the operations is executed by one or more processors in real time or near real time rather than offline. ## Claims ### Claim 1 A method, comprising: receiving a data stream for an application running on a distributed streaming platform over a networked cluster of servers; converting the data into a plurality of data tuples structured according to a schema; repeatedly emitting a plurality of the data tuples as a streaming window, which is separated from other streaming windows by a leading control tuple associated with an ordinal identifier for the streaming window, wherein the streaming window is a sequential sequence of tuples that is associated with a recovery policy; and emitting a checkpointing tuple after a plurality of streaming windows, wherein the checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance, wherein each of the operations is executed by one or more processors in real time or near real time rather than offline. ### Claim 2 (depends on 1) The method of claim 1, wherein checkpointing includes serializing the instance to persistent storage. ### Claim 3 (depends on 2) The method of claim 2, wherein the persistent storage is Hadoop Distributed File System (HDFS) and wherein the serialization uses Kryo. ### Claim 4 (depends on 1) The method of claim 1, wherein the data stream is accessed through a web Application Programming Interface (“API”). ### Claim 5 (depends on 1) The method of claim 1, wherein the data tuples are displayed in a graphical user interface (GUI) following removal of the control tuples. ### Claim 6 (depends on 5) The method of claim 5, wherein the GUI is dashboard for decision support. ### Claim 7 (depends on 1) The method of claim 1, wherein the recovery policy is one of at least once, at most once, or exactly once. ### Claim 8 One or more computer-readable media that are non-transitory and that store one or more programs, wherein the one or more programs, when executed, instruct one or more processors to perform the following operations: receive a data stream for an application running on a distributed streaming platform over a networked cluster of servers; convert the data into a plurality of data tuples structured according to a schema; repeatedly emit a plurality of the data tuples as a streaming window, which is separated from other streaming windows by a leading control tuple associated with an ordinal identifier for the streaming window, wherein the streaming window is a sequential sequence of tuples that is associated with a recovery policy; and emit a checkpointing tuple after a plurality of streaming windows, wherein the checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance, wherein each of the operations is executed in real time or near real time rather than offline. ### Claim 9 (depends on 8) The computer-readable media of claim 8, wherein checkpointing includes serializing the instance to persistent storage. ### Claim 10 (depends on 9) The computer-readable media of claim 9, wherein the persistent storage is Hadoop Distributed File System (HDFS) and wherein the serialization uses Kryo. ### Claim 11 (depends on 8) The computer-readable media of claim 8, wherein the data stream is accessed through a web Application Programming Interface (“APP”). ### Claim 12 (depends on 8) The computer-readable media of claim 8, wherein the data tuples are displayed in a graphical user interface (GUI) following removal of the control tuples. ### Claim 13 (depends on 12) The computer-readable media of claim 12, wherein the GUI is dashboard for decision support. ### Claim 14 (depends on 8) The computer-readable media of claim 8, wherein the recovery policy is one of at least once, at most once, or exactly once. ### Claim 15 A method, comprising: receiving a data stream for an application running on a distributed streaming platform over a networked cluster of servers, wherein the application uses one or more stream modes from the group of stream modes consisting of in-node and in-rack; converting the data into a plurality of data tuples structured according to a schema; repeatedly emitting a plurality of the data tuples as a streaming window, which is separated from other streaming windows by a leading control tuple associated with an ordinal identifier for the streaming window, wherein the streaming window is a sequential sequence of tuples that is associated with a recovery policy; and emitting a checkpointing tuple after a plurality of streaming windows, wherein the checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance, wherein each of the operations is executed by one or more processors in real time or near real time rather than offline. ### Claim 16 (depends on 15) The method of claim 15, wherein checkpointing includes serializing the instance to persistent storage. ### Claim 17 (depends on 16) The method of claim 16, wherein the persistent storage is Hadoop Distributed File System (HDFS) and wherein the serialization uses Kryo. ### Claim 18 (depends on 15) The method of claim 15, wherein the data stream is accessed through a web Application Programming Interface (“API”). ### Claim 19 (depends on 15) The method of claim 15, wherein the data tuples are displayed in a graphical user interface (GUI) following removal of the control tuples. ### Claim 20 (depends on 19) The method of claim 19, wherein the GUI is dashboard for decision support.