# US 12,118,013 B2 · Data structure based on event compaction and read-offsets > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,013 B2 - Document ID: US12118013B2 - Status: Granted - Kind: B2 - Application number: 18/473,562 - Filed: 2023-09-25 - Published: 2024-01-11 - Granted: 2024-10-15 - Earliest claimed: 2020-12-21 - Assignees: Palantir Technologies Inc. - Applicants: Palantir Technologies Inc. - Inventors: Robert Fink; James Baker; Mark Elliot - Web page: https://patentgrep.com/patent/US12118013B2 ## CPC - G06F 16/254: {Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses} - G06F 16/2358: {Change logging, detection, and notification (replication G06F16/27)} - G06F 16/2365: {Ensuring data consistency and integrity} - G06F 16/2457: with adaptation to user needs - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Disclosed herein are systems and computer-implemented methods that include storing a sequence of events, each event associated with a sequence number indicating a temporal position of an event within the sequence of events; further storing one or more read-offsets that are associated with respective consumers and that indicate the sequence number up to which the respective consumers have read events within the sequence of events; determining a smallest read-offset of all read-offsets; compacting events and/or earlier snapshots with sequence numbers smaller than the smallest read-offset into a new snapshot; and replacing, in the sequence of events, the events and/or earlier snapshots with sequence numbers smaller than the smallest read-offset with the new snapshot. ## Claims ### Claim 1 A system comprising: one or more processors configured to execute program instructions to cause the system to: store a sequence of events, each event associated with a sequence number indicating a temporal position of an event within the sequence of events; further store one or more read-offsets that are associated with respective consumers and that indicate the sequence number up to which the respective consumers have read events within the sequence of events; determine a smallest read-offset of all read-offsets; compact events and/or earlier snapshots with sequence numbers smaller than the smallest read-offset into a new snapshot; and replace, in the sequence of events, the events and/or earlier snapshots with sequence numbers smaller than the smallest read-offset with the new snapshot. ### Claim 2 (depends on 1) The system of claim 1, wherein the new snapshot requires less storage than the events and/or earlier snapshots with sequence numbers smaller than the smallest read-offset. ### Claim 3 (depends on 1) The system of claim 1, wherein compacting events and/or earlier snapshots with sequence numbers smaller than the smallest read-offset into the new snapshot comprises compacting all events and/or earlier snapshots with sequence numbers smaller that the smallest read-offset into the new snapshot. ### Claim 4 (depends on 1) The system of claim 1, wherein one event and/or earlier snapshot at a time is compacted with the new snapshot. ### Claim 5 (depends on 1) The system of claim 1, wherein the sequence of events and the one or more read-offsets are stored in a data structure. ### Claim 6 (depends on 5) The system of claim 5, wherein the data structure forms an event log and is part of a database application layer. ### Claim 7 (depends on 1) The system of claim 1, wherein the new snapshot requires less storage than the events and/or earlier snapshots it represents. ### Claim 8 (depends on 1) The system of claim 1, wherein the new snapshot represents all events and/or earlier snapshots having sequence numbers smaller than the smallest read-offset. ### Claim 9 (depends on 1) The system of claim 1, wherein the new snapshot is stored in an external storage which is arranged to handle large key value stores. ### Claim 10 (depends on 1) The system of claim 1, wherein the consumers are users or processes that are arranged to add events or read events. ### Claim 11 A computer-implemented method comprising: storing a sequence of events, each event associated with a sequence number indicating a temporal position of an event within the sequence of events; further storing one or more read-offsets that are associated with respective consumers and that indicate the sequence number up to which the respective consumers have read events within the sequence of events; determining a smallest read-offset of all read-offsets; compacting events and/or earlier snapshots with sequence numbers smaller than the smallest read-offset into a new snapshot; and replacing, in the sequence of events, the events and/or earlier snapshots with sequence numbers smaller than the smallest read-offset with the new snapshot. ### Claim 12 (depends on 11) The computer-implemented method of claim 11, wherein the new snapshot requires less storage than the events and/or earlier snapshots with sequence numbers smaller than the smallest read-offset. ### Claim 13 (depends on 11) The computer-implemented method of claim 11, wherein compacting events and/or earlier snapshots with sequence numbers smaller than the smallest read-offset into the new snapshot comprises compacting all events and/or earlier snapshots with sequence numbers smaller that the smallest read-offset into the new snapshot. ### Claim 14 (depends on 11) The computer-implemented method of claim 11, wherein one event and/or earlier snapshot at a time is compacted with the new snapshot. ### Claim 15 (depends on 11) The computer-implemented method of claim 11, wherein the sequence of events and the one or more read-offsets are stored in a data structure. ### Claim 16 (depends on 15) The computer-implemented method of claim 15, wherein the data structure forms an event log and is part of a database application layer. ### Claim 17 (depends on 11) The computer-implemented method of claim 11, wherein the new snapshot requires less storage than the events and/or earlier snapshots it represents. ### Claim 18 (depends on 11) The computer-implemented method of claim 11, wherein the new snapshot represents all events and/or earlier snapshots having sequence numbers smaller than the smallest read-offset. ### Claim 19 (depends on 11) The computer-implemented method of claim 11, wherein the new snapshot is stored in an external storage which is arranged to handle large key value stores. ### Claim 20 (depends on 11) The computer-implemented method of claim 11, wherein the consumers are users or processes that are arranged to add events or read events.