# US 2020/0401562 A1 · Parallel processing of filtered transaction logs > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401562 A1 - Document ID: US20200401562A1 - Status: Published application - Kind: A1 - Application number: 16/828,953 - Filed: 2020-03-24 - Published: 2020-12-24 - Earliest claimed: 2019-06-24 - Assignees: Western Digital Technologies, Inc. - Applicants: Western Digital Technologies, Inc. - Inventors: Thomas Demoor; Satish Kumar Chakka; Viswanath Chandrasekara Bharathi - Web page: https://patentgrep.com/patent/US20200401562A1 ## CPC - G06F 16/182: Distributed file systems - G06F 16/1865: {Transactional file systems} - G06F 16/164: {File meta data generation} - G06F 16/1734: {Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs} - G06F 16/1858: {Parallel file systems, i.e. file systems supporting multiple processors} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Example storage systems and methods provide data storage management using parallel processing of filtered transaction logs. Transaction logs are comprised of log entries corresponding to storage operations for at least one storage node. Sets of log entries are sequentially retrieved from the transaction log and filtered through multiple transaction log filters to generate multiple subsets of the log entries. Different metadata operations are executed in parallel using the different filtered subsets of log entries. ## Claims ### Claim 1 A storage system, comprising: at least one storage node configured to execute storage operations; a transaction log configured to store a plurality of log entries corresponding to storage operations for the at least one storage node; a first metadata streamer configured to sequentially retrieve a set of log entries from the transaction log; and a plurality of transaction log filters configured to selectively provide subsets of the set of log entries from the first metadata streamer to a plurality of metadata operations, wherein the metadata operations of the plurality of metadata operations are configured to operate in parallel on the subsets of the set of log entries. ### Claim 2 (depends on 1) The storage system of claim 1, further comprising: a fanout socket configured to: receive the set of log entries from the first metadata streamer; replicate the set of log entries for each transaction log filter of the plurality of transaction log filters; and push a replicated set of log entries to each transaction log filter of the plurality of transaction log filters. ### Claim 3 (depends on 2) The storage system of claim 2, wherein: the fanout socket further includes an expandable interface for interfacing with a variable number of transaction log filters; and the plurality of transaction log filters includes at least three pluggable transaction log filters. ### Claim 4 (depends on 1) The storage system of claim 1, further comprising: a plurality of filtered transaction logs configured to temporarily store the subsets of the set of log entries from the plurality of transaction log filters, wherein each transaction log filter of the plurality of transaction log filters corresponds to an assigned filtered transaction log from the plurality of filtered transaction logs. ### Claim 5 (depends on 4) The storage system of claim 4, further comprising: a plurality of additional metadata streamers configured to fetch log entries from the plurality of filtered transaction logs, wherein each assigned filtered transaction log from the plurality of filtered transaction logs corresponds to an assigned metadata streamer from the plurality of additional metadata streamers. ### Claim 6 (depends on 1) The storage system of claim 1, wherein the plurality of metadata operations are selected from one or more of: replication; notification; metadata indexing; garbage collection; data scrub; and parameter calculation. ### Claim 7 (depends on 1) The storage system of claim 1, further comprising: a reference counter configured to: track a processing number of metadata operations corresponding to the plurality of metadata operations; and release, responsive to completing the plurality of metadata operations, the subsets of the set of log entries from the first metadata streamer. ### Claim 8 (depends on 7) The storage system of claim 7, wherein the reference counter is further configured to receive a completion notification from each metadata operation of the plurality of metadata operations. ### Claim 9 (depends on 1) The storage system of claim 1, wherein: each metadata operation of the plurality of metadata operations is configured with an operation group; a selected operation group includes a plurality of synchronized metadata operations; and subsets of the set of log entries are configured for release when each metadata operation for the selected operation group are all complete. ### Claim 10 (depends on 1) The storage system of claim 1, wherein each transaction log filter of the plurality of transaction log filters includes: an operation type; an expiration condition; and at least one log entry selection criteria. ### Claim 11 A computer-implemented method, comprising: storing a transaction log comprised of a plurality of log entries corresponding to storage operations for at least one storage node; sequentially retrieving a set of log entries from the transaction log; selectively providing, using a first transaction log filter, a first subset of the set of log entries to a first metadata operation; selectively providing, using a second transaction log filter, a second subset of the set of log entries to a second metadata operation; and executing in parallel: the first metadata operation on the first subset of the set of log entries; and the second metadata operation on the second subset of the set of log entries. ### Claim 12 (depends on 11) The computer-implemented method of claim 11, further comprising: replicating the set of log entries for: the first transaction log filter; and the second transaction log filter; pushing a first replicated set of log entries to the first transaction log filter; and pushing a second replicated set of log entries to the second transaction log filter. ### Claim 13 (depends on 11) The computer-implemented method of claim 11, further comprising: adding a third transaction log filter for a third metadata operation; selectively providing, using the third transaction log filter, a third subset of the set of log entries to the third metadata operation; and executing, in parallel with the first metadata operation and the second metadata operation, the third metadata operation on the third subset of the set of log entries. ### Claim 14 (depends on 11) The computer-implemented method of claim 11, further comprising: storing the first subset of the set of log entries in a first filtered transaction log; storing the second subset of the set of log entries in a second filtered transaction log; removing, responsive to completing the first metadata operation on the first subset, the first subset from the first filtered transaction log; and removing, responsive to completing the second metadata operation on the second subset, the second subset from the second filtered transaction log. ### Claim 15 (depends on 14) The computer-implemented method of claim 14, further comprising: fetching, using a first metadata streamer, log entries from the first filtered transaction log; fetching, using a second metadata streamer, log entries from the second filtered transaction log; requesting, by the first metadata operation, log entries from the first filtered transaction log; and requesting, by the second metadata operation, log entries from the second filtered transaction log. ### Claim 16 (depends on 11) The computer-implemented method of claim 11, further comprising: selecting the first metadata operation from a plurality of operation types including two or more of: replication; notification; metadata indexing; garbage collection; data scrub; and parameter calculation; and selecting the second metadata operation from the plurality of operation types, wherein the first metadata operation and the second metadata operation have different operation types from the plurality of operation types. ### Claim 17 (depends on 11) The computer-implemented method of claim 11, further comprising: tracking a processing number of metadata operations for a plurality of metadata operations, wherein the plurality of metadata operations includes the first metadata operation and the second metadata operation; and releasing, responsive to the processing number of metadata operations completing, the set of log entries retrieved from the transaction log. ### Claim 18 (depends on 17) The computer-implemented method of claim 17, further comprising: receiving a completion notification from each metadata operation of the plurality of metadata operations; and determining, responsive to receiving the completion notification from each metadata operation of the plurality of metadata operations, the processing number of metadata operations are complete. ### Claim 19 (depends on 11) The computer-implemented method of claim 11, further comprising: determining a first expiration condition for the first transaction log filter; determining a second expiration condition for the second transaction log filter; and releasing, responsive to the first expiration condition and the second expiration condition being met, the set of log entries. ### Claim 20 A system, comprising: at least one storage node configured to execute storage operations; a transaction log comprised of a plurality of log entries corresponding to storage operations for the at least one storage node; means for sequentially retrieving a set of log entries from the transaction log; means for selectively providing, using a first transaction log filter, a first subset of the set of log entries to a first metadata operation; means for selectively providing, using a second transaction log filter, a second subset of the set of log entries to a second metadata operation; and means for executing in parallel: the first metadata operation on the first subset of the set of log entries; and the second metadata operation on the second subset of the set of log entries.