# US 2020/0401320 A1 · Efficient Non-Uniform Object Processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401320 A1 - Document ID: US20200401320A1 - Status: Published application - Kind: A1 - Application number: 16/447,443 - Filed: 2019-06-20 - Published: 2020-12-24 - Earliest claimed: 2019-06-20 - Assignees: Western Digital Technologies, Inc. - Applicants: Western Digital Technologies, Inc. - Inventors: Ameet Pyati; Souvik Roy; Tomy Ammuthan Cheru; Muhammad Tanweer Alam - Web page: https://patentgrep.com/patent/US20200401320A1 ## CPC - G06F 3/067: {Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]} - G06F 11/2094: {Redundant storage or storage space (G06F11/2056 takes precedence)} - G06F 3/0604: {Improving or facilitating administration, e.g. storage management} - G06F 3/0655: {Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices} - G06F 9/5016: {the resource being the memory} - G06F 11/1458: {Management of the backup or restore process} - G06F 11/16: Error detection or correction of the data by redundancy in hardware - G06F 11/20: using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements - G06F 3/061: {Improving I/O performance} - G06F 3/0644: {Management of space entities, e.g. partitions, extents, pools} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The described methods, systems, and other aspects can advantageously provide balanced multi-stage processing of non-uniform object data. An example method may receive a list of buckets. Each of the buckets in the list of buckets can store one or more restorable objects. The method further comprises distributing the list of buckets to the two or more second nodes; determining a number of the one or more restorable objects in each bucket; determining a size of the one or more restorable objects in each bucket; generating batches of to-be-restored data objects based on the determined number of the one or more restorable objects in each bucket and the determined size of the one or more restorable objects in each bucket; and distributing the batches among the two or more second nodes for storage-related task processing. ## Claims ### Claim 1 A distributed storage system, comprising: a cluster of nodes that includes a first node and two or more second nodes; and a distribution component in the first node that is configured to: receive a list of buckets, wherein each bucket in the list of buckets stores one or more restorable objects; distribute the list of buckets to the two or more second nodes; determine a number of the one or more restorable objects in each bucket; determine a size of the one or more restorable objects in each bucket; generate batches of to-be-restored data objects based on the determined number of the one or more restorable objects in each bucket and the determined size of the one or more restorable objects in each bucket; and distribute the batches among the two or more second nodes for storage-related task processing. ### Claim 2 (depends on 1) The system of claim 1, wherein the batches are uniform in size. ### Claim 3 (depends on 1) The system of claim 1, wherein generating the batches includes: selecting one or more data objects from a first bucket in the list of buckets; determining a total volume of the one or more data objects based on a number of the one or more data objects and a size of the one or more data objects; determining that the total volume satisfies a threshold; and responsive to the total volume satisfying the threshold, generating a batch for the one or more data objects. ### Claim 4 (depends on 3) The system of claim 3, wherein the threshold is determined based on optimal utilization of available bandwidths of a processing component of each of the two or more second nodes in the storage-related task processing. ### Claim 5 (depends on 1) The system of claim 1, wherein generating the batches includes: selecting one or more data objects from a first bucket in the list of buckets; determining a total volume of the one or more data objects based on a number of the one or more data objects and a size of the one or more data objects; determining that the total volume dissatisfies a threshold; and responsive to the total volume dissatisfying the threshold, selecting one or more additional data objects from the first bucket or a second bucket such that the total volume is increased to satisfy the threshold. ### Claim 6 (depends on 1) The system of claim 1, wherein the distribution component of the first node is further configured to distribute at least one storage-related tasks to the first node. ### Claim 7 (depends on 1) The system of claim 1, wherein the generated batches are stored in one of a temporary data store and a persistent data store. ### Claim 8 (depends on 1) The system of claim 1, wherein distributing the batches includes: determining a number of processing components associated with the two or more second nodes and a processing factor; and selecting, from the batches, a number of batches for distribution among the processing components of the two or more second nodes, wherein the number of batches is determined based on the number of the processing components and the processing factor. ### Claim 9 (depends on 8) The system of claim 8, wherein the processing factor is greater than or equal to 1. ### Claim 10 A computer-implemented method, comprising: receiving a list of buckets, wherein each bucket in the list of buckets stores one or more restorable objects; distributing the list of buckets to two or more nodes; determining a number of the one or more restorable objects in each bucket; determining a size of the one or more restorable objects in each bucket; generating batches of to-be-restored data objects based on the determined number of the one or more restorable objects in each bucket and the determined size of the one or more restorable objects in each bucket; and distributing the batches among the two or more nodes for storage-related task processing. ### Claim 11 (depends on 10) The computer-implemented method of claim 10, wherein the batches are uniform in size. ### Claim 12 (depends on 10) The computer-implemented method of claim 10, wherein generating the batches includes: selecting one or more data objects from a first bucket in the list of buckets; determining a total volume of the one or more data objects based on a number of the one or more data objects and a size of the one or more data objects; determining that the total volume satisfies a threshold; and responsive to the total volume satisfying the threshold, generating a batch for the one or more data objects. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, wherein the threshold is determined based on optimal utilization of available bandwidths of a processing component of each of the two or more nodes in the storage-related task processing. ### Claim 14 (depends on 10) The computer-implemented method of claim 10, wherein generating the batches includes: selecting one or more data objects from a first bucket in the list of buckets; determining a total volume of the one or more data objects based on a number of the one or more data objects and a size of the one or more data objects; determining that the total volume dissatisfies a threshold; and responsive to the total volume dissatisfying the threshold, selecting one or more additional data objects from the first bucket or a second bucket such that the total volume is increased to satisfy the threshold. ### Claim 15 (depends on 10) The computer-implemented method of claim 10, wherein the generated batches are stored in one of a temporary data store and a persistent data store. ### Claim 16 (depends on 10) The computer-implemented method of claim 10, wherein distributing the batches includes: determining a number of processing components associated with the two or more nodes and a processing factor; and selecting, from the batches, a number of batches for distribution among the processing components of the two or more nodes, wherein the number of batches is determined based on the number of the processing components and the processing factor. ### Claim 17 (depends on 16) The computer-implemented method of claim 16, wherein the processing factor is greater than or equal to 1. ### Claim 18 A system, comprising: means for receiving a list of buckets, wherein each bucket in the list of buckets stores one or more restorable objects; means for distributing the list of buckets to the two or more nodes; means for determining a number of the one or more restorable objects in each bucket; means for determining a size of the one or more restorable objects in each bucket; means for generating batches of to-be-restored data objects based on the determined number of the one or more restorable objects in each bucket and the determined size of the one or more restorable objects in each bucket; and means for distributing the batches among the two or more nodes for storage-related task processing. ### Claim 19 (depends on 18) The system of claim 18, wherein the batches are uniform in size. ### Claim 20 (depends on 18) The system of claim 18, wherein the means for generating the batches includes: means for selecting one or more data objects from a first bucket in the list of buckets; means for determining a total volume of the one or more data objects based on a number of the one or more data objects and a size of the one or more data objects; means for determining that the total volume satisfies a threshold; and means for generating, responsive to the total volume satisfying the threshold, a batch for the one or more data objects.