Scaling a rest cluster using task processing logic
AssigneeNuix Limited
InventorsDaniel Berry, Bruce Brown, Frank Marrone
Life of this patent
Estimating the term…
Abstract
Illustrative embodiments process large amounts of unstructured data by producing, from the unstructured data, a plurality of work items. Each work item includes an item of unstructured data, and corresponding work item metadata. Each work item may be claimed by a corresponding consumer node. In illustrative embodiments, a consumer node claims a work item only when the work item is compatible with the consumer node. The consumer node then processes the claimed work item by performing a specified task on the item of unstructured data associated with the work item.
Claims20
A computer-implemented system for processing a plurality of data items, the system comprising:
- a plurality of consumer nodes, each consumer nodes of the plurality of consumer nodes comprising corresponding node status data, wherein each consumer node of the plurality of consumer nodes is specialized to perform an operation selected from the plurality of operations, and is configured to:
- access a work item queue, the work item queue storing a set of work items, each work item comprising a plurality of work items, each work item of the plurality of work items comprising (i) a data item from the plurality of data items and (ii) work item metadata corresponding to said data item;
- identify, using a filter chain, a work item from the plurality of work items that is compatible with the consumer node;
- and
- claim said work item corresponding to the consumer node.
A computer-implemented method of distributing a plurality of data items among a plurality of consumer nodes, each consumer node of the plurality of consumer nodes having corresponding node status data, the method comprising, for each consumer node of the plurality of consumer node:
- accessing a work item queue, wherein the work item queue comprises a plurality of work items, each work item of the plurality of work items comprising (i) a data item from the plurality of data items and (ii) work item metadata corresponding to said data item, said work item metadata comprising at least an operation tag specifying an operation to be performed on said data item;
- identifying from the plurality of work items in the work item queue a work item that is compatible with the consumer node;
- claiming said work item;
- and
- performing the operation on the data item of the work item.
A non-transitory computer-readable medium storing computer-executable code thereon, the code comprising:
- code for accessing a work item queue, wherein the work item queue comprises a plurality of work items, each work item of the plurality of work items comprising (i) a data item from the plurality of data items and (ii) work item metadata corresponding to said data item, said work item metadata comprising at least an operation tag specifying an operation to be performed on said data item;
- code for identifying from the plurality of work items in the work item queue a work item that is compatible with the consumer node;
- code for claiming said work item;
- and
- code for performing the operation on the data item of the work item.
Dependencies are inferred from the claim text.