Scaling a rest cluster using task processing logic
AssigneeNuix Limited
InventorsDaniel Berry, Bruce Brown, Frank Marrone
Life of this patent
Estimating the term…
Abstract
Illustrative embodiments process large amounts of unstructured data by producing, from the unstructured data, a plurality of work items. Each work item includes an item of unstructured data, and corresponding work item metadata. Each work item may be claimed by a corresponding consumer node. In illustrative embodiments, a consumer node claims a work item only when the work item is compatible with the consumer node. The consumer node then processes the claimed work item by performing a specified task on the item of unstructured data associated with the work item.
Claims20
A computer-implemented system for processing a plurality of data items, the system comprising:
- a computer configured to provide a plurality of consumer nodes, each consumer node of the plurality of consumer nodes comprising corresponding node status data, wherein each consumer node of the plurality of consumer nodes is configured to determine whether it is stressed, and is configured:
- consequent to determining that it is stressed, to refrain from claiming any work item;
- and is configured:
- consequent to determining that it is not stressed, to access a work item queue, the work item queue storing a set of work items, each work item comprising (i) a data item from the plurality of data items and (ii) work item metadata corresponding to said data item;
- identify, using a filter chain, a work item from the plurality of work items that is compatible with the consumer node;
- and
- claim said work item corresponding to the consumer node.
A computer-implemented method of distributing a plurality of data items among a plurality of consumer nodes, each consumer node of the plurality of consumer nodes having corresponding node status data, the method comprising, for each consumer node of the plurality of consumer nodes:
- determining, by the consumer node, that the consumer node is not stressed, wherein the consumer node is note stressed when it has resources sufficient to process additional work items;
- and consequently:
- accessing a work item queue, wherein the work item queue comprises a plurality of work items, each work item of the plurality of work items comprising (i) a data item from a plurality of data items and (ii) work item metadata corresponding to said data item;
- identifying from the plurality of work items in the work item queue a work item that is compatible with the consumer node;
- claiming said work item;
- and
- performing an operation on the data item of the work item.
A non-transitory computer-readable medium storing computer-executable code thereon, which code, when executed on a processing unit of a consumer node, causes the consumer node to perform a method for processing a plurality of data items, the method comprising:
- determining whether the node is stressed, and
- consequent to determining that it is stressed, to refrain from claiming any work item, and otherwise to access a work item queue, the work item queue storing a set of work items, each work item comprising (i) a data item from the plurality of data items and (ii) work item metadata corresponding to said data item;
- identify, using a filter chain, a work item from the plurality of work items that is compatible with the consumer node;
- and
- claim said work item corresponding to the consumer node.
Dependencies are inferred from the claim text.