Scaling a rest cluster using task processing logic
AssigneeNuix Limited
InventorsDaniel Berry, Bruce Brown, Frank Marrone
Life of this patent
Estimating the term…
Abstract
Illustrative embodiments process large amounts of unstructured data by producing, from the unstructured data, a plurality of work items. Each work item includes an item of unstructured data, and corresponding work item metadata. Each work item may be claimed by a corresponding consumer node. In illustrative embodiments, a consumer node claims a work item only when the work item is compatible with the consumer node. The consumer node then processes the claimed work item by performing a specified task on the item of unstructured data associated with the work item.
Claims18
A computer-implemented system for processing a plurality of data items, the system comprising:
- a computer configured to provide a plurality of consumer nodes, each consumer node of the plurality of consumer nodes comprising corresponding node status data, wherein each consumer node of the plurality of consumer nodes is specialized to perform an operation selected from a plurality of operations, and is configured to:
- access a work item queue, the work item queue storing a set of work items, each work item comprising (i) a data item from the plurality of data items and (ii) work item metadata corresponding to said data item;
- identify, using a filter chain, a work item from the plurality of work items that is compatible with the consumer node, wherein the filter chain comprises at least one of
- a case exists filter that assesses whether the consumer node has access to the work item;
- a task status filter that assesses a status tag to determine whether a given task has already been claimed by another consumer node;
- a node tag filter that assesses a consumer node tag and a work item's operation tag to determine whether the consumer node is specialized to perform the operation specified by the work item's operation tag;
- and
- a capacity filter that assesses whether the consumer node is not stressed;
- and
- claim said work item corresponding to the consumer node.
A computer-implemented method of distributing a plurality of data items among a plurality of consumer nodes, each consumer node of the plurality of consumer nodes having corresponding node status data, the method comprising, for each consumer node of the plurality of consumer nodes:
- accessing a work item queue, wherein the work item queue comprises a plurality of work items, each work item of the plurality of work items comprising (i) a data item from the plurality of data items and (ii) work item metadata corresponding to said data item, said work item metadata comprising at least an operation tag specifying an operation to be performed on said data item;
- identifying from the plurality of work items in the work item queue a work item that is compatible with the consumer node by application of a filter chain, wherein the filter chain comprises at least one of
- a case exists filter that assesses whether the consumer node has access to the work item;
- a task status filter that assesses a status tag to determine whether a given task has already been claimed by another consumer node;
- a node tag filter that assesses a consumer node tag and a work item's operation tag to determine whether the consumer node is specialized to perform the operation specified by the work item's operation tag;
- and
- a capacity filter that assesses whether the consumer node is not stressed;
- claiming said work item;
- and
- performing the operation on the data item of the work item.
A non-transitory computer-readable medium storing computer-executable code thereon, the code comprising:
- code for accessing a work item queue, wherein the work item queue comprises a plurality of work items, each work item of the plurality of work items comprising (i) a data item from a plurality of data items and (ii) work item metadata corresponding to said data item, said work item metadata comprising at least an operation tag specifying an operation to be performed on said data item;
- code for identifying from the plurality of work items in the work item queue a work item that is compatible with the consumer node by application of a filter chain, wherein the filter chain comprises at least one of:
- a case exists filter that assesses whether the consumer node has access to the work item;
- a task status filter that assesses a status tag to determine whether a given task has already been claimed by another consumer node;
- a node tag filter that assesses a consumer node tag and a work item's operation tag to determine whether the consumer node is specialized to perform the operation specified by the work item's operation tag;
- and
- a capacity filter that assesses whether the consumer node is not stressed;
- code for claiming said work item;
- and
- code for performing the operation on the data item of the work item.
Dependencies are inferred from the claim text.