Visual effects system for "big data" analysis workflow editors, distribution platforms, execution engines, and management systems comprising same

A system for visualization of elements of a workflow that defines an analysis of very large data sets using multi-node compute clusters, the system comprising:

• a processor configured to:
• define within the workflow, based on predetermined grouping criteria, one or more collapsible groups of elements;
• receive, from a user, a request to collapse the one or more collapsible groups of elements;
• collapse the one or more collapsible groups of elements into one or more collapsed groups of elements;
• and
• selectively readjust a layout of the plurality of elements and the one or more collapsed groups of elements;
• and
• a user interface configured to display the workflow, the workflow including a plurality of elements.

A non-transitory computer-readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for visualization of elements of a workflow that defines an analysis of very large data sets using multi-node compute clusters, the method comprising:

• display, via a user interface, the workflow, the workflow including a plurality of elements;
• define within the workflow, based on predetermined grouping criteria, one or more collapsible groups of elements;
• receive, from a user, a request to collapse the one or more collapsible groups of elements;
• collapse the one or more collapsible groups of elements into one or more collapsed groups of elements;
• selectively readjust a layout of the plurality of elements and the one or more collapsed groups of elements;
• receive a request to add a further element to the workflow;
• add the further element to the workflow;
• receive a request to remove a further element from the workflow;
• remove the further element from the workflow;
• receive a request to modify a further element of the workflow;
• modify the further element of the workflow;
• selectively readjust the layout of the workflow;
• add a space saving element to the layout of the workflow;
• receive a request to create a visualization of an element or a group of elements of the workflow, the visualization allowing the user to edit the element or the group of elements while working on the workflow;
• and
• create the visualization of the element or the group of elements of the workflow.

A distribution platform for a workflow that defines an analysis of very large data sets using multi-node compute clusters comprising:

• a user interface configured to allow a user to select a workflow based on one or more parameters associated with the workflow;
• a distribution module configured to enable:
• a user to acquire the workflow;
• and
• importing the workflow into a user environment;
• and
• a management engine for workflows configured to support development of the workflow imported into the user environment.

A computer-implemented method for distributing a workflow that defines an analysis of very large data sets using multi-node compute clusters comprising:

• receiving, by a user interface, a user command to select a workflow based on one or more parameters associated with the workflow;
• enabling a user to acquire the workflow;
• enabling the user to import the workflow into a user environment;
• and
• supporting, by a management engine for workflows, development of the workflow imported into the user environment.

A non-transitory computer-readable medium comprising instructions, which when executed by one or more processors, perform the following operations:

• receive, by a user interface, a user command to select a workflow that defines an analysis of very large data sets using multi-node compute clusters based on one or more parameters associated with the workflow;
• enable a user to acquire the workflow, enable the user to import the workflow into a user environment;
• and
• support, by a management engine for workflows, development of the workflow imported into the user environment.

An event-driven management engine for workflows that define an analysis of very large data sets using multi-node compute clusters comprising:

• a decision node configured to:
• determine that at least one condition is true, wherein the determination that the at least one condition is true comprises running a conditional loop configured to check whether the at least one condition is true;
• and
• based on the determination, selectively activate at least one computational module;
• a fork join queuing cluster configured to:
• allocate the at least one computational module non-sequentially to participant computational nodes in a distributed cloud computing environment;
• and
• process a data set according to predetermined criteria;
• and
• a distributed database configured to:
• store the at least one computational module;
• and
• store the at least one condition associated with the at least one computational module, wherein the at least one computation module is not activated until the at least one condition is true.

A computer-implemented event-driven management method for workflows that define an analysis of very large data sets using multi-node compute clusters comprising:

• storing, by a distributed database, at least one computational module;
• storing, by the distributed database, at least one condition associated with the at least one computational module, wherein the at least one computation module is not activated until the at least one condition is true;
• determining, by a decision node, that the at least one condition is true, wherein the determination that the at least one condition is true comprises running a conditional loop configured to check whether the at least one condition is true;
• based on the determination, selectively activating, by the decision node, the at least one computational module;
• and
• allocating, by a fork join queuing cluster, the at least one computational module non-sequentially to participant computational nodes in a distributed cloud computing environment, wherein the at least one computational module is configured to process a data set according to predetermined criteria.

A non-transitory computer-readable medium comprising instructions, which when executed by one or more processors, perform the following operations:

• store, by a distributed database, at least one computational module;
• store, by the distributed database, at least one condition associated with the at least one computational module, wherein the at least one computation module is not activated until the at least one condition is true;
• determine, by a decision node, that the at least one condition is true, wherein the determination that the at least one condition is true comprises running a conditional loop configured to check whether the at least one condition is true;
• based on the determination, selectively activate, by the decision node, the at least one computational module;
• and
• allocate, by a fork-join queuing cluster, the at least one computational module non-sequentially to participant computational nodes in a distributed cloud computing environment, wherein the at least one computational module is configured to process a data set according to predetermined criteria.