System architecture with visual modeling tool for designing and deploying complex models to distributed computing clusters

A system comprising:

• graphical user interface circuitry configured to generate a graphical user interface of a visual distributed computing design workspace, the visual distributed computing design workspace comprising:
• a node palette comprising individually selectable nodes, each with a graphical representation and corresponding to a distributed computing function available on a pre-determined target distributed computing cluster;
• a linking tool for establishing connection links between the individually selectable nodes;
• and
• a digital canvas;
• and
• modeling circuitry responsive to interactions with the graphical user interface to facilitate visually building a data model, the modeling circuitry configured to:
• accept node selections of specific nodes from the node palette;
• place the specific nodes on the digital canvas;
• accept linking selections of dataflow connections between the specific nodes;
• and
• link the specific nodes as specified by the linking selections to form a representation of the data model;
• generate a node action selector on the visual distributed computing design workspace, the node action selector populated with actions available on each of the specific nodes on the digital canvas when one of the specific node is selected, the actions including a ‘show data’ action configured to visualize data flowing out of the selected specific node;
• and
• for each of the selected specific nodes:
• set a node flag to a first state to indicate that the selected specific node has not been previously processed when one of the conditions of the selected specific node has not been processed, has been altered or a node linked upstream of the selected specific node has node flag in the first state is met, or set the node flag to a second state different from the first state to indicate that the node has been previously processed;
• interactive session client circuitry further configured to:
• generate an interactive computing session execution code for the data model;
• spawn a dedicated interactive computing session on an interactive session server to isolate execution of the data model from the pre-determined target distributed computing cluster, the interactive session server being different than the pre-determined target distributed computing cluster;
• execute the interactive computing session execution code within the dedicated interactive computing session, wherein the interactive computing session execution code is executed for the selected specific nodes having the node flag in first state and not executed for the selected specific nodes having the node flag in the second state;
• in response to selection of the ‘show data’ action on the node action selector for the selected one of the specific nodes, provide output data generated by the selected one of the specific nodes, and present the output data in the graphical user interface, wherein the output data is newly generated by executing the execution code associated with the selected one of the specific nodes when the node flag is set to the first state, and the output data is a previously stored output data generated in a prior iteration of processing the execution code when the node flag is set to the second state;
• present execution results for the data model within the dedicated interactive computing session;
• and
• model file building circuitry operable to:
• generate distributed computing execution code for the data model, the distributed computing execution code generated for execution in the pre-determined target distributed computing cluster;
• package the distributed computing execution code into a deployment package;
• and
• deploy the deployment package to the pre-determined target distributed computing cluster.

A method comprising:

• at a visual modeling machine:
• providing a graphical user interface comprising a visual distributed computing design workspace;
• receiving selection of nodes and a linking selection linking one selected node to at least one other node, the linking selection comprising a portion of a data model;
• generating a node action selector on the visual distributed computing design workspace, the node action selector populated with actions available on the node when a specific node is selected, the actions including a ‘show data’ action configured to visualize data flowing out of the selected specific node;
• setting a node flag for the selected specific node to a first state to indicate that the selected specific node has not been previously processed when one of the conditions of the selected specific node has not been processed, been altered or a node linked upstream of the selected specific node has a node flag in the first state is met, or setting the node flag for the selected specific node to a second state different from the first state to indicate that the node has been previously processed;
• generating interactive computing session execution code for the data model;
• spawning a dedicated interactive computing session on an interactive session server to isolate execution of the data model from the pre-determined target distributed computing cluster, the interactive session server being different than the target distributed computing cluster;
• causing the interactive session server to execute the interactive computing session execution code within the dedicated interactive computing session, wherein the interactive computing session execution code is executed for the selected specific node having the node flag in first state and not executed for the selected specific node having the node flag in the second state;
• in response to selecting the ‘show data’ action on the node action selector for the selected specific node, providing output data generated by the selected specific node, and presenting the output data in the graphical user interface, wherein the output data is newly generated by executing the execution code associated with the selected specific node when the node flag is set to the first state and the output data is a previously stored output data generated in a prior iteration of execution when the node flag is set to the second state;
• training the interactive computing session execution code with a subset of data to develop a trained data model;
• generating distributed computing execution code for the trained data model, the distributed computing execution code generated for execution in the pre-determined target distributed computing cluster;
• packaging, by model file building circuitry, the distributed computing execution code into a deployment package;
• and
• deploying, by model deployment circuitry, the deployment package to the pre-determined target distributed computing cluster.