Generation of directed acyclic graphs from task routines

A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor to perform operations comprising:

• receive, by the processor and from an input device, a first request to generate a visualization of a first directed acyclic graph (DAG) of multiple task routines of a first analysis routine, wherein:
• the first analysis routine comprises a first job flow of multiple tasks specified in a first job flow definition;
• each task routine of the multiple task routines comprises executable instructions to perform one of the multiple tasks when executed;
• a performance of the first analysis routine comprises execution of a subset of the multiple task routines;
• and
• each task routine comprises comments that specify input and output (I/O) parameters that comprise at least one characteristic of an input to the task routine and at least one characteristic of an output generated during execution of the task routine;
• parse the comments of each task routine of the multiple task routines to identify and retrieve the specification of the I/O parameters of the task routine from the comments;
• generate the visualization of the first DAG to include a visual representation of each task routine of the multiple task routines, wherein each representation of a task routine comprises:
• a task graph object comprising an identifier of the task routine;
• at least one input data graph object that represents an input to the task routine, that is visually connected to the task graph object in the visualization, and that comprises a visual indication of the at least one characteristic of the input;
• and
• at least one output data graph object that represents an output of the task routine that is visually connected to the task graph object in the visualization, and that comprises an indication of the at least one characteristic of the output;
• identify, in the I/O parameters of the multiple task routines, each dependency between an output of one task routine and an input of another task routine;
• for each dependency identified between an output and an input of a pair of task routines of the multiple task routines, arrange the visual representations of each task routine of the pair of task routines horizontally within the visualization and augment the visualization with a dependency marker that visually links the visual representations of the pair of task routines in the visualization;
• vertically arrange, within the visualization, visual representations of one or more pairs of task routines between which no dependency is identified between an output and an input;
• visually output the visualization for display;
• determine whether each task routine identified in a second DAG is stored among a plurality of task routines within at least one federated area, wherein:
• the second DAG is based on the visualization as edited through operation of the input device;
• the second DAG depicts a reuse of a portion of the first job flow depicted in the first DAG to define at least a portion of a second job flow of a second analysis routine derived from the first analysis routine;
• and
• at least a subset of the multiple task routines to be executed in a performance of the first job flow, as indicated in the first DAG, are to be executed in a performance of the second job flow, as indicated in the second DAG;
• determine whether each task data object identified in the second DAG is stored among a plurality of data objects within the at least one federated area;
• and
• in response to a determination that each task routine identified in the second DAG and each data object identified in the second DAG is stored within the at least one federated area, store the second DAG as a second job flow definition among a plurality of job flow definitions within the at least one federated area.

A computer-implemented method comprising:

• receiving, by a processor and from an input device, a first request to generate a visualization of a first directed acyclic graph (DAG) of multiple task routines of a first analysis routine, wherein:
• the first analysis routine comprises a first job flow of multiple tasks specified in a first job flow definition;
• each task routine of the multiple task routines comprises executable instructions to perform one of the multiple tasks when executed;
• a performance of the first analysis routine comprises execution of a subset of the multiple task routines;
• and
• each task routine comprises comments that specify input and output (I/O) parameters that comprise at least one characteristic of an input to the task routine and at least one characteristic of an output generated during execution of the task routine;
• parsing, by the processor, the comments of each task routine of the multiple task routines to identify and retrieve the specification of the I/O parameters of the task routine from the comments;
• generating the visualization of the first DAG to include a visual representation of each task routine of the multiple task routines, wherein each representation of a task routine comprises:
• a task graph object comprising an identifier of the task routine;
• at least one input data graph object that represents an input to the task routine that is visually connected to the task graph object in the visualization, and that comprises a visual indication of the at least one characteristic of the input;
• and
• at least one output data graph object that represents an output of the task routine that is visually connected to the task graph object in the visualization, and that comprises an indication of the at least one characteristic of the output;
• identifying, in the I/O parameters of the multiple task routines, each dependency between an output of one task routine and an input of another task routine;
• for each dependency identified between an output and an input of a pair of task routines of the multiple task routines, arranging the visual representations of each task routine of the pair of task routines horizontally within the visualization and augmenting the visualization with a dependency marker that visually links the visual representations of the pair of task routines in the visualization;
• vertically arranging, within the visualization, visual representations of one or more pairs of task routines between which no dependency is identified between an output and an input;
• visually outputting the visualization for display;
• determining, by the processor, whether each task routine identified in a second DAG is stored among a plurality of task routines within at least one federated area, wherein:
• the second DAG is based on the visualization as edited through operation of the input device;
• the second DAG depicts a reuse of a portion of the first job flow depicted in the first DAG to define at least a portion of a second job flow of a second analysis routine derived from the first analysis routine;
• and
• at least a subset of the multiple task routines to be executed in a performance of the first job flow, as indicated in the first DAG, are to be executed in a performance of the second job flow, as indicated in the second DAG;
• determining, by the processor, whether each task data object identified in the second DAG is stored among a plurality of data objects within the at least one federated area;
• and
• in response to a determination that each task routine identified in the second DAG and each data object identified in the second DAG is stored within the at least one federated area, store the second DAG as a second job flow definition among a plurality of job flow definitions within the at least one federated area.