System to uncover root cause of non-deterministic (flaky) tests

A method for determining the root cause of non-deterministic tests, the method comprising:

• receiving a test that has been identified as non-deterministic;
• running the test, enabling call trace collection to record call traces from the test until at least one execution has succeeded and one execution has failed;
• for each trace in the call trace collection,
• determining whether the trace was part of a passing test or a failing test and
• labeling the trace according to the determination;
• constructing a dynamic call tree for the passing traces;
• constructing a dynamic call tree for the failing traces;
• comparing the dynamic call tree for the passing traces with the dynamic call tree for the failing traces to determine the subtree patterns that only occur in one tree or the other;
• ranking the subtree patterns based on the number of hits in the traces to determine the root cause of the non-deterministic test;
• mapping the subtree patterns to the collected call traces and merging overlapping subtrees.

A non-transitory computer-readable medium having stored therein computer executable code that causes one or more processors to execute the steps of:

• receiving a test that has been identified as non-deterministic;
• running the test, enabling call trace collection to record call traces from the test until at least one execution has succeeded and one execution has failed;
• for each trace in the call trace collection,
• determining whether the trace was part of a passing test or a failing test and
• labeling the trace according to the determination;
• constructing a dynamic call tree for the passing traces;
• constructing a dynamic call tree for the failing traces;
• comparing the dynamic call tree for the passing traces with the dynamic call tree for the failing traces to determine the subtree patterns that only occur in one tree or the other;
• ranking the subtree patterns based on the number of hits in the traces to determine the root cause of the non-deterministic test;
• and
• mapping the subtree patterns to the collected call traces and merging overlapping subtrees.