Software testing with reliability metric

A system comprising:

• one or more processors;
• memory;
• and
• one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform:
• in a software testing environment comprising a communication link between a software test orchestration tool and a reliability condition dashboard, receiving from the software test orchestration tool software test results at an application level of granularity, wherein the software test results comprise a plurality of reliability status category indicators for a plurality of software applications, wherein the reliability status category indicators comprise an indication of pass and an indication of fail;
• calculating a reliability metric based on the plurality of reliability status category indicators for the plurality of software applications, wherein the reliability metric incorporates non-binary reliability status category indicators configured to have more than two different values, and at least one of the non-binary reliability status category indicators comprises a value indicating that an error result has been manually overridden as having a testing-related error determined to be harmless;
• displaying the reliability metric in a user interface of the reliability condition dashboard, wherein the user interface comprises filter criteria and filters applications having, for a reliability status category indicator, the value indicating that an error result has been manually overridden as having a testing-related error determined to be harmless, whereby the applications having, for a reliability status category indicator, the value indicating that an error result has been manually overridden as having a testing-related error determined to be harmless are not displayed;
• receiving a reliability metric threshold for the reliability metric;
• and
• providing an alert responsive to determining that the reliability metric does not meet the reliability metric threshold.

One or more non-transitory computer-readable media storing computer instructions that when executed by a computing system cause the computing system to perform operations comprising:

• during a software smoke test scenario applying a one-to-one mapping between test cases and a plurality of software applications, in a software testing environment comprising a communication link between a software test orchestration tool and a reliability condition dashboard, receiving from the software test orchestration tool software test results at an application level of granularity, wherein the software test results comprise a plurality of reliability status category indicators for the plurality of software applications comprising a given software application, wherein the reliability status category indicators comprise an indication of pass and an indication of fail;
• receiving from the software test orchestration tool an indication of a plurality of software packages to which the plurality of software applications belong;
• calculating a plurality of package reliability metrics for the plurality of software packages, wherein the package reliability metrics incorporate non-binary reliability status category indicators configured to have more than two different values, and at least one of the non-binary reliability status category indicators comprises a value indicating that an error result has been manually overridden as having a testing-related error determined to be harmless;
• displaying, in the reliability condition dashboard, the plurality of package reliability metrics;
• calculating a first overall reliability metric for the software test results that are displayed;
• receiving a reliability metric threshold for the overall reliability metric;
• and
• responsive to determining that the first overall reliability metric does not meet the reliability metric threshold, blocking an attempted release of the given software application for which the software test results are received;
• displaying the first overall reliability metric in the reliability condition dashboard;
• displaying a filter panel in the reliability condition dashboard configured to receive a filter of the reliability status category indicators;
• receiving, via the filter panel, a selection of a subset of the reliability status category indicators;
• calculating an updated overall reliability metric based on the subset of the reliability status category indicators;
• and
• displaying the updated overall reliability metric;
• wherein applications having, for a reliability status category indicator, the value indicating that an error result has been manually overridden as having a testing-related error determined to be harmless are deemed as passed applications during calculation of the first overall reliability metric or ignored.