Testing a software application used in a database system

A non-transitory computer-readable storage medium having one or more instructions thereon for testing a software application used in a database system in a multi-tenant database system, the instructions when executed by one or more processors causing the one or more processors to:

• receiving multiple successive versions of the software application where each successive version includes a single new change to the software application and changes from one or more prior versions of the software application;
• running a plurality of tests on a version of the successive versions of the software application to identify one or more test failures;
• and
• responsive to identifying the one or more test failures,
• running a subset of the plurality of tests, where the subset comprises one or more tests that failed, on a first successive version of the software application;
• responsive to determining that the first successive version fails the subset of the plurality of tests, determining that the first successive version caused the one or more test failures of the plurality of tests;
• and
• responsive to determining that the first successive version passes the subset of the plurality of tests, running the subset of the plurality of tests on one or more remaining successive versions of the software application until a particular successive version fails the subset of the plurality of tests, where the particular successive version is determined to have caused the one or more test failures of the plurality of tests.

An apparatus for testing a software application used in a database system in a multi-tenant database, the apparatus comprising:

• a hardware processor;
• and
• one or more stored sequences of instructions which, when executed by the processor, cause the processor to:
• receive multiple successive versions of the software application where each successive version includes a single new change to the software application and changes from one or more prior versions of the software application;
• run a plurality of tests on a version of the successive versions of the software application to identify one or more test failures;
• and
• responsive to identifying the one or more test failures,
• run a subset of the plurality of tests, where the subset comprises one or more tests that failed, on a first successive version of the software application;
• responsive to determining that the first successive version fails the subset of the plurality of tests, determine that the first successive version caused the one or more test failures of the plurality of tests;
• and
• responsive to determining that the first successive version passes the subset of the plurality of tests, run the subset of the plurality of tests on one or more remaining successive versions of the software application until a particular successive version fails the subset of the plurality of tests, where the particular successive version is determined to have caused the one or more test failures of the plurality of tests.