Software testing systems and methods

A method comprising:

• generating a plurality of builds from a plurality of branches of software code, wherein the plurality of builds are executable versions of the plurality of branches;
• applying a plurality of software tests to the plurality of builds based on associated test filters to produce test results, wherein different builds have different associated test filters to control which of the plurality of software tests are applied to an associated build, wherein the test filters each comprise a test filter data structure to control, at least in part, which of the plurality of software tests are applied to an associated build;
• and
• automatically recording software tests that did not pass for a particular build in the associated test filter for the particular build based on an association between the software test that did not pass and a particular runtime feature under test,
• wherein the test filter data structures comprise:
• a record of all core software tests associated with a user modified runtime feature stored as a first file;
• and
• a record of software tests that both passed and failed that are also core software tests, stored as a second file different from the first file.

A method comprising:

• receiving a plurality of branches of software code from a plurality of users, wherein the branches of software code correspond to copies of a main version of software code modified by the plurality of users, and wherein different users modify different portions of the software code corresponding to different runtime features to produce different branches of software code;
• generating a plurality of builds from the plurality of branches of software code, wherein the plurality of builds are executable versions of the plurality of branches having different user modified runtime features;
• applying a plurality of software tests to the plurality of builds based on an associated test filter to produce test results, wherein each software test is associated with at least one of the runtime features, and wherein different builds have different associated test filters, the test filters comprising a test filter data structure to control, at least in part, which of the plurality of software tests are applied to an associated build;
• and
• determining whether each software test was both passed and failed, wherein when a software test associated with a user modified runtime feature of a particular build both passes and fails, then the software test is recorded in the test filter data structure for the particular build,
• wherein the test filter data structure comprises:
• a record of all core software tests associated with the user modified runtime feature stored as a first file;
• and
• a record of software tests that both passed and failed that are also core software tests, stored as a second file different from the first file.

A non-transitory machine-readable medium storing a program executable by at least one processing unit of a device, the program comprising sets of instructions for:

• generating a plurality of builds from a plurality of branches of software code, wherein the plurality of builds are executable versions of the plurality of branches;
• applying a plurality of software tests to the plurality of builds based on associated test filters to produce test results, wherein different builds have different associated test filters to control which of the plurality of software tests are applied to an associated build, wherein the test filters each comprise a test filter data structure to control, at least in part, which of the plurality of software tests are applied to an associated build;
• and
• automatically recording software tests that did not pass for a particular build in the associated test filter for the particular build based on an association between the software test that did not pass and a particular runtime feature under test,
• wherein the test filter data structures comprise:
• a record of all core software tests associated with a user modified runtime feature stored as a first file;
• and
• a record of software tests that both passed and failed that are also core software tests stored as a second file different from the first file.