Test sequence for stop-at-first-fail testing

A system for determining an optimized test sequence, wherein the optimized test sequence defines an optimized sequential order of performing multiple tests on an instance of a product using stop-at-first-fail testing, the system comprising:

• an input interface for accessing one or more historical test data instances, wherein each test data instance represents respective outcomes of respective tests previously performed for an instance of the product;
• a processor subsystem configured to:
• access test cost values representing respective costs of performing the respective tests;
• and
• perform an optimization of the sequential order based on the historical test data instances and the respective test cost values, wherein:
• the optimization computes, for respective sequential orders of a plurality of candidate test sequences, a respective expected overall test cost as a respective sum of the respective test cost values for those of the multiple tests expected to be performed under a stop-at-first-fail constraint within the respective candidate test sequence;
• the computation includes, for each of the plurality of candidate test sequences, determining from the one or more historical test data instances and for each test in the respective sequential order of the respective candidate test sequence, a sequence-specific probability that the respective test will not be reached for execution due to failure of one or more tests preceding the test in the respective candidate test sequence, the expected overall test cost for the respective candidate test sequence being determined by weighting the test cost value of each test by a probability that the test will be reached for execution in the respective candidate test sequence;
• and
• the optimization includes selecting, as the optimized test sequence, one of the plurality of candidate test sequences having a lowest expected overall test cost among the computed expected overall test costs;
• and
• an output interface for outputting the selected optimized test sequence.

A testing system, comprising:

• a system for determining an optimized test sequence, wherein the optimized test sequence defines an optimized sequential order of performing multiple tests on an instance of a product using stop-at-first-fail testing, the system for determining the test sequence including:
• an input interface for accessing one or more historical test data instances, wherein each test data instance represents respective outcomes of respective tests previously performed for an instance of the product;
• a processor subsystem configured to:
• access test cost values representing respective costs of performing the respective tests;
• and
• perform an optimization of the sequential order based on the historical test data instances and the respective test cost values, wherein;
• the optimization computes, for respective sequential orders of a plurality of candidate test sequences, a respective expected overall test cost as a respective sum of the respective test cost values for those of the multiple tests expected to be performed under a stop-at-first-fail constraint within the respective candidate test sequence;
• the computation includes, for each of the plurality of candidate test sequences, determining from the one or more historical test data instances and for each test in the respective sequential order of the respective candidate test sequence, a sequence-specific probability that the respective test will not be reached for execution due to failure of one or more tests preceding the test in the respective candidate test sequence, the expected overall test cost for the respective candidate test sequence being determined by weighting the test cost value of each test by a probability that the test will be reached for execution in the respective candidate test sequence;
• and
• the optimization includes selecting, as the optimized test sequence, one of the plurality of candidate test sequences having a lowest expected overall test cost among the computed expected overall test costs;
• and
• an output interface for outputting the selected optimized test sequence;
• and
• one or more test units configured to carry out the multiple tests of the selected optimized test sequence.

A non-transitory computer-readable medium on which are stored data representing instructions that are executable by a processor system for determining a test sequence, wherein the optimized test sequence defines an optimized sequential order of performing multiple tests on an instance of a product using stop-at-first-fail testing, the instructions, when executed by the processor system, causing the processor system to perform the following steps:

• accessing one or more historical test data instances, wherein each test data instance represents respective outcomes of respective tests previously performed for an instance of the product;
• performing an optimization of the sequential order based on the historical test data instances and the respective test cost values, wherein;
• the optimization computes, for respective sequential orders of a plurality of candidate test sequences, a respective expected overall test cost as a respective sum of the respective test cost values for those of the multiple tests expected to be performed under a stop-at-first-fail constraint within the respective candidate test sequence;
• the computation includes, for each of the plurality of candidate test sequences, determining from the one or more historical test data instances and for each test in the respective sequential order of the respective candidate test sequence, a sequence-specific probability that the respective test will not be reached for execution due to failure of one or more tests preceding the test in the respective candidate test sequence, the expected overall test cost for the respective candidate test sequence being determined by weighting the test cost value of each test by a probability that the test will be reached for execution in the respective candidate test sequence;
• and
• the optimization includes selecting, as the optimized test sequence, one of the plurality of candidate test sequences having a lowest expected overall test cost among the computed expected overall test costs;
• and
• outputting the selected optimized test sequence.