Automated test case generation for deep neural networks and other model-based artificial intelligence systems

A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by at least one computing device to cause the at least one computing device to perform steps of:

• selecting sample input data as a test case for a system under test;
• executing the test case on the system under test to obtain a result;
• applying at least a portion of the result to a local explainer function to obtain at least a portion of a corresponding decision tree;
• determining at least one path constraint from the decision tree subject to one or more user-specified constraints, wherein said determining at least one path constraint from the decision tree comprises determining multiple path constraints for respective distinct subsets of branches of the decision tree;
• solving the at least one path constraint to obtain a solution;
• and
• generating at least one other test case for the system under test based at least in part on the solution of the at least one path constraint.

A system comprising:

• a memory;
• and
• at least one processor coupled to the memory and configured for:
• selecting sample input data as a test case for a system under test;
• executing the test case on the system under test to obtain a result;
• applying at least a portion of the result to a local explainer function to obtain at least a portion of a corresponding decision tree;
• determining at least one path constraint from the decision tree subject to one or more user-specified constraints, wherein said determining at least one path constraint from the decision tree comprises determining multiple path constraints for respective distinct subsets of branches of the decision tree;
• solving the at least one path constraint to obtain a solution;
• and
• generating at least one other test case for the system under test based at least in part on the solution of the at least one path constraint.