Method of measuring test coverage of backend verification runsets and automatically identifying ways to improve the test suite

A machine-readable medium having stored thereon sequences of instructions for implementing a method of measuring test coverage of backend verification runsets, the instruction sequences including instructions that, when executed by a data processing system, cause the data processing system to perform:

• reading a runset;
• preparing a database of all rules and devices identified in reading the runset;
• determining whether output summary files of the runset have been read;
• if all of the output summary files have been read, preparing a coverage report for every rule present in the rule database and outputting a final coverage report;
• if all of the output summary files have not been read, reading a current summary file and constructing a corresponding layer graph as follows:
• determining if a rule that is read in the current summary file is present in the rule database;
• if the rule is not present in the rule database, providing an error message;
• if the rule is present in the rule database, determining a corresponding exact rule from the rule database;
• determining whether the exact rule has previous reference graph;
• if the exact rule does not have a previous reference graph, creating a reference graph for exact rule object;
• if the exact rule does have a previous reference graph, updating the vertices of the reference graph with a value of the corresponding layer graph;
• if the current summary file has been read completely, returning to the above step of determining whether all of the output summary files have been read and proceeding with the subsequent steps;
• and
• if the current summary file has not been read completely, returning to the above step of reading the current summary file and constructing the corresponding layer graph and proceeding with the subsequent steps.