Method for treating uncertainty estimates for non-deterministic fuzz executions

A computer system configured to treat uncertainty estimates for non-deterministic fuzz executions in a fuzzing system having a fuzzer, the computer system configured to:

• initialize the fuzzing system by setting observations of the fuzzer to a blank state or to one or more initial test cases that are executed on a target program;
• execute test cases;
• for each of the executed test cases:
• create an output report that includes information about a code coverage of the target program;
• repeat the execution of the respective test case a plurality of times to determine variability in execution behavior of the target program and to thereby determine whether the test case leads to jitter or non-deterministic behavior in the target program;
• and
• determine, based on the variability in execution behavior across the plurality of executions, an uncertainty estimate associated with the execution behavior of the respective test case;
• for each of those of the test cases determined to generate jitter or non-deterministic behavior, decide, using an optimizer configured to process the uncertainty estimate determined for the respective test case, how often the respective test case is to be executed based on the uncertainty estimate in order to obtain an optimized estimate of a code path distribution of the target program;
• decide, using the optimizer and based on the uncertainty estimate, whether and how the test cases are mutated to obtain new cases in order to reduce the uncertainty estimate, minimize overall uncertainty, and investigate more areas in the target program;
• and
• add, to a corpus of the test cases, the new test cases which cause a new behavior of the target program.

A non-transitory computer-readable medium on which is stored a computer program for treating uncertainty estimates for non-deterministic fuzz executions in a fuzzing system having a fuzzer, the computer program, when executed by a computer, causing the computer to perform the following steps:

• initialize the fuzzing system by setting observations of the fuzzer to a blank state [blank] or to one or more initial test cases that are executed on a target program;
• execute test cases;
• for each of the executed test cases:
• create an output report that includes information about a code coverage of the target program;
• repeat the execution of the respective test case a plurality of times to determine variability in execution behavior of the target program and to thereby determine whether the test case leads to jitter or non-deterministic behavior in the target program;
• and
• determine, based on the variability in execution behavior across the plurality of executions, an uncertainty estimate associated with the execution behavior of the respective test case;
• for each of those of the test cases determined to generate jitter or non-deterministic behavior, decide, using an optimizer configured to process the uncertainty estimate determined for the respective test case, how often the respective test case is to be executed based on the uncertainty estimate in order to obtain an optimized estimate of a code path distribution of the target program;
• decide, using the optimizer and based on the uncertainty estimate, whether and how the test cases are mutated to obtain new cases in order to reduce the uncertainty estimate, minimize overall uncertainty, and investigate more areas in the target program;
• and
• add, to a corpus of the test cases, the new test cases which cause a new behavior of the target program.