Method and apparatus with software debugging benchmark system
AssigneesSamsung Electronics Co., Ltd., Korea Advanced Institute of Science and Technology
InventorsDo-Ha HWANG, Sung Min KANG, Jae Yong LEE, Shin YOO
Life of this patent
Estimating the term…
Abstract
An operating method of a software debugging benchmark system includes: monitoring pull requests (PRs) created in a project and identifying, in a PR identified by the monitoring, a commit that adds a source code change and test code; determining whether the commit is a potential failure benchmark component by applying predetermined criteria to the commit; based on determining that the commit is a potential failure benchmark component, verifying, in a virtual environment, whether the commit has fixed an actual failure; and based on the verifying, storing a failure PR corresponding to the commit in a database as a failure detection test and failure fix commit.
Claims20
An operating method of a software debugging benchmark system, the operating method comprising:
- monitoring pull requests (PRs) created in a project and identifying, in a PR identified by the monitoring, a commit that adds a source code change and test code;
- determining whether the commit is a potential failure benchmark component by applying predetermined criteria to the commit;
- based on determining that the commit is a potential failure benchmark component, verifying, in a virtual environment, whether the commit has fixed an actual failure;
- and
- based on the verifying, storing a failure PR corresponding to the commit in a database as a failure detection test and failure fix commit.
An electronic device comprising:
- a memory configured to store instructions;
- and
- one or more processors,
- wherein the instructions, when executed by the one or more processors, cause the electronic device to:
- monitor pull requests (PRs) created in a project and identify, in a PR identified by the monitoring, a commit that adds a source code change and test code;
- determine whether the commit is a potential failure benchmark component by applying predetermined criteria to the criteria;
- based on determining that the commit is a potential failure benchmark component, verify, in a virtual environment, whether the commit has fixed an actual failure;
- and
- based on the verifying, store a failure PR corresponding to the commit in a database as a failure detection test and failure fix commit.
A method of collecting code commits, the method comprising:
- detecting commits of source code to a project;
- selecting, from among the detected commits, candidate commits that are determined to be suitable for failure benchmarking;
- for each of the candidate commits, performing a corresponding pre-commit test of the project and a corresponding post-commit test of the project;
- collecting those of the candidate commits whose pre-commit test produces a failure of the pre-commit project and whose corresponding post-commit test does not have the corresponding failure;
- and
- training a large language model based on the collected candidate commits.
Dependencies are inferred from the claim text.