Scalable Code Testing and Benchmarking

A non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to perform steps comprising:

• extracting commits from a history of a first code repository, the history recording a sequence of commits made on the first code repository, each commit of the sequence of commits including at least one change made on at least one code file in the first code repository;
• combining the extracted commits into a sequence of patches, each patch including changes made on one or more code files that occurred during consecutive commits, wherein each patch includes a plurality of commits;
• establishing a connection with a system under test (SUT) having a second code repository, the second code repository corresponding to a historical version of the first code repository before the extracted commits were made;
• and
• sequentially applying the sequence of patches to the second code repository, sequentially applying the sequence of patches including:
• applying a patch in the sequence on the second code repository;
• executing one or more performance evaluation scripts to the SUT to generate performance data of the SUT after applying the patch;
• evaluating the performance data of the SUT to determine whether the SUT performs as expected after applying the patch;
• and
• responsive to determining that the SUT performs as expected after applying the patch, applying a next patch in the sequence to the second code repository.

A computing system, comprising:

• one or more processors;
• and
• a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to perform steps comprising:
• extracting commits from a history of a first code repository, the history recording a sequence of commits made on the first code repository, each commit of the sequence of commits including at least one change made on at least one code file in the first code repository;
• combining the extracted commits into a sequence of patches, each patch including changes made on one or more code files that occurred during consecutive commits, wherein each patch includes a plurality of commits;
• establishing a connection with a system under test (SUT) having a second code repository, the second code repository corresponding to a historical version of the first code repository before the extracted commits were made;
• and
• sequentially applying the sequence of patches to the second code repository, sequentially applying the sequence of patches including:
• applying a patch in the sequence on the second code repository;
• executing one or more performance evaluation scripts to the SUT to generate performance data of the SUT after applying the patch;
• evaluating the performance data of the SUT to determine whether the SUT performs as expected after applying the patch;
• and
• responsive to determining that the SUT performs as expected after applying the patch, applying a next patch in the sequence to the second code repository.