US 10,474,562 B2Granted

Machine learning based ranking of test cases for software development

Assigneesalesforce.com, inc.

InventorsJ. Justin Donaldson, Benjamin Busjaeger, Siddharth Rajaram, Berk Coker, Hormoz Tarevern

Life of this patent

Estimating the term…

Loading patent…

Abstract

An online system ranks test cases run in connection with check-in of sets of software files in a software repository. The online system ranks the test cases higher if they are more likely to fail as a result of defects in the set of files being checked in. Accordingly, the online system informs software developers of potential defects in the files being checked in early without having to run the complete suite of test cases. The online system determines a vector representation of the files and test cases based on a neural network. The online system determines an aggregate vector representation of the set of files. The online system determines a measure of similarity between the test cases and the aggregate vector representation of the set of files. The online system ranks the test cases based on the measures of similarity of the test cases.

Claims20

Dependent claims

A non-transitory computer readable storage medium comprising computer executable code that when executed by one or more processors causes the one or more processors to perform operations comprising:

storing in a software repository, a plurality of files and a plurality of test cases;
training a neural network using historical data comprising sets of files checked in together into the software repository and the test cases that failed as a result of modifications to each set of files, the neural network comprising an input layer, a hidden layer, and an output layer;
receiving a set of files identified for checking in together into the software repository;
for each file from the set of files, extracting a vector representation of the file from the hidden layer of the neural network;
aggregating the vector representations of the set of files to determine an aggregate vector representation of the set of files;
for each test case from the plurality of test cases, extracting a vector representation of the test case from the hidden layer of the neural network;
for each test case, determining a measure of similarity between the vector representation of the test case and the aggregate vector representation of the set of files;
ranking the plurality of test cases based on the measure of similarity between the vector representation of the test case and the aggregate vector representation of the set of files;
and
executing at least a subset of the test cases from the plurality of test cases in an order determined based on the rank.

7 more elements

A computer implemented method for ordering test cases executed for testing modifications to files checked into a software repository, the method comprising:

storing in a software repository, a plurality of files and a plurality of test cases;
receiving a set of files identified for checking in together into the software repository;
for each file from the set of files, extracting a vector representation of the file from a hidden layer of a neural network;
aggregating the vector representations of the set of files to determine an aggregate vector representation of the set of files;
for each test case from the plurality of test cases, extracting a vector representation of the test case from the hidden layer of the neural network;
for each test case, determining a measure of similarity between the vector representation of the test case and the aggregate vector representation of the set of files;
ranking the plurality of test cases based on the measure of similarity between the vector representation of the test case and the aggregate vector representation of the set of files;
and
executing at least a subset of the test cases from the plurality of test cases in an order determined based on the rank.

6 more elements

A computer system comprising:

one or more computer processors;
and
a non-transitory computer readable storage medium comprising computer executable code that when executed by the one or more processors causes the one or more processors to perform operations comprising:
storing in a software repository, a plurality of files and a plurality of test cases;
receiving a set of files identified for checking in together into the software repository;
for each file from the set of files, extracting a vector representation of the file from a hidden layer of a neural network;
aggregating the vector representations of the set of files to determine an aggregate vector representation of the set of files;
for each test case from the plurality of test cases, extracting a vector representation of the test case from the hidden layer of the neural network;
for each test case, determining a measure of similarity between the vector representation of the test case and the aggregate vector representation of the set of files;
ranking the plurality of test cases based on the measure of similarity between the vector representation of the test case and the aggregate vector representation of the set of files;
and
executing at least a subset of the test cases from the plurality of test cases in an order determined based on the rank.

9 more elements

Dependencies are inferred from the claim text.

Facts

Application: 15/710,127Filed 20 Sep 2017
Published: US 2019/0087311 A121 Mar 2019
Granted: 12 Nov 2019
Term ends: Not estimatedThe source records don’t support an estimate.

Classification10 codes

G06F 11/3684
Test management: for test design, e.g. generating new test cases
G06F 8/71
Software maintenance or management: version control ; Configuration management
G06F 11/3688
Test management: for test execution, e.g. scheduling of test suites
G06N 3/045
Architecture, e.g. interconnection topology: combinations of networks