# US 2019/0171550 A1 · System and method for continuous testing and delivery of software

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0171550 A1
- Document ID: US20190171550A1
- Status: Published application
- Kind: A1
- Application number: 16/323,263
- Filed: 2017-06-20
- Published: 2019-06-06
- Earliest claimed: 2016-08-09
- Assignees: SeaLights Technologies LTD
- Applicants: SeaLights Technologies LTD
- Inventors: Alon EIZENMAN; Eran SHER
- Web page: https://patentgrep.com/patent/US20190171550A1

## CPC

- G06F 11/3676: {for coverage analysis}
- G06F 8/60: Software deployment
- G06F 8/71: Version control (security arrangements therefor G06F21/57); Configuration management
- G06F 8/75: Structural analysis for program understanding
- G06F 11/36: Prevention of errors by analysis, debugging or testing of software
- G06F 11/3604: Analysis of software for verifying properties of programs (testing of software G06F11/3668)
- G06F 11/3688: {for test execution, e.g. scheduling of test suites}
- G06F 11/3698: Environments for analysis, debugging or testing of software
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system and method for CI/CT/CD, which is continuous integration/continuous testing/continuous delivery, in which testing is fully integrated to the needs of rapid code development and delivery. The system is determining test coverage of a plurality of tests, comprising a code builder for providing code, a code tester for testing the code with said plurality of tests, and a code test analyzer for analyzing results of said plurality of tests to determine coverage of said code by said plurality of tests.

## Claims

### Claim at position 1

1-96. (canceled)

### Claim 97

A system for CI/CT/CD (continuous integration/continuous testing/continuous delivery), comprising an integrated code development, testing and release system, said integrated system comprising:
a. a build listener running on a computer adapted for detecting code changes in consecutive code builds;
b. a test listener running on a computer adapted for determining tests performed on at least said code builds; and
c. an analysis engine running on a server adapted for determining whether a code build is ready for production based on said code changes and said tests;
wherein said integrated system is adapted to the needs of rapid code development and delivery.

### Claim 98 (depends on 97)

The system of claim 97, seamlessly integrated to the CI/CD system, which is
a. able to raise a flag according to one or more of build quality analysis, code quality reduction or a hole in test coverage;
b. for determining build quality according to a plurality of tests;
c. for determining test coverage for each test specifically or for a plurality of tests by context, wherein said context comprises an execution environment;
d. wherein said plurality of tests are performed on each build of code across every environment tests are executed on the application under test;
e. wherein said tests are performed on the completed compiled build;
f. further comprising providing test coverage for at least one test over a plurality of services; and
g. adapted for providing test coverage for at least one test over a distributed system with a plurality of services.

### Claim 99 (depends on 97)

The system of claim 97,
a. for detecting at least one parameter on build quality selected from the group consisting of detection of at least one change in the code from a previous build to a current build and analysis of at least one test in one test environment to determine whether such change has been tested; assessment of at least one previously test;
assessment of at least one test coverage; assessment of at least one test coverage hole; or a combination thereof;
b. wherein a test coverage hole comprises a determination of one or more of the following parameters: at least a portion of the code that have been modified, has not adequately been tested by test(s) that have been run, which may optionally include but is not limited to zero test coverage, in which no test that has been run tests that portion of the code;
c. further comprising providing test coverage for at least one test over a plurality of services; and
d. adapted for providing test coverage for at least one test over a distributed system with a plurality of services.

### Claim 100 (depends on 97)

The system of claim 97,
a. wherein test coverage is considered according to test environment and wherein test coverage comprises determining a portion of a build and/or an environment covered by at least one test;
b. wherein test coverage is determined in a two step process, in which test coverage is first determined across all environments (or at least a plurality of environments) to avoid repeating footprints, and are then determined according to a specific environment;
c. further comprising providing test coverage for at least one test over a plurality of services; and
d. adapted for providing test coverage for at least one test over a distributed system with a plurality of services.

### Claim 101 (depends on 97)

The system of claim 97,
a. wherein test coverage is considered according to test environment and wherein test coverage comprises both determining a portion of a build and/or an environment covered by at least one test and a portion of a build and/or an environment each test covers;
b. wherein test coverage is determined in a two step process, in which test coverage is first determined across all environments (or at least a plurality of environments) to avoid repeating footprints, and are then determined according to a specific environment;
c. further comprising providing test coverage for at least one test over a plurality of services; and
d. adapted for providing test coverage for at least one test over a distributed system with a plurality of services.

### Claim 102 (depends on 101)

The system of claim 101, wherein test coverage is determined according to environmental footprint, which relates to the build in a particular environment.

### Claim 103 (depends on 101)

The system of claim 101, wherein test coverage is determined first per environment and then on the build level.

### Claim 104 (depends on 97)

The system of claim 97,
a. wherein tests are automatically selected to provide improved test coverage, according to one or more of changes in the code; run-time environment; previous failed tests or test coverage holes; in a shorter time frame; or other priority components (such as a user request, dependency or connection to a particular code area or code family); or a combination thereof;
b. further comprising detecting a change between consecutive builds; and determining test coverage for said change;
c. further comprising providing test coverage for at least one test over a plurality of services; and
d. adapted for providing test coverage for at least one test over a distributed system with a plurality of services.

### Claim 105 (depends on 97)

The system of claim 97,
a. further comprising displaying to a user build quality information one or more of test status, coverage, quality holes, trends, or timing, or a combination thereof;
b. further comprising a Build Quality analyzer for performing a build quality analysis, wherein said build quality analysis includes calculating a build scorecard, wherein said scorecard comprises a calculation of one or more of failed tests, quality holes, decreased trends, or coverage determination; the system further comprising a rule based engine to determine build readiness for production deployment; wherein said engine calculates the metrics for the scorecard for the user and coverage on distributed application; and wherein said build quality metrics include quality analytics such as automated test maintenance analysis, to detect irrelevant tests, redundant or never failing tests;
i. wherein said quality analytics include the detection of problematic code areas, detection of duplicate and similar tests; comparing footprints of production and QA execution to highlight relevant and irrelevant tests, and improve priority; detecting failed tests, to filter out failed test coverage from coverage calculation; and the automatic detection of quality holes for automatically identifying when a quality hole is closed or covered;
c. wherein said determining said scorecard further comprises determining whether a build is ready to be released and sending a notification regarding a build that is not ready to be released;
d. further comprising providing test coverage for at least one test over a plurality of services; and
e. adapted for providing test coverage for at least one test over a distributed system with a plurality of services.

### Claim 106

A system for determining test coverage of a plurality of tests, comprising a code builder for providing code, a code tester for testing the code with said plurality of tests, and a code test analyzer for analyzing results of said plurality of tests to determine coverage of said code by said plurality of tests, wherein said code test analyzer further comprises a coverage hole analyzer for determining if a test coverage hole exists, such that at least a portion of the code is not tested by any of said plurality of tests.

### Claim 107

A method for performing continuous quality management of a code build comprising:
a. providing a quality management platform comprising: a build listener, a test listener and an analysis engine;
b. analyzing said code build by said build listener to determine code changes made from an immediately preceding code build;
c. receiving test results by said test listener; and
d. determining the quality of said code build by said analysis engine.

### Claim 108 (depends on 107)

The method of claim 107, further comprising
a. determining whether to deploy said code build based on said determined quality;
b. performing test selection of relevant tests by said analysis engine according to said build and said code changes, wherein said relevant tests are run automatically;
c. wherein said relevant tests comprise one or more of a unit-test, component tests, integration tests, API tests, UI tests, performance test, or regression tests;
d. wherein said relevant tests are ordered according to a test management framework;
e. wherein said test order is altered based on said build quality;
f. where said determining said quality is based on at least one of:
i. detection of at least one change in the code from a previous build to a current build and analysis of at least one test in one test environment to determine whether such change has been tested;
ii. assessment of at least one previous test;
iii. assessment of at least one test coverage;
iv. assessment of at least one test coverage hole;
v. or a combination thereof; and
g. wherein said assessment of at least one test coverage hole comprises a determination of one or more of the following parameters:
i. at least a portion of the code that has been modified, has not adequately been tested by test(s) that have been run, which may optionally include but is not limited to zero test coverage, or
ii. at least a portion of the code that has been modified, in which no test that has been run tests that portion of the code.

### Claim 109 (depends on 108)

The method of claim 108, wherein said relevant tests are prioritized and wherein tests with higher priority are run before tests with lower priority,
a. wherein said relevant tests are ordered according to a test management framework;
b. wherein said test order is altered based on said build quality;
c. where said determining said quality is based on at least one of:
i. detection of at least one change in the code from a previous build to a current build and analysis of at least one test in one test environment to determine whether such change has been tested;
ii. assessment of at least one previous test;
iii. assessment of at least one test coverage;
iv. assessment of at least one test coverage hole;
v. or a combination thereof; and
d. wherein said assessment of at least one test coverage hole comprises a determination of one or more of the following parameters:
i. at least a portion of the code that has been modified, has not adequately been tested by test(s) that have been run, which may optionally include but is not limited to zero test coverage, or
ii. at least a portion of the code that has been modified, in which no test that has been run tests that portion of the code.

### Claim 110 (depends on 108)

The method of claim 108 wherein said relevant tests are prioritized and wherein only tests with high priority are run;
a. wherein said relevant tests are ordered according to a test management framework;
b. wherein said test order is altered based on said build quality;
c. where said determining said quality is based on at least one of:
i. detection of at least one change in the code from a previous build to a current build and analysis of at least one test in one test environment to determine whether such change has been tested;
ii. assessment of at least one previous test;
iii. assessment of at least one test coverage;
iv. assessment of at least one test coverage hole;
v. or a combination thereof;
d. wherein said assessment of at least one test coverage hole comprises a determination of one or more of the following parameters:
i. at least a portion of the code that has been modified, has not adequately been tested by test(s) that have been run, which may optionally include but is not limited to zero test coverage, or
ii. at least a portion of the code that has been modified, in which no test that has been run tests that portion of the code;
e. wherein tests are automatically selected to provide improved test coverage, according to one or more of:
i. changes in the code;
ii. run-time environment;
iii. previous failed tests or test coverage holes; and
iv. tests run in a shorter time frame.

### Claim 111 (depends on 110)

The method of claim 110, wherein test coverage is considered according to test environment and wherein test coverage comprises determining a portion of a build and/or an environment covered by at least one test,
a. wherein tests are automatically selected to provide improved test coverage, according to one or more of:
i. changes in the code;
ii. run-time environment;
iii. previous failed tests or test coverage holes; and
iv. tests run in a shorter time frame.

### Claim 112 (depends on 110)

The method of claim 110, wherein test coverage is determined in a two step process, in which test coverage is first determined according to a specific environment, and then across all environments (or at least a plurality of environments) in aggregate to avoid repeating footprints;
a. wherein tests are automatically selected to provide improved test coverage, according to one or more of:
i. changes in the code;
ii. run-time environment;
iii. previous failed tests or test coverage holes; and
iv. tests run in a shorter time frame.

### Claim 113 (depends on 110)

The method of claim 110, wherein test coverage is determined according to environmental footprint, which relates to said build in a particular environment,
a. wherein tests are automatically selected to provide improved test coverage, according to one or more of:
i. changes in the code;
ii. run-time environment;
iii. previous failed tests or test coverage holes; and
iv. tests run in a shorter time frame.

### Claim 114 (depends on 110)

The method of claim 110, wherein test coverage is determined first per environment and then on the build level;
a. wherein tests are automatically selected to provide improved test coverage, according to one or more of:
i. changes in the code;
ii. run-time environment;
iii. previous failed tests or test coverage holes; and
iv. tests run in a shorter time frame.

### Claim 115 (depends on 107)

The method of claim 107, further comprising performing test selection of relevant tests by said analysis engine according to said build and said code changes,
a. wherein test selection is based on one of:
i. a calculation based on historical test status and build content;
ii. user input;
iii. environmental changes;
iv. realtime calculations; or
v. realtime dynamic test priority calculation based on realtime coverage data collection;
b. wherein said relevant tests comprise one or more of a unit-test, component tests, or integration tests, functional tests, UI tests, performance tests, API tests;
c. wherein said relevant tests are automatically run across different environments and testing tools;
d. wherein said relevant tests are prioritized and wherein tests with higher priority are run before tests with lower priority, or wherein only tests with high priority are run;
e. wherein said test order is altered based on said build quality;
f. wherein test selection is based on at least one of:
i. test priority
ii. irrelevant tests;
iii. redundant tests;
iv. never failing tests;
v. problematic code areas;
vi. duplicate and similar tests;
vii. failed tests;
viii. differences between test footprint per environment; and
ix. test content change detection;
g. wherein test selection comprises not running a test that does not test at least a part of the code build;
h. wherein test selection comprises replacing an existing test with a different test for at least a part of the code build; and
i. wherein test selection comprises determining that at least a part of the code build has not changed and does not need to be tested.

### Claim 116 (depends on 107)

The method of claim 107, further comprising:
a. receiving test footprints by said test listener;
b. analysis of at least one test in one test environment by the analysis engine;
c. determining build quality based on one of:
i. whether the code changes have been tested;
ii. whether the code changes have not been adequately tested;
iii. assessment of at least one previously performed test;
iv. assessment of at least one test coverage;
v. assessment of at least one test coverage hole; or
vi. a combination thereof;
d. wherein the build quality comprises a build score.
