# US 2014/0196012 A1 · Methods, devices, systems, and non-transitory machine-readable medium for performing an automated calibration for testing of a computer software application > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0196012 A1 - Document ID: US20140196012A1 - Status: Published application - Kind: A1 - Application number: 14/149,503 - Filed: 2014-01-07 - Published: 2014-07-10 - Earliest claimed: 2013-01-07 - Assignees: Appvance Inc. - Applicants: Appvance Inc. - Inventors: Frank Cohen - Web page: https://patentgrep.com/patent/US20140196012A1 ## CPC - G06F 11/3672: {Test management} - G06F 11/3698: Environments for analysis, debugging or testing of software - G06F 11/362: Debugging of software - G06F 11/366: {using diagnostics (G06F11/0703 takes precedence)} - G06F 11/3684: {for test design, e.g. generating new test cases} - G06F 11/3688: {for test execution, e.g. scheduling of test suites} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The present invention enables an automated testing of computer software applications for efficiently determining the quality and/or performance characteristics of the computer software applications and assists testing designers when determining software application scalability and performance under load. Embodiments of the present invention may be implemented to, for example, determine how many test servers are required to test computer software applications for correct function under the load of many concurrently active users, and periodically test and/or monitor computer software applications for quality control and/or other purposes. Additionally, embodiments of the present invention may be implemented to, for example calibrate a set of one or more test servers for testing a computer software application. ## Claims ### Claim 1 A method comprising: determining, by a computer system, a count of concurrently running virtual users for performing a transaction on software application, wherein each virtual user included within the count is enabled to perform the transaction during a virtual user session; determining, by the computer system, a test regime for testing the software application, the test regime including a number of series of tests, each series including a plurality of the transactions, wherein each series of tests of the test regime is performed by different a percentage of the concurrently running virtual user count; transmitting, by the computer system, the test regime from the processor to a test server, the test server being enabled to communicate with the computer system and execute the test regime; receiving, from the test server by the computer system, test results generated by execution of the test regime by the test server; analyzing, by the computer system, the test results to determine a level of virtual user throughput for each series of tests; determining, by the computer system, a maximum virtual user count for the test server responsively to the analysis, the maximum virtual user count being the maximum number of virtual users the test server can employ and maintain linear scalability of throughput between the series of tests of the test regime; determining, by the computer system, a size of a set of one or more test servers to perform the test regime while maintaining linear scalability of throughput between the series of tests of the test regime using the maximum virtual user count; and calibrating, by the computer system, the set of test servers required to execute the test regime, the calibration of the set of test servers including transmission of the testing regime to the set of test servers. ### Claim 2 (depends on 1) The method of claim 1, further comprising: receiving, at the computer system, test results generated by execution of the test regime by the set of servers from the set of test servers; analyzing, by the computer system, the received test results; adjusting, by the computer system, the count of concurrently running virtual users responsively to the analysis; and transmitting, by the computer system, the adjusted count of concurrently running virtual users to the test server. ### Claim 3 (depends on 1) The method of claim 1, further comprising: receiving, at the computer system, test results generated by execution of the test regime by the set of servers; analyzing, by the computer system, the received test results; adjusting, by the computer system, the calibration of the set of test servers responsively to the analysis; and transmitting, by the computer system, the adjusted calibration of the set of test servers to the test server. ### Claim 4 (depends on 1) The method of claim 1, further comprising: receiving, by the computer system, test results generated by execution of the test regime by the set of servers; analyzing, by the computer system, the received test results; adjusting, by the computer system, the number of series of tests responsively to the analysis; and transmitting, by the computer system, the adjusted number of series of tests to the test server. ### Claim 5 (depends on 1) The method of claim 1, further comprising: receiving, by the computer system, test results generated by execution of the test regime by the set of servers; analyzing, by the computer system, the received test results; adjusting, by the computer system, the transaction responsively to the analysis; and transmitting, by the computer system, the adjusted transaction to the test server. ### Claim 6 (depends on 1) The method of claim 1, wherein the determination of the count of concurrently running virtual users is made using at least one of a default value, a value associated with a characteristic of the software application, and a value associated with a characteristic of the test server. ### Claim 7 (depends on 1) The method of claim 1, further comprising: determining, by the computer system prior to transmission of test regime to the test server, a time period for executing the virtual user session by the test server; determining, by the computer system, a time period for executing the series of tests based on the time period for executing the virtual user session; determining, by the computer system, a time period for executing the test regime based on the time period for executing the series of tests; updating, by the computer system, the test regime to include the time period for executing the test regime; and transmitting, by the computer system, the updated test regime to the test server. ### Claim 8 A system comprising: a computer system configured to determine a count of concurrently running virtual users for performing a transaction on software application, wherein each virtual user included within the count is enabled to perform the transaction during a virtual user session, determine a test regime for testing the software application, the test regime including a number of series of tests, each series including a plurality of the transactions, wherein each series of tests of the test regime is performed by different a percentage of the concurrently running virtual user count, transmit the test regime from the processor to a test server, receive test results generated by execution of the test regime by the test server, analyze the test results to determine a level of virtual user throughput for each series of tests, determine a maximum virtual user count for the test server responsively to the analysis, the maximum virtual user count being the maximum number of virtual users the test server can employ and maintain linear scalability of throughput between the series of tests of the test regime, determine a size of a set of one or more test servers to perform the test regime while maintaining linear scalability of throughput between the series of tests of the test regime using the maximum virtual user count, and calibrate the set of test servers required to execute the test regime, the calibration of the set of test servers including transmission of the testing regime to the set of test servers. the test server communicatively coupled to the computer system via a communication device, the test server being configured to receive the test regime from the computer system, execute the test regime, and transmit the test results to the computer system; and the communication device communicatively coupled to the computer system and test server, the communication device being configured to facilitate communication between the computer system and the test server. ### Claim 9 (depends on 8) The system of claim 8, wherein the computer system is further configured to receive test results generated by execution of the test regime by the set of servers from the set of test servers, analyze the received test results, adjust the count of concurrently running virtual users responsively to the analysis and transmit the adjusted count of concurrently running virtual users to the test server and the test server is further configured to generate the test results, transmit the test results to the computer system and receive the adjusted count of concurrently running virtual users from the computer system. ### Claim 10 (depends on 8) The system of claim 8, wherein the computer system is further configured to receive test results generated by execution of the test regime by the set of servers, analyze the received test results, adjust the calibration of the set of test servers responsively to the analysis, and transmit the adjusted calibration of the set of test servers to the test server and the test server is further configured to generate the test results, transmit the test results to the computer system and receive the adjusted calibration of the set of test servers from the computer system. ### Claim 11 (depends on 8) The system of claim 8, wherein the computer system is further configured to receive test results generated by execution of the test regime by the set of servers, analyze the received test results, adjust the number of series of tests responsively to the analysis and transmit the adjusted number of series of tests to the test server and the test server is further configured to generate the test results, transmit the test results to the computer system and receive adjusted number of series of tests to the test server from the computer system. ### Claim 12 (depends on 8) The system of claim 8, wherein the computer system is further configured to receive test results generated by execution of the test regime by the set of servers, analyze the received test results, adjust the transaction responsively to the analysis, and transmit the adjusted transaction to the test server and the test server is further configured to generate the test results, transmit the test results to the computer system and receive the adjusted transaction from the computer system. ### Claim 13 (depends on 8) The system of claim 8, wherein the computer system is further configured to determine, prior to transmission of test regime to the test server, a time period for executing the virtual user session by the test server, determine a time period for executing the series of tests based on the time period for executing the virtual user session, determine a time period for executing the test regime based on the time period for executing the series of tests, update the test regime to include the time period for executing the test regime, and transmit the updated test regime to the test server. ### Claim 14 A tangible computer-readable media configured to store a set of instructions, which when executed by a computer system cause the computer system to: determine a count of concurrently running virtual users for performing a transaction on software application, wherein each virtual user included within the count is enabled to perform the transaction during a virtual user session; determine a test regime for testing the software application, the test regime including a number of series of tests, each series including a plurality of the transactions, wherein each series of tests of the test regime is performed by different a percentage of the concurrently running virtual user count; transmit the test regime from the processor to a test server, the test server being enabled to communicate with the computer system and execute the test regime; receive, from the test server, test results generated by execution of the test regime by the test server; analyze the test results to determine a level of virtual user throughput for each series of tests; determine a maximum virtual user count for the test server responsively to the analysis, the maximum virtual user count being the maximum number of virtual users the test server can employ and maintain linear scalability of throughput between the series of tests of the test regime; determine a size of a set of one or more test servers to perform the test regime while maintaining linear scalability of throughput between the series of tests of the test regime using the maximum virtual user count; and calibrate the set of test servers required to execute the test regime, the calibration of the set of test servers including transmission of the testing regime to the set of test servers. receiving, at the computer system, test results generated by execution of the test regime by the set of servers from the set of test servers; analyzing, by the computer system, the received test results; adjusting, by the computer system, the count of concurrently running virtual users responsively to the analysis; and transmitting, by the computer system, the adjusted count of concurrently running virtual users to the test server. ### Claim 15 (depends on 14) The tangible computer-readable media of claim 14, further comprising: receive test results generated by execution of the test regime by the set of servers from the set of test servers; analyze the received test results; adjust the count of concurrently running virtual users responsively to the analysis; and transmit the adjusted count of concurrently running virtual users to the test server. ### Claim 16 (depends on 14) The tangible computer-readable media of claim 14, further comprising: receive test results generated by execution of the test regime by the set of servers from the set of test servers; analyze the received test results; adjust the number of series of tests responsively to the analysis; and transmit the adjusted number of series of tests to the test server. ### Claim 17 (depends on 14) The tangible computer-readable media of claim 14, further comprising: receive test results generated by execution of the test regime by the set of servers from the set of test servers; analyze the received test results; adjust the transaction responsively to the analysis; and transmit the adjusted transaction to the test server. ### Claim 18 (depends on 14) The tangible computer-readable media of claim 14, further comprising: determine, prior to transmission of test regime to the test server, a time period for executing the virtual user session by the test server; determine a time period for executing the series of tests based on the time period for executing the virtual user session; determine a time period for executing the test regime based on the time period for executing the series of tests; updating the test regime to include the time period for executing the test regime; and transmit the updated test regime to the test server.