Systems and methods of building sequenceable test methodologies

A method for sending simulated input over network connections from simulated first and second endpoints to a system under test (SUT) in communication with a test generator system that monitors at a packet analysis level traffic received from the SUT, the method including:

• implementing a test protocol using at least one multi-layered test definition data structure that runs on a test harness device, wherein the multi-layered test definition data structure includes:
• a base object layer including test processor configuration definitions that implement basic devices and protocols;
• a command sequencer object layer that incorporates devices and protocols from the base object layer and that defines a test in a series of command, including (a) topology commands that configure simulated first and second endpoint devices and protocols running on those devices, (b) one or more traffic commands that configure test traffic messages to send between the first and second endpoint devices, (c) test and control commands that implement test operations, including iteration over test operations, and (d) one or more result commands that specify result assembly for dynamic results reported during a test and after-test results reported after the test completes;
• command parameter data that specifies parameters of at least some of the commands (a)-(d);
• and
• exposed parameter modification data that an end user can modify to specify operation of a test instance, wherein the exposed parameter modification data is a subset of less than one quarter of the parameters in the command parameter data;
• receiving, at an end user test modification tool, at least one test selection message specifying a selected test and loads into processing memory of the test harness device the multi-layered test definition data structure that implements a selected test;
• receiving, at the end user test modification tool, at least one exposed parameter modification message specifying a value to apply to an exposed command parameter of a selected test instance;
• and
• combining, using a test sequence processor, test protocol data from the multi-layered test definition data structure that implements the selected test, runs the selected test on the test harness device, and reports results specified by the result commands of the selected test.

A tangible non-volatile computer readable medium that stores computer instructions that, when run on hardware, implement sending simulated input over network connections from simulated first and second endpoints to a system under test (SUT) in communication with a test generator system that monitors at a packet analysis level traffic received from the SUT, the implementation including:

• implementing a test protocol using at least one multi-layered test definition data structure that runs on a test harness device, wherein the multi-layered test definition data structure includes:
• a base object layer including test processor configuration definitions that implement basic devices and protocols;
• a command sequencer object layer that incorporates devices and protocols from the base object layer and that defines a test in a series of command, including (a) topology commands that configure simulated first and second endpoint devices and protocols running on those devices, (b) one or more traffic commands that configure test traffic messages to send between the first and second endpoint devices, (c) test and control commands that implement test operations, including iteration over test operations, and (d) one or more result commands that specify result assembly for dynamic results reported during a test and after-test results reported after the test completes;
• command parameter data that specifies parameters of at least some of the commands (a)-(d);
• and
• exposed parameter modification data that an end user can modify to specify operation of a test instance, wherein the exposed parameter modification data is a subset of less than one quarter of the parameters in the command parameter data;
• receiving, at an end user test modification tool, at least one test selection message specifying a selected test and loads into processing memory of the test harness device the multi-layered test definition data structure that implements a selected test;
• receiving, at the end user test modification tool, at least one exposed parameter modification message specifying a value to apply to an exposed command parameter of a selected test instance;
• and
• combining, using a test sequence processor, test protocol data from the multi-layered test definition data structure that implements the selected test, runs the selected test on the test harness device, and reports results specified by the result commands of the selected test.

A test generator system coupled in communication with a system under test (SUT), wherein the test generator system sends simulated input over network connections from one or more simulated endpoints to the SUT and monitors at a packet analysis level traffic received from the SUT, the system including:

• a test harness device;
• at least one multi-layered test definition data structure that implements a test protocol when run on the test harness device, wherein the multi-layered test definition data structure includes:
• a base object layer including test processor configuration definitions that implement basic devices and protocols;
• a command sequencer object layer that incorporates devices and protocols from the base object layer and that defines a test in a series of command, including (a) topology commands that configure the simulated endpoint devices and protocols running on those devices, (b) one or more traffic commands that configure test traffic messages to send between the endpoint devices, (c) test and control commands that implement test operations, including iteration over test operations, and (d) one or more result commands that specify result assembly for dynamic results reported during a test and after-test results reported after the test completes;
• command parameter data that specifies parameters of at least some of the commands (a)-(d);
• and
• exposed parameter modification data that an end user can modify to specify operation of a test instance, wherein the exposed parameter modification data is a subset of less than one quarter of the parameters in the command parameter data;
• an end user test modification tool that:
• receives at least one test selection message specifying a selected test and loads into processing memory of the test harness device the multi-layered test definition data structure that implements a selected test;
• and
• receives at least one exposed parameter modification message specifying a value to apply to an exposed command parameter of a selected test instance;
• and
• a test sequence processor that combines test protocol data from the multi-layered test definition data structure that implements the selected test, runs the selected test on the test harness device, and reports results specified by the result commands of the selected test.