Metrics-related testing of an operational support system (OSS) of an incumbent provider for compliance with a regulatory scheme

A method for testing one or more operational support systems (OSSs) of an incumbent provider for compliance with a regulatory scheme, the testing being associated with metrics capture, tracking, and reporting activities of the incumbent provider with respect to a particular operational domain, the method being performed by an independent testing entity attempting to emulate a competitive provider that would access the OSSs in attempting to compete with the incumbent provider in an open competitive market, the method comprising for testing of an OSS:

• within a data integrity testing component during active testing, to determine whether filtered data the incumbent provider uses to calculate values for each of one or more metrics is accurately derived from raw data the incumbent provider collects in connection with transactions associated with the OSS:
• identifying one or more fields in the filtered data that the incumbent provider uses in calculating values for the metric;
• identifying a random sample of transactions associated with the OSS;
• obtaining from the incumbent provider copies of records associated with the identified transactions, the records containing raw data for these transactions from which filtered data for these transactions in the identified fields was derived;
• and
• for each identified field, determining whether the filtered data for the identified transactions in the identified field is consistent with the raw data for the identified transactions contained in the associated records;
• within a metrics replication testing component during active testing, for each of one or more metrics for which the incumbent provider has calculated values:
• obtaining from the incumbent provider a description of the metric, filtered data on which the incumbent provider based its calculation of values for the metric, and an algorithm according to which the incumbent provider made its calculation of values for the metric, the filtered data being associated with a specified time period, the filtered data being derived from raw data the incumbent provider collected in connection with transactions associated with the OSS;
• re-calculating values for the metric according to the description, data, and algorithm obtained from the incumbent provider;
• and
• determining whether the re-calculated values are consistent with the values calculated by the incumbent provider;
• evaluating performance of the incumbent provider during active testing according to predetermined evaluation criteria;
• generating test results according to the evaluation;
• and
• issuing a final report comprising the test results and providing a sufficient basis for a regulatory entity administering the regulatory scheme to determine compliance of the incumbent provider with the regulatory scheme.

A method for testing one or more operational support systems (OSSs) of an incumbent provider for compliance with a regulatory scheme, the testing being associated with performance metrics capture, tracking, and reporting activities of the incumbent provider across a plurality of operational domains, the method performed by an independent testing entity attempting to emulate a competitive provider that would access the OSSs in attempting to compete with the incumbent provider in an open competitive market, the method comprising:

• conducting a metrics data collection and storage test during active testing to evaluate incumbent provider policies and practices for collecting and storing data used for calculating performance metric values, both for internal operations of the incumbent provider and for operations of the incumbent provider associated with its interaction with competitive providers;
• conducting a metrics definition documentation and implementation test during active testing to evaluate incumbent provider policies and practices for documenting and implementing performance metric definitions, implementation of performance metrics definitions comprising applying exclusions and business rules in calculating performance metric values based on raw data;
• conducting a metrics change management test during active testing to evaluate incumbent provider policies and practices for developing, implementing, monitoring, and reporting changes concerning performance metrics;
• conducting a metrics data integrity test during active testing to evaluate incumbent provider policies and practices for processing data necessary to calculate performance metric values;
• conducting a metrics calculation and reporting test during active testing to evaluate policies and practices of the incumbent provider for calculating performance metric values, this test comprising:
• accessing calculated performance metric values for operations of the incumbent provider associated with its interaction with competitive providers, these values having been calculated by the incumbent provider and thereafter reported to the regulatory entity;
• re-calculating the accessed performance metric values;
• comparing the re-calculated values with the accessed values to identify any discrepancies;
• attempting to reconcile any identified discrepancies between the re-calculated values and the accessed values;
• and
• determining whether the incumbent provider calculated the accessed values consistent with documented instructions for doing so;
• conducting a statistical evaluation of transactions test during active testing to evaluate metrics-related service performance of the incumbent provider according to statistical techniques, this test comprising:
• accessing test performance metric values calculated based on interaction between the incumbent provider and the testing entity;
• accessing benchmark performance metric values calculated based on interaction between the incumbent provider and one or more competitive providers;
• and
• comparing the test performance metric values with the benchmark performance metric values according to statistical techniques to determine whether and to what extent parity exists between the compared values;
• evaluating performance of the incumbent provider during active testing according to predetermined evaluation criteria;
• generating test results according to the evaluation;
• and
• issuing a final report comprising the test results and providing a sufficient basis for a regulatory entity administering the regulatory scheme to determine a net impact of identified metrics-related issues on the ability of competitive providers to compete with the incumbent provider in an open competitive market.