Joint optimization of FIR filters in a non-linear compensator

A method comprising:

• applying calibration signals to a signal converter input in a test and measurement system;
• measuring signal output from the signal converter;
• performing at least one of spectral and amplitude analysis on the signal output;
• determining non-linear signal components in the signal output from the at least one of spectral and amplitude analysis;
• comparing non-linear signal components in a signal converter output to non-linear filter components at a Linear, Memory-less Non-linear, Linear (LNL) compensator, the LNL compensator including first stage Finite Impulse Response (FIR) filters and second stage FIR filters;
• receiving a minimum performance threshold for the LNL compensator;
• determining lengths for the first stage FIR filters and lengths for the second stage FIR filters to meet the minimum performance threshold;
• and
• determining a number of the first stage FIR filters and a number of the second stage FIR filters to meet the minimum performance threshold.

A non-transitory computer-readable storage medium having instructions embodied thereon that, when executed by a processor, cause a processor to:

• apply calibration signals to a signal converter input in a test and measurement system;
• measuring signal output from the signal converter;
• performing at least one of spectral and amplitude analysis on the signal output;
• determining non-linear signal components in the signal output from the at least one of spectral and amplitude analysis;
• compare non-linear signal components at a Linear, Memory-less Non-linear, Linear (LNL) compensator to the non-linear filter components in signal output from the signal converter;
• employ the comparison of the non-linear signal components to the non-linear filter components to determine initial filter coefficients for Finite Impulse Response (FIR) filters in the LNL compensator;
• determine a Jacobian matrix for the LNL compensator;
• and
• employ the initial filter coefficients and the Jacobian matrix to determine final filter coefficients for the FIR filters in the LNL compensator.