In DSP-based IQ modulators generating CPFSK signals, shortcomings in the implementation of the analog reconstruction filters result in the loss of the constant envelope property of the output CPFSK signal. These ripples cause undesirable spreading of the transmitted signal spectrum into adjacent channels when the signal passes through nonlinear elements in the transmission path and the consequent failure of the transmitted signal in meeting transmission standards requirements. Therefore, digital techniques compensating for these shortcomings play an important role in enhancing the performance of the IQ modulation system. Recently, several methods have been proposed in the literature to digitally compensate for the imperfections in the transfer characteristics of the analog reconstruction filters. Although these methods have been shown to be effective in removing the output envelope ripples, they result in filters of high orders and are therefore computationally demanding to implement on the DSP. Furthermore, previous techniques suffer from numerical instabilities as a result of matrix inversion in the process of calculating the solution vector. In this paper, we present two new techniques for designing the digital compensation filters by means of H ∞ optimization to address the limitations of previous solutions. Design of control systems by H ∞ optimization is now a standard technique. Simulation examples show that these techniques are effective and lead to substantial improvement of the output envelope ripples.