This paper presents a digitally self-calibration method with a recursive discrete Fourier transform (RDFT) algorithm for successive approximation (SAR) analog-to-digital converters (ADCs). A behavior model of 12-bit SAR ADC based on Matlab Simulink is proposed to combine with the compensation algorithm and digital input testing signal. In addition, the SAR ADC error caused by 1% capacitor mismatch of the DAC array is also built for evaluation of the real radixes of the DAC capacitor array. The proposed digital calibration method utilizes a RDFT to detect the third harmonic distortion value and to evaluate the error factor by using the calibration formula. Then, the error factor is applied to generate a new digital output code for the targeted SAR ADC. Under the sampling rate of 126.26 S/s, the simulation results reveals the third harmonic distortion is -47.98 dB before calibration and -84.34 dB after calibration with 1% capacitor mismatch. In addition, the SNDR and ENOB can be enhanced to 20.9 dB and 3.52 bit, respectively. Therefore, the proposed method can be adopted to enhance the performance of the ADC for various applications in the future.