This paper presents a novel scheme on steady-state performance of an isolated three-phase self-excited reluctance generator (SERG) under unbalanced excitation capacitors. It is assumed that the Y-connected excitation capacitor bank is properly connected at the machine's stator terminals. The values of excitation capacitance of both phases b and c are kept to be the same. The excitation capacitance of phase a is varied in order to investigate the unbalanced excitation effects on voltages and currents of the studied SERG. An approach based on a three-phase reluctance machine model is employed to derive dynamic equations of the studied SERG. The steady-state characteristics of the studied machine can be directly obtained from the dynamic simulated results. Both voltage and current unbalanced factors of the studied machine under no-load and loaded conditions are examined and compared. Experimental results obtained from a laboratory 370 W reluctance machine driven by a DC motor are also performed to confirm the feasibility of the proposed method.