Chatter identification is necessary in order to achieve stable machining conditions. However, the linear approximation in regenerative chatter vibration is problematic because of the rich nonlinear characteristics in machining. In this study, a novel method to detect chatter is proposed. Firstly, measured cutting force signals are decomposed into a set of intrinsic mode functions by using ensemble empirical mode decomposition. Hilbert transform is following to extract the instantaneous frequency. Fast Fourier transform is also utilized for each intrinsic mode function to determine the intrinsic mode function that contains rich chatter. Finally, the standard deviation and energy ratio in frequency domain of intrinsic mode functions are found as simply dimensionless chatter indicators. The effectively proposed approach is validated by analyzing the machined surface topography and also compared to the stability lobe diagram.