In mood disorder diagnosis, bipolar disorder (BD) patients are often misdiagnosed as unipolar depression (UD) on initial presentation. It is crucial to establish an accurate distinction between BD and UD to make an accurate and early diagnosis, leading to improvements in treatment. In this work, facial expressions of the subjects are collected when they were watching the eliciting emotional video clips. In mood disorder detection, first, facial features extracted from the DISFA database are used to train a support vector machine (SVM) for generating facial action unit (AU) profiles. The modulation spectrum characterizing the fluctuation of AU profile sequence over a video segment are further extracted and then used for mood disorder detection using an ANN model. Comparative experiments clearly show the promising advantage of the modulation spectrum features for mood disorder detection.