Cell-Coupled Long Short-Term Memory with L-Skip Fusion Mechanism for Mood Disorder Detection through Elicited Audiovisual Features

Ming Hsiang Su, Chung Hsien Wu, Kun Yi Huang, Tsung Hsien Yang

Research output: Contribution to journalArticlepeer-review

7 Citations (Scopus)


In early stages, patients with bipolar disorder are often diagnosed as having unipolar depression in mood disorder diagnosis. Because the long-term monitoring is limited by the delayed detection of mood disorder, an accurate and one-time diagnosis is desirable to avoid delay in appropriate treatment due to misdiagnosis. In this paper, an elicitation-based approach is proposed for realizing a one-time diagnosis by using responses elicited from patients by having them watch six emotion-eliciting videos. After watching each video clip, the conversations, including patient facial expressions and speech responses, between the participant and the clinician conducting the interview were recorded. Next, the hierarchical spectral clustering algorithm was employed to adapt the facial expression and speech response features by using the extended Cohn-Kanade and eNTERFACE databases. A denoizing autoencoder was further applied to extract the bottleneck features of the adapted data. Then, the facial and speech bottleneck features were input into support vector machines to obtain speech emotion profiles (EPs) and the modulation spectrum (MS) of the facial action unit sequence for each elicited response. Finally, a cell-coupled long short-term memory (LSTM) network with an L-skip fusion mechanism was proposed to model the temporal information of all elicited responses and to loosely fuse the EPs and the MS for conducting mood disorder detection. The experimental results revealed that the cell-coupled LSTM with the L-skip fusion mechanism has promising advantages and efficacy for mood disorder detection.

Original languageEnglish
Article number8668691
Pages (from-to)124-135
Number of pages12
JournalIEEE Transactions on Neural Networks and Learning Systems
Issue number1
Publication statusPublished - 2020 Jan

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence


Dive into the research topics of 'Cell-Coupled Long Short-Term Memory with L-Skip Fusion Mechanism for Mood Disorder Detection through Elicited Audiovisual Features'. Together they form a unique fingerprint.

Cite this