Two-level hierarchical alignment for semi-coupled HMM-based audiovisual emotion recognition with temporal course

Chung Hsien Wu, Jen Chun Lin, Wen Li Wei

研究成果: Article同行評審

48 引文 斯高帕斯(Scopus)

摘要

A complete emotional expression typically contains a complex temporal course in face-to-face natural conversation. To address this problem, a bimodal hidden Markov model (HMM)-based emotion recognition scheme, constructed in terms of sub-emotional states, which are defined to represent temporal phases of onset, apex, and offset, is adopted to model the temporal course of an emotional expression for audio and visual signal streams. A two-level hierarchical alignment mechanism is proposed to align the relationship within and between the temporal phases in the audio and visual HMM sequences at the model and state levels in a proposed semi-coupled hidden Markov model (SC-HMM). Furthermore, by integrating a sub-emotion language model, which considers the temporal transition between sub-emotional states, the proposed two-level hierarchical alignment-based SC-HMM (2H-SC-HMM) can provide a constraint on allowable temporal structures to determine an optimal emotional state. Experimental results show that the proposed approach can yield satisfactory results in both the posed MHMC and the naturalistic SEMAINE databases, and shows that modeling the complex temporal structure is useful to improve the emotion recognition performance, especially for the naturalistic database (i.e., natural conversation). The experimental results also confirm that the proposed 2H-SC-HMM can achieve an acceptable performance for the systems with sparse training data or noisy conditions.

原文English
文章編號6542683
頁(從 - 到)1880-1895
頁數16
期刊IEEE Transactions on Multimedia
15
發行號8
DOIs
出版狀態Published - 2013

All Science Journal Classification (ASJC) codes

  • 訊號處理
  • 媒體技術
  • 電腦科學應用
  • 電氣與電子工程

指紋

深入研究「Two-level hierarchical alignment for semi-coupled HMM-based audiovisual emotion recognition with temporal course」主題。共同形成了獨特的指紋。

引用此