Speaking effect removal on emotion recognition from facial expressions based on eigenface conversion

Chung Hsien Wu, Wen Li Wei, Jen Chun Lin, Wei Yu Lee

Research output: Contribution to journalArticlepeer-review

35 Citations (Scopus)


Speaking effect is a crucial issue that may dramatically degrade performance in emotion recognition from facial expressions. To manage this problem, an eigenface conversion-based approach is proposed to remove speaking effect on facial expressions for improving accuracy of emotion recognition. In the proposed approach, a context-dependent linear conversion function modeled by a statistical Gaussian Mixture Model (GMM) is constructed with parallel data from speaking and non-speaking facial expressions with emotions. To model the speaking effect in more detail, the conversion functions are categorized using a decision tree considering the visual temporal context of the Articulatory Attribute (AA) classes of the corresponding input speech segments. For verification of the identified quadrant of emotional expression on the Arousal-Valence (A-V) emotion plane, which is commonly used to dimensionally define the emotion classes, from the reconstructed facial feature points, an expression template is constructed to represent the feature points of the non-speaking facial expressions for each quadrant. With the verified quadrant, a regression scheme is further employed to estimate the A-V values of the facial expression as a precise point in the A-V emotion plane. Experimental results show that the proposed method outperforms current approaches and demonstrates that removing the speaking effect on facial expression is useful for improving the performance of emotion recognition.

Original languageEnglish
Article number6557471
Pages (from-to)1732-1744
Number of pages13
JournalIEEE Transactions on Multimedia
Issue number8
Publication statusPublished - 2013

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Media Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering


Dive into the research topics of 'Speaking effect removal on emotion recognition from facial expressions based on eigenface conversion'. Together they form a unique fingerprint.

Cite this