Attentively-Coupled Long Short-Term Memory for Audio-Visual Emotion Recognition

Jia Hao Hsu, Chung Hsien Wu

研究成果: Conference contribution

摘要

There have been more and more studies on emotion recognition through multiple modalities. In the existing audiovisual emotion recognition methods, few studies focused on modeling emotional fluctuations in the signals. Besides, how to fuse multimodal signals, such as audio-visual signals, is still a challenging issue. In this paper, segments of audio-visual signals are extracted and considered as the recognition unit to characterize the emotional fluctuation. An Attentively-Coupled long-short term memory (ACLSTM) is proposed to combine the audio-based and visual-based LSTMs to improve the emotion recognition performance. In the Attentively-Coupled LSTM, the Coupled LSTM is used as the fusion model, and the neural tensor network (NTN) is employed for attention estimation to obtain the segment-based emotion consistency between audio and visual segments. Compared with previous approaches, the experimental results showed that the proposed method achieved the best results of 70.1% in multi-modal emotion recognition on the dataset BAUM-I.

原文English
主出版物標題2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2020 - Proceedings
發行者Institute of Electrical and Electronics Engineers Inc.
頁面1048-1053
頁數6
ISBN(電子)9789881476883
出版狀態Published - 2020 十二月 7
事件2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2020 - Virtual, Auckland, New Zealand
持續時間: 2020 十二月 72020 十二月 10

出版系列

名字2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2020 - Proceedings

Conference

Conference2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2020
國家New Zealand
城市Virtual, Auckland
期間20-12-0720-12-10

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Computer Networks and Communications
  • Computer Vision and Pattern Recognition
  • Hardware and Architecture
  • Signal Processing
  • Decision Sciences (miscellaneous)
  • Instrumentation

指紋 深入研究「Attentively-Coupled Long Short-Term Memory for Audio-Visual Emotion Recognition」主題。共同形成了獨特的指紋。

引用此