True or False: Does the Deep Learning Model Learn to Detect Rumors?

Shiwen Ni, Jiawen Li, Hung Yu Kao

研究成果: Conference contribution

2 引文 斯高帕斯(Scopus)

摘要

It is difficult for humans to distinguish the true and false of rumors, but current deep learning models can surpass humans and achieve excellent accuracy on many rumor datasets. In this paper, we investigate whether deep learning models that seem to perform well actually learn to detect rumors. We evaluate models on their generalization ability to out-of-domain examples by fine-tuning BERT-based models on five real-world datasets and evaluating against all test sets. The experimental results indicate that the generalization ability of the models on other unseen datasets are unsatisfactory, even common-sense rumors cannot be detected. Moreover, we found through experiments that models take shortcuts and learn absurd knowledge when the rumor datasets have serious data pitfalls. This means that simple modifications to the rumor text based on specific rules will lead to inconsistent model predictions. To more realistically evaluate rumor detection models, we proposed a new evaluation method called paired test (PairT), which requires models to correctly predict a pair of test samples at the same time. Furthermore, we make recommendations on how to better create rumor dataset and evaluate rumor detection model at the end of this paper.

原文English
主出版物標題Proceedings - 2021 International Conference on Technologies and Applications of Artificial Intelligence, TAAI 2021
發行者Institute of Electrical and Electronics Engineers Inc.
頁面119-124
頁數6
ISBN(電子)9781665408257
DOIs
出版狀態Published - 2021
事件26th International Conference on Technologies and Applications of Artificial Intelligence, TAAI 2021 - Taichung, Taiwan
持續時間: 2021 11月 182021 11月 20

出版系列

名字Proceedings - 2021 International Conference on Technologies and Applications of Artificial Intelligence, TAAI 2021

Conference

Conference26th International Conference on Technologies and Applications of Artificial Intelligence, TAAI 2021
國家/地區Taiwan
城市Taichung
期間21-11-1821-11-20

All Science Journal Classification (ASJC) codes

  • 人工智慧
  • 電腦科學應用
  • 資訊系統與管理

指紋

深入研究「True or False: Does the Deep Learning Model Learn to Detect Rumors?」主題。共同形成了獨特的指紋。

引用此