A similarity measure for text classification and clustering

Yung Shen Lin, Jung Yi Jiang, Shie Jue Lee

Research output: Contribution to journalArticlepeer-review

164 Citations (Scopus)

Abstract

Measuring the similarity between documents is an important operation in the text processing field. In this paper, a new similarity measure is proposed. To compute the similarity between two documents with respect to a feature, the proposed measure takes the following three cases into account: a) The feature appears in both documents, b) the feature appears in only one document, and c) the feature appears in none of the documents. For the first case, the similarity increases as the difference between the two involved feature values decreases. Furthermore, the contribution of the difference is normally scaled. For the second case, a fixed value is contributed to the similarity. For the last case, the feature has no contribution to the similarity. The proposed measure is extended to gauge the similarity between two sets of documents. The effectiveness of our measure is evaluated on several real-world data sets for text classification and clustering problems. The results show that the performance obtained by the proposed measure is better than that achieved by other measures.

Original languageEnglish
Article number6420834
Pages (from-to)1575-1590
Number of pages16
JournalIEEE Transactions on Knowledge and Data Engineering
Volume26
Issue number7
DOIs
Publication statusPublished - 2014 Jul

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Computer Science Applications
  • Computational Theory and Mathematics

Fingerprint Dive into the research topics of 'A similarity measure for text classification and clustering'. Together they form a unique fingerprint.

Cite this