Opportunities or risks to reduce labor in crowdsourcing translation? Characterizing cost versus quality via a PageRank-HITS hybrid model

Rui Yan, Yiping Song, Cheng Te Li, Ming Zhang, Xiaohua Hu

研究成果: Conference contribution

7 引文 斯高帕斯(Scopus)

摘要

Crowdsourcing machine translation shows advantages of lower expense in money to collect the translated data. Yet, when compared with translation by trained professionals, results collected from non-professional translators might yield lowquality outputs. A general solution for crowdsourcing practitioners is to employ a large amount of labor force to gather enough redundant data and then solicit from it. Actually we can further save money by avoid collecting bad translations. We propose to score Turkers by their authorities during observation, and then stop hiring the unqualified Turkers. In this way, we bring both opportunities and risks in crowdsourced translation: we can make it cheaper than cheaper while we might suffer from quality loss. In this paper, we propose a graph-based PageRank-HITS Hybrid model to distinguish authoritative workers from unreliable ones. The algorithm captures the intuition that good translation and good workers are mutually reinforced iteratively in the proposed frame. We demonstrate the algorithm will keep the performance while reduce work force and hence cut cost. We run experiments on the NIST 2009 Urdu-to-English evaluation set with Mechanical Turk, and quantitatively evaluate the performance in terms of BLEU score, Pearson correlation and real money.

原文English
主出版物標題IJCAI 2015 - Proceedings of the 24th International Joint Conference on Artificial Intelligence
編輯Michael Wooldridge, Qiang Yang
發行者International Joint Conferences on Artificial Intelligence
頁面1025-1032
頁數8
ISBN(電子)9781577357384
出版狀態Published - 2015 一月 1
事件24th International Joint Conference on Artificial Intelligence, IJCAI 2015 - Buenos Aires, Argentina
持續時間: 2015 七月 252015 七月 31

出版系列

名字IJCAI International Joint Conference on Artificial Intelligence
2015-January
ISSN(列印)1045-0823

Other

Other24th International Joint Conference on Artificial Intelligence, IJCAI 2015
國家Argentina
城市Buenos Aires
期間15-07-2515-07-31

    指紋

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence

引用此

Yan, R., Song, Y., Li, C. T., Zhang, M., & Hu, X. (2015). Opportunities or risks to reduce labor in crowdsourcing translation? Characterizing cost versus quality via a PageRank-HITS hybrid model. 於 M. Wooldridge, & Q. Yang (編輯), IJCAI 2015 - Proceedings of the 24th International Joint Conference on Artificial Intelligence (頁 1025-1032). (IJCAI International Joint Conference on Artificial Intelligence; 卷 2015-January). International Joint Conferences on Artificial Intelligence.