Forced-Choice Ranking Models for Raters’ Ranking Data

Su Pin Hung, Hung Yu Huang

研究成果: Article同行評審

2 引文 斯高帕斯(Scopus)


To address response style or bias in rating scales, forced-choice items are often used to request that respondents rank their attitudes or preferences among a limited set of options. The rating scales used by raters to render judgments on ratees’ performance also contribute to rater bias or errors; consequently, forced-choice items have recently been employed for raters to rate how a ratee performs in certain defined traits. This study develops forced-choice ranking models (FCRMs) for data analysis when performance is evaluated by external raters or experts in a forced-choice ranking format. The proposed FCRMs consider different degrees of raters’ leniency/severity when modeling the selection probability in the generalized unfolding item response theory framework. They include an additional topic facet when multiple tasks are evaluated and incorporate variations in leniency parameters to capture the interactions between ratees and raters. The simulation results indicate that the parameters of the new models can be satisfactorily recovered and that better parameter recovery is associated with more item blocks, larger sample sizes, and a complete ranking design. A technological creativity assessment is presented as an empirical example with which to demonstrate the applicability and implications of the new models.

頁(從 - 到)603-634
期刊Journal of Educational and Behavioral Statistics
出版狀態Published - 2022 10月

All Science Journal Classification (ASJC) codes

  • 教育
  • 社會科學(雜項)


深入研究「Forced-Choice Ranking Models for Raters’ Ranking Data」主題。共同形成了獨特的指紋。