Polysomnography scoring–related training and quantitative assessment for improving interscorer agreement

Study Objectives: To efficiently improve the scoring competency of scorers with varying levels of experience across regions in Taiwan, we developed a training program with a cloud-based polysomnography scoring platform to evaluate and improve interscorer agreement. Methods: A total of 70 scorers from 34 sleep centers in Taiwan (job tenure: 0.5–39.0 years) completed a scoring test. All scorers scored a 742-epoch (30 s/epoch) overnight polysomnography recording of a patient with a moderate apnea-hypopnea index. Subsequently, 8 scoring experts delivered 8 interactive online lectures (each lasting 30 minutes). The training program included identifying scoring weaknesses, highlighting the latest scoring rules, and providing physicians’ perspectives. Afterward, the scorers completed the second scoring test on the same participant. Changes in agreement from the first to second scoring test were identified. Sleep staging, sleep parameters, and respiratory events were considered for evaluating scoring agreement. Results: The scorers’ agreement in overall sleep stage scoring significantly increased from 74.6 to 82.3% (median score). The proportion of scorers with an agreement of ≥ 80% increased from 20.0% (14/70) to 58.6% (41/70) after the online training program. In addition, the scorers’ agreement in overall respiratory-event scoring increased to 88.8% (median score) after training. The scorers with a job tenure of 2.0–4.9 years exhibited the highest level of improvement in overall sleep staging (their median agreement increased from 72.8 to 84.9%; P < .001). Conclusions: Our interactive online training program efficiently targeted the scorers’ scoring weaknesses identified in the first scoring test, leading to substantial improvements in scoring proficiency.