Integration of eye tracking and lip motion for hands-free computer access

Ing Shiou Hwang, Yi Ying Tsai, Bo Han Zeng, Chien Ming Lin, Huei Sheng Shiue, Gwo Ching Chang

研究成果: Article同行評審

6 引文 斯高帕斯(Scopus)


Healthy people use a keyboard and mouse as standard input devices for controlling a computer. However, these input devices are usually not suitable for people with severe physical disabilities. This study aims to design and implement a suitable and reliable human–computer interactive (HCI) interface for disabled users by integrating both eye tracking technology and lip motion recognition. Eye movements control the cursor position on the computer screen. An eye gaze followed by a lip motion of the mouth opening served as a mouse click action. Seven lip motion features were extracted to discriminate mouth openings and mouth closures with the cumulative sum control chart algorithm. A novel smoothing technique called the threshold-based Savitzky–Golay smoothing filter was proposed to stabilize the cursor movement due to the inherent jittery motions of eyes and to reduce eye tracking latencies. In this study, a fixation experiment with nine dots was carried out to evaluate the efficacy of eye gaze data smoothing. A Chinese text entry experiment based on the on-screen keyboard with four keypad sizes was designed to evaluate the influence of the keypad size on the Chinese text entry rate. The results of the fixation experiment indicated that the threshold-based Savitzky–Golay smoothing filter with a threshold value of two standard deviations, a polynomial order of 3, and a window length of 61 can significantly improve the stability of eye cursor movements by 44.86% on average. The averaged Chinese text entry rate achieved 4.41 wpm with the dynamical enlargeable keypads. The current results encouraged us to utilize the proposed HCI interface for disabled users in the future.

頁(從 - 到)405-416
期刊Universal Access in the Information Society
出版狀態Published - 2021 6月

All Science Journal Classification (ASJC) codes

  • 軟體
  • 資訊系統
  • 人機介面
  • 電腦網路與通信


深入研究「Integration of eye tracking and lip motion for hands-free computer access」主題。共同形成了獨特的指紋。