Personalized time-sync comment generation based on a multimodal transformer

Hei Chia Wang, Martinus Maslim, Wei Ting Hong

Research output: Contribution to journalArticlepeer-review


Online video entertainment has attracted large audiences and sustained viewing in various fields. With more than 4.5 billion Internet users worldwide, online video entertainment continues to be the most popular activity for users. Time synchronization comments (TSCs) are a new type of text information in videos. Unlike traditional online video-sharing platforms, where users can only leave comments in the comments section, TSCs can "fly through" the screen at each video playback time. However, the current research on TSC generation does not address the problem of personalization but only focuses on the relationship between images and TSC modalities. Therefore, we propose a multimodal transformer, personalized time-sync comment generation (PTSCG), to generate personalized TSCs. The generated TSCs are more suitable for different users. According to the experimental results, the F − 1 score evaluated for PTSCG after comparing the generated TSC with the original TSCs reached 0.58, which is better than those of other existing models, showing the effectiveness of the method proposed in this study.

Original languageEnglish
Article number105
JournalMultimedia Systems
Issue number2
Publication statusPublished - 2024 Apr

All Science Journal Classification (ASJC) codes

  • Software
  • Information Systems
  • Media Technology
  • Hardware and Architecture
  • Computer Networks and Communications


Dive into the research topics of 'Personalized time-sync comment generation based on a multimodal transformer'. Together they form a unique fingerprint.

Cite this