Personalized time-sync comment generation based on a multimodal transformer

Online video entertainment has attracted large audiences and sustained viewing in various fields. With more than 4.5 billion Internet users worldwide, online video entertainment continues to be the most popular activity for users. Time synchronization comments (TSCs) are a new type of text information in videos. Unlike traditional online video-sharing platforms, where users can only leave comments in the comments section, TSCs can "fly through" the screen at each video playback time. However, the current research on TSC generation does not address the problem of personalization but only focuses on the relationship between images and TSC modalities. Therefore, we propose a multimodal transformer, personalized time-sync comment generation (PTSCG), to generate personalized TSCs. The generated TSCs are more suitable for different users. According to the experimental results, the F − 1 score evaluated for PTSCG after comparing the generated TSC with the original TSCs reached 0.58, which is better than those of other existing models, showing the effectiveness of the method proposed in this study.