Exploring Feature Fusion from A Contrastive Multi-Modality Learner for Liver Cancer Diagnosis

Yang Fan Chiang, Pei Xuan Li, Ding You Wu, Hsun Ping Hsieh, Ching Chung Ko

研究成果: Conference contribution

摘要

Self-supervised contrastive learning has achieved promising results in computer vision, and recently it also received attention in the medical domain. In practice, medical data is hard to collect and even harder to annotate, but leveraging multi-modality medical images to make up for small datasets has proved to be helpful. In this work, we focus on mining multi-modality Magnetic Resonance (MR) images to learn multi-modality contrastive representations. We first present multi-modality data augmentation (MDA) to adapt contrastive learning to multi-modality learning. Then, the proposed cross-modality group convolution (CGC) is used for multi-modality features in the downstream fine-tune task. Specifically, in the pre-training stage, considering different behaviors from each MRI modality with the same anatomic structure, yet without designing a handcrafted pretext task, we select two augmented MR images from a patient as a positive pair, and then directly maximize the similarity between positive pairs using Simple Siamese networks. To further exploit multi-modality representation, we combine 3D and 2D group convolution with a channel shuffle operation to efficiently incorporate different modalities of image features. We evaluate our proposed methods on liver MR images collected from a well-known hospital in Taiwan. Experiments show our framework has significantly improved from previous methods.

原文English
主出版物標題Proceedings of the 5th ACM International Conference on Multimedia in Asia, MMAsia 2023
發行者Association for Computing Machinery, Inc
ISBN(電子)9798400702051
DOIs
出版狀態Published - 2023 12月 6
事件5th ACM International Conference on Multimedia in Asia, MMAsia 2023 - Hybrid, Tainan, Taiwan
持續時間: 2023 12月 62023 12月 8

出版系列

名字Proceedings of the 5th ACM International Conference on Multimedia in Asia, MMAsia 2023

Conference

Conference5th ACM International Conference on Multimedia in Asia, MMAsia 2023
國家/地區Taiwan
城市Hybrid, Tainan
期間23-12-0623-12-08

All Science Journal Classification (ASJC) codes

  • 電腦繪圖與電腦輔助設計
  • 人機介面

指紋

深入研究「Exploring Feature Fusion from A Contrastive Multi-Modality Learner for Liver Cancer Diagnosis」主題。共同形成了獨特的指紋。

引用此