LAD: Layer-Wise Adaptive Distillation for BERT Model Compression

Ying Jia Lin, Kuan Yu Chen, Hung Yu Kao

研究成果: Article同行評審

9 引文 斯高帕斯(Scopus)

摘要

Recent advances with large-scale pre-trained language models (e.g., BERT) have brought significant potential to natural language processing. However, the large model size hinders their use in IoT and edge devices. Several studies have utilized task-specific knowledge distillation to compress the pre-trained language models. However, to reduce the number of layers in a large model, a sound strategy for distilling knowledge to a student model with fewer layers than the teacher model is lacking. In this work, we present Layer-wise Adaptive Distillation (LAD), a task-specific distillation framework that can be used to reduce the model size of BERT. We design an iterative aggregation mechanism with multiple gate blocks in LAD to adaptively distill layer-wise internal knowledge from the teacher model to the student model. The proposed method enables an effective knowledge transfer process for a student model, without skipping any teacher layers. The experimental results show that both the six-layer and four-layer LAD student models outperform previous task-specific distillation approaches during GLUE tasks.

原文English
文章編號1483
期刊Sensors
23
發行號3
DOIs
出版狀態Published - 2023 2月

All Science Journal Classification (ASJC) codes

  • 分析化學
  • 資訊系統
  • 儀器
  • 原子與分子物理與光學
  • 電氣與電子工程
  • 生物化學

指紋

深入研究「LAD: Layer-Wise Adaptive Distillation for BERT Model Compression」主題。共同形成了獨特的指紋。

引用此