TY - JOUR
T1 - KPT++
T2 - Refined knowledgeable prompt tuning for few-shot text classification
AU - Ni, Shiwen
AU - Kao, Hung Yu
N1 - Funding Information:
This work was funded in part by Qualcomm, United States through a Taiwan University Research Collaboration Project NAT-487842 and in part by the Ministry of Science and Technology, Taiwan , under grant MOST 111-2221-E-006-001 .
Publisher Copyright:
© 2023 Elsevier B.V.
PY - 2023/8/15
Y1 - 2023/8/15
N2 - Recently, the new paradigm “pre-train, prompt, and predict” has achieved remarkable few-shot learning achievements compared with the “pre-train, fine-tune” paradigm. Prompt-tuning inserts the prompt text into the input and converts the classification task into a masked language modeling task. One of the key steps is to build a projection between the labels and the label words, i.e., the verbalizer. Knowledgeable prompt-tuning (KPT), which integrates external knowledge into the verbalizer to improve and stabilize prompt-tuning. KPT uses word embeddings and various knowledge graphs to expand the label words space to hundreds of words per class. However, some unreasonable label words in the verbalizer may damage the accuracy. In this paper, a new method called KPT++ is proposed to improve the few-shot text classification. KPT++ is refined knowledgeable prompt-tuning, which can also be regarded as an upgraded version of KPT. Specifically, KPT++ uses two newly proposed prompt grammar refinement (PGR) and probability distribution refinement (PDR) to refine the knowledgeable verbalizer. Extensive experiments on few-shot text classification tasks demonstrate that our KPT++ outperforms state-of-the-art method KPT and other baseline methods. Furthermore, ablation experiments and case studies demonstrate the effectiveness of both PGR and PDR refining methods.
AB - Recently, the new paradigm “pre-train, prompt, and predict” has achieved remarkable few-shot learning achievements compared with the “pre-train, fine-tune” paradigm. Prompt-tuning inserts the prompt text into the input and converts the classification task into a masked language modeling task. One of the key steps is to build a projection between the labels and the label words, i.e., the verbalizer. Knowledgeable prompt-tuning (KPT), which integrates external knowledge into the verbalizer to improve and stabilize prompt-tuning. KPT uses word embeddings and various knowledge graphs to expand the label words space to hundreds of words per class. However, some unreasonable label words in the verbalizer may damage the accuracy. In this paper, a new method called KPT++ is proposed to improve the few-shot text classification. KPT++ is refined knowledgeable prompt-tuning, which can also be regarded as an upgraded version of KPT. Specifically, KPT++ uses two newly proposed prompt grammar refinement (PGR) and probability distribution refinement (PDR) to refine the knowledgeable verbalizer. Extensive experiments on few-shot text classification tasks demonstrate that our KPT++ outperforms state-of-the-art method KPT and other baseline methods. Furthermore, ablation experiments and case studies demonstrate the effectiveness of both PGR and PDR refining methods.
UR - http://www.scopus.com/inward/record.url?scp=85160571999&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85160571999&partnerID=8YFLogxK
U2 - 10.1016/j.knosys.2023.110647
DO - 10.1016/j.knosys.2023.110647
M3 - Article
AN - SCOPUS:85160571999
SN - 0950-7051
VL - 274
JO - Knowledge-Based Systems
JF - Knowledge-Based Systems
M1 - 110647
ER -