Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Texts
arxiv(2024)
摘要
Pretrained Language Models (PLMs) have advanced Natural Language Processing
(NLP) tasks significantly, but finetuning PLMs on low-resource datasets poses
significant challenges such as instability and overfitting. Previous methods
tackle these issues by finetuning a strategically chosen subnetwork on a
downstream task, while keeping the remaining weights fixed to the pretrained
weights. However, they rely on a suboptimal criteria for sub-network selection,
leading to suboptimal solutions. To address these limitations, we propose a
regularization method based on attention-guided weight mixup for finetuning
PLMs. Our approach represents each network weight as a mixup of task-specific
weight and pretrained weight, controlled by a learnable attention parameter,
providing finer control over sub-network selection. Furthermore, we employ a
bi-level optimization (BLO) based framework on two separate splits of the
training dataset, improving generalization and combating overfitting. We
validate the efficacy of our proposed method through extensive experiments,
demonstrating its superiority over previous methods, particularly in the
context of finetuning PLMs on low-resource datasets.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要