Beating Full Fine-Tuning with Just 0.2% of Parameters

AdaMix is a new framework for parameter-efficient fine-tuning (PEFT) of large pretrained language models. Unlike single adaptation methods, AdaMix leverages a mixture of modules with stochastic routing and weight merging, achieving state-of-the-art results in both natural language understanding and generation tasks. By tuning only 0.1–0.2% of parameters, it outperforms full model fine-tuning and existing PEFT approaches like adapters and LoRA, though at a slightly higher training cost.