How to Improve AI Models While Training Only 0.1% of Parameters

AdaMix is a parameter-efficient fine-tuning (PEFT) method for large language models that outperforms both full fine-tuning and existing PEFT approaches like LoRA and adapters. By using a mixture of adaptation modules with stochastic routing and merging, AdaMix trains only 0.1–0.2% of parameters while maintaining the same computational cost as baseline PEFT methods. This innovation dramatically reduces storage needs and boosts performance across NLU and NLG tasks, making it one of the most effective fine-tuning techniques to date.