You Should Stop Fine-Tuning Blindly: What to Do Instead

Fine-tuning is not one thing. You’re choosing a point on a spectrum: Full FT → PEFT (Adapters/Prompt Tuning/LoRA) → QLoRA → Preference tuning (RLHF/DPO).- Most teams should start with PEFT (LoRA/QLoRA). Full fine-tuning is expensive, fragile, and easier to overfit.- The best decision rule is boring: **data quality + task stability + deployment constraints** decide everything.- If you have <100 labelled samples, you probably shouldn’t fine-tune. Do prompting + retrieval + synthetic data first.