Why LLMs Struggle with Arithmetic Puzzles

This article explores how large language models like GPT-4, Llama-2, and Deepseek-Coder perform on a challenging symbolic arithmetic puzzle benchmark. Despite extensive hyperparameter tuning with LoRA, AdamW, and cosine learning schedulers, even state-of-the-art models fail to generate correct solutions. The findings highlight the limitations of Chain-of-Thought prompting and emphasize the need for specialized fine-tuning on synthetic data to tackle symbolic reasoning tasks effectively.