Testing Large Language Models on Math Puzzles

Large language models (LLMs) excel at zero-shot and multi-task learning but still struggle with complex mathematical reasoning. This study introduces a 24-point puzzle problem, alongside a synthetic data pipeline, to train LLMs for multi-step calculations and extrapolation. Experiments show that scaling high-quality data improves both in-domain and out-of-domain performance, though limitations remain in tackling advanced mathematical tasks. The research highlights both progress and challenges in teaching AI to “think” numerically.