Why Transformers Struggle with Global Reasoning

This study examines Transformer architectures' reasoning limitations using global reasoning challenges and syllogism composition as a framework. The authors show that Transformers encounter an exponential rise in learning difficulty as task complexity increases by formalizing the cycle problem, a synthetic benchmark that necessitates long-chain logical inference. Distribution localization, a measure of how many tokens beyond the fundamental statistics are required to meaningfully correlate with the goal output, is the idea they put up to explain this.