The Training Technique That Teaches AI to Think, Not Memorize

This study demonstrates that although Transformers are theoretically expressive, their capacity to learn problems demanding global composition of inputs is constrained by a local reasoning barrier. The authors demonstrate that Transformers trained from scratch are unable to learn high-locality targets in the absence of extra structure using a novel metric known as distribution locality.