Understanding the Local Reasoning Barrier in Transformer Models

By proposing a new metric called distribution locality, which measures the number of tokens beyond local statistics required to correlate with the target distribution, this section formalizes the local reasoning barrier in Transformers. The authors demonstrate that tasks with high locality are intrinsically challenging for Transformers trained using stochastic gradient descent using the cycle task.