The Science of Reasoning in Large Language Models

This study places its conclusions about the "local reasoning barrier" in the context of a larger body of work on Transformer capacities. It makes a distinction between tasks that need real reasoning, which frequently entail combinatorial complexity and poor out-of-distribution (OOD) generalization, especially on length generalization, and those that can be solved by memorization, where Transformers flourish because of the large amount of training data. Important architectural elements that affect these capabilities, such as positional embeddings and recurrent alterations, are covered in the literature review.