How Inductive Scratchpads Help Transformers Learn Beyond Their Training Data

This section examines how Transformers get beyond the local reasoning barrier with the aid of scratchpads, which are explicit sequences of reasoning stages. By teaching models intermediary steps like cumulative operations or depth-first search (DFS) traces, traditional "educated scratchpads" effectively reduce task localization. Nevertheless, these methods frequently restrict out-of-distribution generalization by overfitting to particular sequence lengths. In order to overcome this, the study presents the inductive scratchpad, a novel formulation that simulates algorithmic induction by having the model iteratively update a state variable using masked attention and reindexed word positions.