Overcoming Locality in Auto-Regressive Transformers

In order to overcome Transformers' incapacity to extend to longer reasoning chains, the inductive scratchpad was developed. This section describes its architectural advances and practical implementation. The authors employ decoder-only Transformers trained from scratch in the GPT-2 approach. A training and inference process that imposes an inductive structure on the scratchpad is the main novelty. The format of the sequence is Question . State1 # State2 #..., and the model is compelled to create each new state (s[i]) solely using the prior state (s[i-1]) and the initial query (Q) thanks to advanced attention masking and positional re-indexing.