Code for the paper "Recurrent Reasoning on Symbolic Puzzles with Sequence Models", accepted at the ICLR 2026 Workshop on Logical Reasoning of LLMs.
Authors: Gowrav Mannem, Mahjabin Chowdhury, Jason Chen, Shivank Garg, Kevin Zhu
This repository contains the training and evaluation scripts for all experiments in the paper. We study how sequence models [ T5 (encoder-decoder) and GPT-2 (decoder-only) ] reason autoregressively on four symbolic planning puzzles of increasing difficulty, comparing random initialization against pre-trained weights and evaluating generalization to out-of-distribution puzzle sizes.
Key finding: Puzzle structure, not model size or pre-training, is the primary determinant of learnability. T5's bidirectional goal-conditioning gives it a consistent advantage over GPT-2 across all puzzles.
All training and evaluation scripts in this repository were implemented by Chowdhury Mahjabin.
All data is loaded automatically from HuggingFace at runtime. No manual download required.
Dataset: gmannem/RecurrReason
| Subset | HuggingFace name |
|---|---|
| Checkers Jumping | checkers_jumping |
| Tower of Hanoi | tower_of_hanoi |
| Block World | block_world |
| River Crossing | river_crossing |
Each subset has a train split (N=1–7) and a test split (N=8–10) used for out-of-distribution [OOD] evaluation.
RecurrReason/
├── README.md
├── requirements.txt
└── training/
├── checkers_jumping/
│ ├── t5_scratch.py # T5 trained from random initialization
│ ├── t5_pretrained.py # T5 with zero-shot eval + fine-tuning
│ ├── gpt2_scratch.py # GPT-2 trained from random initialization
│ └── gpt2_pretrained.py # GPT-2 with zero-shot eval + fine-tuning
├── tower_of_hanoi/
│ ├── t5_scratch.py
│ ├── t5_pretrained.py
│ ├── gpt2_scratch.py
│ └── gpt2_pretrained.py
├── block_world/
│ ├── t5_scratch.py
│ ├── t5_pretrained.py
│ ├── gpt2_scratch.py
│ └── gpt2_pretrained.py
└── river_crossing/
├── t5_scratch.py
├── t5_pretrained.py
├── gpt2_scratch.py
└── gpt2_pretrained.py
Each script is self-contained and runs end-to-end: it loads data, trains the model with early stopping, and evaluates autoregressively on both the validation split and the OOD test split.
git clone https://github.com/chowdhury-mahjabin/RecurrReason.git
cd RecurrReason
pip install -r requirements.txtA GPU is strongly recommended. All scripts automatically detect and use CUDA if available.
Each script is run directly. For example, to train T5 from scratch on Checkers Jumping:
python training/checkers_jumping/t5_scratch.pyTo run the pre-trained GPT-2 experiment on Tower of Hanoi (includes zero-shot evaluation before fine-tuning):
python training/tower_of_hanoi/gpt2_pretrained.pyAll scripts follow the same pattern:
*_scratch.py— trains from random initialization, evaluates on validation and OOD*_pretrained.py— first evaluates zero-shot, then fine-tunes, then evaluates again
Training uses early stopping (patience=5, max 100 epochs), AdamW with lr=1e-4, batch size=16. Checkpoints and training curve plots are saved to the working directory.
Autoregressive accuracy on unique puzzles (validation N=1–7 / OOD N=8–10):
| Puzzle | T5 Scratch | T5 Pre-trained | GPT-2 Scratch | GPT-2 Pre-trained |
|---|---|---|---|---|
| Block World | — / — | 97.27% / 81.00% | — / — | — / — |
| Tower of Hanoi | 0.00% / 0.00% | 11.11% / 0.00% | 0.00% / 0.00% | 0.00% / 0.00% |
| Checkers Jumping | 0.00% / 0.00% | 1.11% / 0.10% | 0.00% / 0.00% | 0.00% / 0.00% |
| River Crossing | 0.00% / 0.00% | 0.00% / 0.00% | 0.00% / 0.00% | 0.00% / 0.00% |
T5 (encoder-decoder):
- Both input state and goal state are encoded together in the encoder, giving the decoder full bidirectional context at every generation step.
- Pre-trained on C4 corpus (t5-small, ~60M parameters).
GPT-2 (decoder-only):
- Causal attention means goal context is only available from earlier positions in the sequence, which limits its ability to condition generation on the goal at each step.
- Pre-trained on WebText corpus (~124M parameters).
Block World uses a custom symbolic vocabulary (SOS, EOS, SEP, PEG_A/B/C, D, #1–#10, N1–N3) with max_input_length=256 to accommodate longer state representations. GPT-2 Block World scripts also add these tokens to the vocabulary via tokenizer.add_tokens().
River Crossing includes missionary/cannibal constraint checking during autoregressive evaluation — moves that violate the constraint (cannibals outnumbering missionaries on any bank or in the boat) are rejected as invalid.
If you use this code or dataset, please cite:
@inproceedings{mannem2026recurreason,
title = {Recurrent Reasoning on Symbolic Puzzles with Sequence Models},
author = {Mannem, Gowrav and Chowdhury, Mahjabin and Chen, Jason and Garg, Shivank and Zhu, Kevin},
booktitle = {ICLR 2026 Workshop on Logical Reasoning of LLMs},
year = {2026}
}MIT