[
A GPT-style Decoder Transformer implemented completely from scratch in PyTorch, trained on the Tiny Shakespeare dataset to learn autoregressive character-level text generation.
This project implements the core architecture behind modern GPT models from scratch without relying on HuggingFace Transformer implementations.
The model learns to predict the next character given a sequence of previous characters and generates Shakespeare-style text through autoregressive decoding.
The implementation includes:
- Character Embeddings
- Positional Embeddings
- Causal Self-Attention
- Multi-Head Attention
- Feed Forward Networks
- Residual Connections
- Layer Normalization
- Dropout Regularization
- Checkpoint Saving & Loading
The original Transformer architecture consists of:
Encoder + Decoder
However GPT models only use the Decoder portion.
This project implements the Decoder stack shown below.
graph TD
A[Input Characters]
--> B[Character Embeddings]
B --> C[Positional Embeddings]
C --> D[Transformer Block x6]
D --> E[Final LayerNorm]
E --> F[Linear Projection]
F --> G[Softmax]
G --> H[Next Character Prediction]
Each Transformer block contains:
graph TD
A[Input]
A --> B[LayerNorm]
B --> C[Multi-Head Attention]
C --> D[Residual Add]
D --> E[LayerNorm]
E --> F[Feed Forward Network]
F --> G[Residual Add]
G --> H[Output]
The model uses causal self-attention.
Each token creates:
- Query (Q)
- Key (K)
- Value (V)
Attention scores are computed as:
Attention(Q,K,V)
=
Softmax(QKᵀ / √dₖ)V
A causal mask prevents the model from seeing future tokens during training.
1 0 0 0
1 1 0 0
1 1 1 0
1 1 1 1
This ensures autoregressive generation.
graph TD
A[Tiny Shakespeare Dataset]
--> B[Character Tokenization]
B --> C[Training Batches]
C --> D[Transformer]
D --> E[Cross Entropy Loss]
E --> F[Backpropagation]
F --> G[Adam Optimizer]
G --> H[Updated Parameters]
The project was developed incrementally:
Bigram Language Model
↓
Single Attention Head
↓
Multi-Head Attention
↓
Feed Forward Network
↓
Residual Connections
↓
Layer Normalization
↓
Dropout
↓
6-Layer Transformer
| Parameter | Value |
|---|---|
| Batch Size | 64 |
| Context Length | 256 |
| Embedding Size | 384 |
| Attention Heads | 6 |
| Transformer Layers | 6 |
| Dropout | 0.2 |
| Learning Rate | 3e-4 |
| Optimizer | Adam |
| Training Steps | 5000 |
| Step | Train Loss | Validation Loss |
|---|---|---|
| 500 | 2.0010 | 2.0880 |
| 1000 | 1.5949 | 1.7786 |
| 1500 | 1.4397 | 1.6438 |
| 2000 | 1.3404 | 1.5714 |
| 2500 | 1.2786 | 1.5331 |
| 3000 | 1.2264 | 1.5101 |
| 3500 | 1.1820 | 1.4916 |
| 4000 | 1.1464 | 1.4904 |
| 4500 | 1.1095 | 1.4804 |
| 4999 | 1.0763 | 1.4873 |
LUCIO:
We muse hath resistes him so soveree:
son't his other wrough stands of coverent sh'd:
he has here, and stand it and poor exceder...
The model successfully learns:
- Shakespeare-style dialogue formatting
- Character names
- Sentence structure
- English grammar patterns
- Long-range context relationships
nano-gpt/
│
├── README.md
├── requirements.txt
├── .gitignore
│
├── data/
│ └── input.txt
│
├── checkpoints/
│ └── .gitkeep
│
├── samples/
│
└── src/
├── config.py
├── dataset.py
├── model.py
├── train.py
└── generate.py
git clone https://github.com/yourusername/nano-gpt.git
cd nano-gpt
pip install -r requirements.txtpython src/train.pypython src/generate.py- GPT-2 Tokenizer (BPE)
- GELU Activations
- Flash Attention
- Mixed Precision Training
- Larger Datasets
- HuggingFace Integration
- GPT-2 Scale Architecture
- Attention Is All You Need (2017)
- Andrej Karpathy's NanoGPT
- PyTorch Documentation