cbow-lora

Learning PyTorch by building two things from scratch: a CBOW word-embedding model, and a LoRA adapter that fine-tunes it without touching its weights.

Corpus is two excerpts from the Odyssey in data/.

python main.py

Layout

file	what it does
dataset.py	cleans the text, builds vocab, turns it into (context, target) pairs
src/cbow.py	the CBOW model + its training loop
src/lora.py	the LoRA adapter + its fine-tuning loop
main.py	wires it together

CBOW

Predict a word from the 4 words before it.

X  (6869, 4)      word ids, 4 per context window
y  (6869,)        the word id that follows

The model is three layers:

nn.Embedding(1346, 10)   (N, 4)      -> (N, 4, 10)   look up each id's vector
nn.Flatten()             (N, 4, 10)  -> (N, 40)      lay the 4 vectors end to end
nn.Linear(40, 1346)      (N, 40)     -> (N, 1346)    score every vocab word

N is just how many windows you pass in. 1346 is the vocab size — in the Embedding it is which word came in, in the Linear it is which word might come out.

The embedding table is the actual product here. The Linear head exists only to create a training signal; embedding_matrix() returns the learned vectors.

LoRA

LoRATune replaces the Linear head with a frozen copy of itself plus a small trainable branch:

y = W0 @ x  +  (alpha / rank) * B @ A @ x
    frozen     trainable

W0 is 53,840 weights. At rank 4, A and B are 5,544 — about 10%. Only those train; everything else is requires_grad=False.

Two details that matter:

• B starts at zero, so B @ A == 0 and the model is unchanged at step 0. Training departs from the pretrained model rather than a corrupted one.
• The adapter is swapped into the Sequential (cbow.model[2] = self). It is an nn.Module with the same in/out shape, so the rest of the model is unaffected and you keep forwarding cbow.model(X) as before.

Forward and backward are ordinary. The only LoRA-specific line is the optimizer:

torch.optim.Adam([p for p in cbow.model.parameters() if p.requires_grad])

Known limitations

This is a learning exercise, and the numbers reflect that.

• The corpus is tiny — 6,873 tokens, 1,346 unique words, 55% of which appear exactly once. Far too little to learn real embeddings.
• No train/test split. Xa and Xb are built from the same words, so the reported accuracy is measured on training data and is memorization (~99%). On held-out data it is closer to 6%.
• Because Xb == Xa, the LoRA fine-tune has nothing to learn and its loss stays flat. It needs a different corpus to do anything useful.
• Not really CBOW. True CBOW predicts a centre word from context on both sides; this uses the 4 preceding words, which makes it a 4-gram model.