Convert handwritten equations into publication-ready math code using Google's open-source AI model, fine-tuned on a free cloud GPU.
Built with Unsloth · Runs on free Google Colab · Based on Gemma 4 E2B by Google DeepMind
This notebook teaches an AI model to look at a photo of a handwritten math equation and convert it into LaTeX — the formatting language used by researchers, scientists, and students to write clean, typeset mathematics.
Before training, the model has no idea what to do with a handwritten image. After just a few minutes of fine-tuning on a free GPU, it can read handwriting and output properly formatted math code.
Input: A photo of a handwritten equation
Output: LaTeX code like \frac{N}{M} \in \mathbf{Z}
This kind of tool, converting handwriting to LaTeX automatically, could help students, researchers, and educators in resource-limited environments make their work shareable and publishable, instantly.
By working through this notebook, you will understand how to:
- Load a large AI vision model efficiently on a free GPU
- Prepare a real dataset of handwritten math images for training
- Fine-tune a model using a technique called LoRA (explained below)
- Run inference — meaning: give the model a new image and get a prediction back
- Save your trained model for reuse
Gemma 4 E2B Google's open-source AI model with 2 billion parameters. It can understand both images and text — making it "multimodal." The E2B variant is designed to be efficient enough to run on a laptop or free cloud GPU.
Unsloth A tool that makes loading and training large AI models faster and cheaper. It reduces the memory needed by up to 50%, which is why this notebook runs on a free GPU instead of expensive hardware.
LoRA (Low-Rank Adaptation) Instead of retraining the entire model (which would take days and cost a lot), LoRA only updates a tiny slice of the model's weights — about 1% of all parameters. It's like teaching someone a new skill without erasing everything they already know. Faster, cheaper, and surprisingly effective.
Fine-tuning The process of taking a pre-trained model (one that already knows a lot about language and images) and training it further on a specific task — in this case, converting handwritten math to LaTeX.
LaTeX
A text-based formatting system used in academic and scientific writing to display math. For example, \frac{a}{b} renders as a fraction. It's the standard format for publishing equations in research papers.
Inference Running the model on new data to get a prediction. After training, this is where you give the model a handwritten image and it outputs the LaTeX code.
Processor In multimodal models, the processor handles both the image and the text — preparing them in a format the model can understand. Think of it as a translator that converts your raw input into something the AI can read.
- A Google account (to access Google Colab)
- No local GPU or special hardware needed
- Free Google Colab account (uses a Tesla T4 GPU)
- A Hugging Face account (free) to download the model
Note: The model download requires accepting Google's Gemma license on Hugging Face. It's free and takes about 30 seconds.
Step 1 — Open in Colab
Click the "Open in Colab" button at the top of the notebook file.
Step 2 — Switch to a GPU runtime
In the top-right corner of Colab, click the dropdown arrow next to "Connect," then select "Change runtime type." Under "Hardware accelerator," choose T4 GPU. Click Save.
Step 3 — Run all cells
Go to Runtime > Run all. The notebook will run from top to bottom automatically.
Step 4 — Wait for training
Training takes about 5 minutes for 60 steps on the free T4 GPU. You will see a progress bar and a loss table (the lower the loss number, the better the model is learning).
Step 5 — See the results
After training, the notebook runs inference on a test image and prints the LaTeX output. Compare the result before and after training to see how much the model improved.
| Section | What It Does |
|---|---|
| Installation | Installs all required libraries, including Unsloth, and sets up the GPU environment |
| Load Model | Downloads Gemma 4 E2B and loads it in memory-efficient 16-bit mode |
| LoRA Setup | Adds lightweight training adapters so only 1% of model weights are updated |
| Data Prep | Loads 68,000+ handwritten math images paired with their LaTeX labels |
| Before Training | Runs the model on a test image before fine-tuning — the output is poor |
| Training | Fine-tunes for 60 steps (~5 minutes). Loss drops from ~13 to ~1.5 |
| Inference | Runs the fine-tuned model on a new image — output is now accurate LaTeX |
| Save | Saves the LoRA adapter locally or pushes it to Hugging Face Hub |
This notebook uses the LaTeX OCR dataset — a collection of handwritten math formula images paired with their LaTeX representations.
- Training split: 68,686 examples
- Test split: 7,632 examples
The full original dataset is available at linxy/LaTeX_OCR.
| Metric | Value |
|---|---|
| GPU | Tesla T4 (free Colab) |
| Training time | ~5.79 minutes |
| Steps | 60 |
| Starting loss | ~13 |
| Final loss | ~1.5 |
| Trainable parameters | 59.7M out of 5.18B (1.15%) |
| Peak GPU memory used | 10.85 GB out of 14.56 GB |
A lower loss number means the model is getting better at predicting the correct LaTeX output.
The notebook saves the fine-tuned LoRA adapter (the small set of updated weights) locally to a folder called gemma_4_lora.
To share your model publicly on Hugging Face, uncomment these lines in the Save section:
model.push_to_hub("your_username/gemma_4_lora", token="YOUR_HF_TOKEN")
processor.push_to_hub("your_username/gemma_4_lora", token="YOUR_HF_TOKEN")The saved file contains only the LoRA adapter, not the full model. To use it again, you load the base Gemma 4 model and attach the adapter on top.
- Unsloth Documentation
- Gemma 4 on Hugging Face
- LaTeX OCR Dataset
- Unsloth Discord — helpful community for troubleshooting
This notebook is based on Unsloth's open-source notebooks available on their Github account, licensed under LGPL-3.0. The Gemma 4 model is released under the Apache 2.0 license.