MAMIS — Multi-Agent Manufacturing Inspection System

Automated defect detection and root-cause analysis for solar cell electroluminescence (EL) images, built as an eval-first multi-agent pipeline. Uses a domain-specific severity classifier (EfficientNet-B0) combined with a local vision-language model (Qwen3-VL via Ollama) to achieve 75.3% overall quality score across 20 evaluation criteria — a +10.4pp improvement over the VLM-only baseline.

Designed to demonstrate production-grade ML engineering practices: eval-first development, systematic ablation, structured logging, and a documented iteration path from 57% to 75%.

---

Architecture

                    ┌─────────────────────────────────────────────┐
                    │         classifier-hybrid pipeline           │
                    │                                             │
  EL Image ─────►  │  EfficientNet-B0    ──► severity label      │
  (300×300 px)      │  severity classifier   (none/low/med/high)  │
                    │                              │               │
                    │  Qwen3-VL (Ollama)  ◄────────┘              │
                    │  defect detection              │             │
                    │  + root cause analysis         │             │
                    │  + corrective actions          │             │
                    └────────────────────────────────┼────────────┘
                                                     │
                                          InspectionResult
                                          ├── severity (from classifier)
                                          ├── defects (from VLM)
                                          ├── root_cause
                                          ├── actions
                                          └── requires_human_review

Key insight: The VLM (at 57% crack recall) outperforms domain-transferred YOLO (0% crack recall) for defect detection on ELPV cell images. The severity classifier handles what the VLM does poorly: ordinal 4-class prediction on a domain-specific dataset.

---

Results

Pipeline Ablation (50-image eval suite, 20 weighted criteria)

Pipeline	Overall	Criteria Passed	Key Change
Dummy baseline	5.0%	0/20	–
VLM-only (initial)	60.3%	8/20	Qwen3-VL standalone
VLM-only + prompt v4	64.9%	10/20	Prompt engineering
YOLO-World hybrid	57.7%	10/20	Zero-shot YOLO detection
Custom YOLO hybrid	57.4%	9/20	PVEL-AD trained YOLO
Classifier hybrid	75.3%	13/20	EfficientNet-B0 severity
Oracle (perfect severity)	96.0%	19/20	Upper bound

Classifier-Hybrid: Category Breakdown

Category	Score	Status
Cell type identification	96.0%	✓
Root cause analysis	100.0%	✓
System (latency, human review)	100.0%	✓
Severity classification	65.4%	partial
Actions	80.1%	partial
Defect detection	45.5%	partial

The oracle experiment (96%) confirms the architecture is sound — severity prediction is the primary bottleneck, not the VLM reasoning layer.

See docs/ for per-pipeline failure analysis, the oracle ablation, and confusion-matrix breakdowns by category.

---

Domain Challenge

ELPV individual cell images (300×300px, subtle grayscale features) are a different visual domain from the PVEL-AD module images (1024×1024px, high contrast) used to train off-the-shelf YOLO models. The domain gap causes 0% crack recall for transferred YOLO, motivating the shift to a same-domain severity classifier.

PVEL-AD (module level)          ELPV (cell level)
┌──────────────────────┐        ┌───────────────┐
│   [high contrast]    │        │[subtle, 300px]│
│  1024 × 1024         │   vs.  │   300 × 300   │
│  multi-cell panel    │        │  single cell  │
└──────────────────────┘        └───────────────┘

---

Setup

Requirements: Python 3.13+, uv, Ollama with qwen3-vl:8b-instruct, NVIDIA GPU (tested on RTX 4060 8GB).

# Install dependencies
uv sync --all-extras

# Pull VLM (requires Ollama running)
ollama pull qwen3-vl:8b-instruct

# Download ELPV dataset
uv run python scripts/download_elpv.py

# Train severity classifier (~8 min on RTX 4060)
uv run python scripts/train_severity_classifier.py

# Run evaluation
uv run python scripts/run_eval.py --agent classifier-hybrid

---

Evaluation Framework

The eval suite (configs/eval.yaml) defines 20 criteria across 6 categories, each with a threshold and weight:

Category	Criteria	Weight
Severity classification	6	30%
Defect detection	4	20%
Root cause	3	15%
Cell type identification	2	10%
Actions	3	15%
System	2	10%

Eval cases are 50 stratified images from the ELPV dataset, held out from training. The eval pipeline is agent-agnostic — swap --agent to compare any combination.

uv run python scripts/run_eval.py --agent dummy        # 5.0%  — baseline
uv run python scripts/run_eval.py --agent single       # 64.9% — VLM only
uv run python scripts/run_eval.py --agent hybrid       # 57.4% — YOLO + VLM
uv run python scripts/run_eval.py --agent classifier-hybrid  # 75.3% — best

---

Deployment

The classifier-hybrid pipeline is wrapped as a FastAPI service with a single-page upload UI. The model loads once at app startup (via lifespan) so per-request latency is just inference.

# Install with the serve extra
uv sync --extra serve

# Run (Ollama must already be running with qwen3-vl:8b-instruct)
uv run uvicorn mamis.api.server:app --port 8000

Endpoints:

Method	Path	Purpose
GET	/	Upload UI (vanilla HTML/JS)
GET	/healthz	Liveness probe → {"status": "ok"}
POST	/inspect	multipart/form-data image upload → InspectionResult JSON

curl -F file=@cell.png http://localhost:8000/inspect

{
  "image_id": "cell",
  "cell_type": "mono",
  "severity": "medium",
  "defects": [{"defect_type": "crack", "confidence": 0.86, "description": "..."}],
  "root_cause": "Mechanical stress during stringing — a single linear crack ...",
  "actions": ["Audit stringer alignment", "Inspect adjacent cells in the batch"],
  "confidence": 0.83,
  "requires_human_review": false
}

Or run a single image from the CLI:

uv run python scripts/inspect.py path/to/cell.png

---

What I'd Do Next

The oracle ceiling at 96% means there's a ~21pp gap remaining. The two main levers:

1. Severity classifier (current bottleneck): 71.8% val accuracy, MEDIUM class recall is poor (26%) due to only 75 training samples. Options: TTA ensembling, pseudo-labelling with high-confidence predictions, or sourcing MEDIUM-heavy additional data.

2. VLM defect detection: Finger interruption recall is 0% (VLM blind to this defect type). A fine-tuned detection head or few-shot prompting with example images would address this.

---

Tech Stack

• Python 3.13 with strict type hints, Pydantic v2, pydantic-settings
• EfficientNet-B0 (torchvision) — severity classifier, ImageNet pretrained, bfloat16 AMP
• Qwen3-VL 8B via Ollama — defect detection + root cause reasoning
• structlog — structured JSON logging
• pytest — 236 unit tests

---

Project Structure

src/mamis/
├── agents/
│   ├── classifier.py      # EfficientNet-B0 severity classifier
│   ├── pipeline.py        # Hybrid pipeline orchestration
│   ├── single.py          # VLM-only agent (Ollama)
│   └── detector.py        # YOLO detection agent
├── api/
│   ├── server.py          # FastAPI app: /, /healthz, POST /inspect
│   └── static/index.html  # Single-page upload UI
├── core/
│   ├── types.py           # Domain types (SeverityLevel, DefectType, etc.)
│   ├── config.py          # pydantic-settings config
│   └── logging.py         # structlog setup
├── eval/
│   ├── criteria.py        # EvalSuite YAML loader
│   ├── scorer.py          # 18 scoring functions
│   ├── runner.py          # Agent-agnostic eval runner
│   └── report.py          # Markdown report generator
├── models/
│   └── severity.py        # Rule-based severity (YOLO bbox area ratios)
└── data/
    └── download.py        # ELPV dataset download