NIST Cybersecurity Fine-Tuning Pipeline

This repo is the release version of the NIST dataset and MLX fine-tuning workflow I built under hackIDLE.

The original nist-tuned-model workspace started on October 12, 2025. Claude assisted as an agent while I was building and cleaning this up. Everything here is meant to be runnable without digging through the old working directory.

On Apple Silicon, you can use it to:

• download the source publications from NIST
• extract them with Docling and a fallback path for ugly PDFs
• generate the training dataset locally
• fine-tune with MLX
• package the result for Ollama through GGUF
• branch the same corpus into CMMC and HIPAA follow-on datasets
• generate vector embeddings for RAG-style follow-up work

It includes:

• download public NIST cybersecurity publications from NIST CSRC
• extract structured text from PDFs with Docling
• fall back to MarkItDown or simpler extractors when needed
• generate chat-style training data with sentence-aware chunking
• fine-tune a local model with MLX on Apple Silicon
• package the result for Ollama through GGUF
• run a small Ollama smoke eval against known NIST questions
• create CMMC and HIPAA dataset variants from the same project
• generate optional embeddings and publish the outputs to Hugging Face

The original workspace picked up model artifacts, large datasets, embeddings, CMMC and HIPAA spinouts, Hugging Face upload utilities, and a lot of dead-end debugging. This repo pulls the reproducible parts back out and leaves the heavy artifacts behind.

Workflow

The main path is:

1. Download NIST PDFs into data/raw/
2. Extract them into data/extracted/
3. Build JSONL training data in data/training/
4. Fine-tune a Qwen2.5-Coder model with MLX LoRA
5. Fuse the adapter, export GGUF files, and create an Ollama model

The key files are:

• src/extract_with_fallback.py uses Docling first and MarkItDown as fallback.
• src/extract_single_pdf.py exists for stubborn edge-case PDFs.
• src/prepare_data_chunked.py creates chat examples with sentence-aware chunking.

The CMMC, HIPAA, embeddings, and Hugging Face upload flows are in docs/extensions.md.

Current model-eval notes and known limitations are in docs/model-evaluation.md.

The dated build timeline and public artifact map are in docs/project-history.md and docs/artifact-map.md.

What you can rebuild

If you're on Apple Silicon, you should be able to:

1. download the source publications from NIST
2. regenerate the extracted corpus
3. regenerate the training dataset locally
4. run the MLX fine-tuning workflow
5. package the result for local use in Ollama
6. branch the extracted corpus into CMMC and HIPAA training datasets
7. generate embeddings from any finished training split

It does not include the already-generated large artifacts. It includes the code to reproduce them.

What is not in this repo

I'm not checking in:

• raw PDFs
• extracted document dumps
• training JSONL artifacts
• generated CMMC and HIPAA corpora
• generated embeddings, Parquet files, or FAISS indexes
• fused model weights, GGUF files, or LoRA checkpoints

The code for those workflows is here. The generated artifacts are not.

Quick Start

Start small:

make setup
make sample-download SAMPLE_DOCS=2
make extract
make prepare

Then scale up to the full dataset pipeline:

make download
make extract
make prepare
make train
make deploy

If your llama.cpp tools are not already on PATH, point the deploy step at a local checkout:

LLAMA_CPP_DIR=/path/to/llama.cpp make deploy

After packaging a local model, run the smoke eval:

make eval

If you want the wrapper script:

make pipeline

For the step-by-step version, read QUICKSTART.md.

For the extension paths for CMMC, HIPAA, embeddings, and Hugging Face publication, read docs/extensions.md.

Repository Layout

.
├── config/
│   ├── Modelfile
│   └── training_config.yaml
├── data/
│   ├── cmmc/
│   ├── embeddings/
│   └── hipaa/
├── docs/
│   ├── artifact-map.md
│   ├── extensions.md
│   ├── model-evaluation.md
│   └── project-history.md
├── eval/
│   ├── results/
│   └── ollama_smoke_cases.json
├── examples/
│   └── example_queries.md
├── model-cards/
│   ├── huggingface/
│   └── ollama/
├── requirements-extensions.txt
├── scripts/
│   ├── chat_with_model.py
│   ├── copy_cmmc_extracted_files.py
│   ├── create_cmmc_document_lists.py
│   ├── deploy_to_ollama.sh
│   ├── eval_ollama.py
│   ├── download_cfr_html.py
│   ├── download_fda_guidance.py
│   ├── download_hipaa_docs.py
│   ├── download_ocr_guidance.py
│   ├── extract_failed.sh
│   ├── extract_hipaa_v11_docs.py
│   ├── generate_embeddings.py
│   ├── monitor_progress.sh
│   ├── run_full_pipeline.sh
│   ├── upload_cmmc_to_huggingface.py
│   ├── upload_raw_pdfs.py
│   └── upload_to_huggingface.py
├── src/
│   ├── download_all_nist.py
│   ├── download_samples.py
│   ├── extract.py
│   ├── extract_single_pdf.py
│   ├── extract_with_fallback.py
│   ├── prepare_cmmc_data.py
│   ├── prepare_data_chunked.py
│   ├── prepare_hipaa_data.py
│   └── train.py
├── QUICKSTART.md
└── Makefile

Defaults

The defaults in this repo are:

• base model: mlx-community/Qwen2.5-Coder-7B-Instruct-4bit
• fine-tuning method: LoRA with MLX
• default Ollama model name: hackidle-nist-coder

If you want to experiment with a smaller base model during QA, override BASE_MODEL in the shell or edit config/training_config.yaml.

For the optional extension dependencies for embeddings and publication helpers, run make setup-extensions.

Requirements

• Apple Silicon Mac if you want to follow the MLX training path directly
• Python 3
• network access on the first extraction run so Docling can fetch its model weights
• ollama plus llama.cpp conversion tools if you want the local Ollama packaging step
• enough disk for the downloaded PDFs, extracted text, and generated training data
• enough RAM for the model variant you choose

The deploy path refreshes tokenizer metadata from the base model before GGUF conversion so the local Ollama model keeps the right chat template. If you swap away from the default Qwen base model for QA, use a matching MODELFILE_TEMPLATE when you run make deploy.

If your main goal is only to recreate the dataset, you do not need to complete the training step.

What was in the original workspace

The numbers changed as the dataset grew. In the original workspace:

• 596 raw NIST PDFs in the main data/raw/ folder
• 596 extracted JSON files
• 596 extracted Markdown files
• a later training_v1.1_complete/data_stats.json artifact reporting 530,912 examples

One historical bug in the prep flow was that both JSON and Markdown outputs could be counted for the same source document. This repo fixes that by preferring JSON and deduplicating by document stem during training-data generation.

Published outputs

These are the public artifacts from the same project:

• main training dataset: ethanolivertroy/nist-cybersecurity-training
• raw source PDFs: ethanolivertroy/nist-publications-raw
• CMMC variants: core, balanced, comprehensive
• HIPAA dataset: ethanolivertroy/hipaa-compliance-training
• Ollama model: etgohome/hackidle-nist-coder
• Hugging Face models: MLX 4-bit, GGUF, v1.1 MLX 4-bit, v1.1 GGUF
• GitHub repo: hackIDLE/nist-cybersecurity-mlx-pipeline