KaziLab — Implementation Guide (NixOS)

Phase 1: Network Foundation (MikroTik hAP ax3)

1.1 Basic Configuration

[ ] Reset router to factory defaults (Netinstall if needed)
[ ] Configure WAN interface: ether5 as DHCP client (Airtel cellular modem)
[ ] Create internal bridge for management ports (ether2–ether4)
[ ] Assign management IP: 172.20.0.1/24 on the bridge
[ ] Enable SSH: disable password auth, add admin Ed25519 public key

1.2 VLAN Setup

[ ] Create VLAN 89 (Teachers) and VLAN 90 (Students) on ether1 trunk
[ ] Configure ether1 as trunk port: VLAN 1 untagged, VLAN 89/90 tagged
[ ] Create bridge per VLAN with appropriate IP addresses (see network.org)

1.3 DHCP Servers

[ ] DHCP for Management (172.20.0.0/24) — for admin access
[ ] DHCP for Students (172.20.90.0/24) — with PXE options:
- Option 66 (Next Server): IP of web01 container (172.20.90.21 or 0.21)
- Option 67 (Boot File Name): http://172.20.0.21/boot/boot.ipxe
[ ] DHCP for Teachers (172.20.89.0/24) — dynamic pool, no PXE options
[ ] Register static DHCP leases for all thin clients by MAC address (see network.org)

1.4 DNS, NTP, NAT

[ ] Upstream DNS: Cloudflare Family 1.1.1.3 / 1.0.0.3 (malware + adult content filter)
[ ] Enable NTP client (pool.ntp.org); enable NTP server broadcast for local network
[ ] Timezone: Africa/Dar_es_Salaam
[ ] NAT masquerade on ether5 (WAN)

1.5 Firewall Rules

[ ] Drop all traffic from Students/Wi-Fi (.90.x, .91.x) to Teachers (.89.x) and Management (.0.x)
[ ] Drop all traffic from Teachers to Management network
[ ] Management and admin interfaces accessible only from Management subnet and WireGuard peers
[ ] Wi-Fi: enable AP isolation (station-roaming=no)

1.6 WireGuard VPN (Remote Access)

[ ] Generate WireGuard keypair on hAP ax3
[ ] Configure tunnel to VPS/home server in Germany (persistent-keepalive to punch through CGNAT)
[ ] Verify tunnel stays up after cellular reconnect

1.7 Bandwidth Management (QoS)

[ ] Per-student downstream/upstream queue on 172.20.90.0/24
[ ] Block or throttle video streaming services via Layer-7 or DNS blackhole

Phase 2: NixOS Server Installation

2.1 Prepare Installation Media

[ ] Download latest NixOS minimal ISO
[ ] Write to USB: dd if=nixos-minimal.iso of=/dev/sdX bs=4M status=progress
[ ] Boot server from USB

2.2 Partition and ZFS Pool Setup (System Drives — 3-way Mirror)

# Identify the three datacenter SSDs (SM883), e.g. /dev/sda, /dev/sdb, /dev/sdc
# Partition each identically:
for disk in /dev/sda /dev/sdb /dev/sdc; do
  parted $disk -- mklabel gpt
  parted $disk -- mkpart ESP fat32 1MB 512MB
  parted $disk -- set 1 esp on
  parted $disk -- mkpart primary 512MB 100%
done

# Create ZFS 3-way mirror (rpool) on partition 2 of each drive
zpool create -f \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -O xattr=sa \
  -O mountpoint=none \
  rpool mirror /dev/sda2 /dev/sdb2 /dev/sdc2

# System dataset (128 GB quota)
zfs create -o refreservation=none -o quota=128G rpool/nixos
zfs create rpool/nixos/root
zfs create rpool/nixos/nix
zfs create rpool/nixos/var

# Admin/teacher home and critical data (remaining space)
zfs create rpool/critical
zfs create rpool/critical/homes

2.3 ZFS Pool Setup (Student Data — 2-drive Mirror)

# Two Samsung 870 EVO 2TB drives, e.g. /dev/sdd, /dev/sde
zpool create -f \
  -o ashift=12 \
  -O compression=zstd \
  -O atime=off \
  -O xattr=sa \
  -O mountpoint=none \
  tank mirror /dev/sdd /dev/sde

# Student home directories with per-student quotas (set via NixOS config)
zfs create tank/students

# Shared folders
zfs create -o readonly=off  tank/ClassMaterial   # teachers write, students read
zfs create -o readonly=off  tank/DropBox         # students write-only drop-in

2.4 Mount Filesystems for Installation

mount -t zfs rpool/nixos/root /mnt
mkdir -p /mnt/{boot,nix,var}
mount -t zfs rpool/nixos/nix  /mnt/nix
mount -t zfs rpool/nixos/var  /mnt/var
# Boot: use the EFI partition from the first system drive
mkfs.fat -F32 /dev/sda1
mount /dev/sda1 /mnt/boot

2.5 Generate and Edit NixOS Configuration

nixos-generate-config --root /mnt
# Edit /mnt/etc/nixos/configuration.nix to set:
# - hostname: ngarumavtc1
# - static IP: 172.20.0.10/24, gateway 172.20.0.1
# - SSH: passwordAuthentication = false; authorizedKeys for admin
# - ZFS pool import and dataset mounts
# - NUT (UPS) integration
# - Enable WireGuard peer config

2.6 Install and Reboot

nixos-install
reboot
# Remove USB stick

Phase 3: NixOS Declarative Configuration

3.1 Git Repository Structure

After first boot, initialise the config repository:

cd /etc/nixos
git init
git add .
git commit -m "initial NixOS configuration"
# Push to GitHub: NgarumaVTC/gitNixos (public)

The repository becomes the Single Point of Truth for the entire system. All changes go through Git. Roll back with nixos-rebuild switch --rollback or by reverting a commit and rebuilding.

3.2 Recommended Module Structure

/etc/nixos/
  configuration.nix       # top-level imports
  common/
    network.nix           # ALL IPs, MACs, VLANs defined here
    zfs.nix               # pool, dataset, snapshot schedules
    users.nix             # admin accounts, SSH keys
    nut.nix               # UPS shutdown integration
    wireguard.nix         # VPN peer config
  containers/
    ct-auth.nix           # LLDAP container
    web01.nix             # Nginx + iPXE server
    kiwix01.nix           # offline knowledge base
    students.nix          # 20x XFCE4/RDP workspace containers
  hardware-configuration.nix

3.3 NixOS Containers (systemd-nspawn)

Each service runs in an isolated nixos-containers unit. Example skeleton for a student workspace container:

# containers/students.nix (generates ct90051 through ct90070)
containers."ct90051" = {
  autoStart = true;
  privateNetwork = true;
  hostAddress = "172.20.90.1";
  localAddress = "172.20.90.51";
  bindMounts."/home/student" = {
    hostPath = "/tank/students/student51";
    isReadOnly = false;
  };
  config = { pkgs, ... }: {
    imports = [ ./studentssoftware.nix ];
    services.xrdp.enable = true;
    services.xrdp.defaultWindowManager = "xfce4-session";
  };
};

3.4 UPS Integration (NUT)

# common/nut.nix
power.ups = {
  enable = true;
  mode = "standalone";
  ups."apc750" = {
    driver = "usbhid-ups";
    port = "auto";
    description = "APC Smart-UPS 750VA";
  };
};
# Configure upsmon to trigger clean shutdown at battery low

Phase 4: iPXE Boot Infrastructure

4.1 web01 Container (Nginx + Boot Images)

The web01 NixOS container serves iPXE boot images over HTTP. Multi-homed: accessible from both Admin (172.20.0.21) and Students (172.20.90.21) networks via bind-mounts.

# containers/web01.nix
containers."web01" = {
  autoStart = true;
  bindMounts."/var/www/boot" = {
    hostPath = "/srv/boot";
    isReadOnly = true;
  };
  config = { pkgs, ... }: {
    services.nginx.enable = true;
    services.nginx.virtualHosts."_".root = "/var/www/boot";
  };
};

4.2 NixOS Diskless Thin Client

The thin client is a full NixOS system, declared in hosts/client/configuration.nix and netbooted via iPXE:

tmpfs root (512 MB) — no local writes survive a reboot
/nix/store mounted read-only via NFS from the server (single store, all 20 clients see the same paths)
/home mounted read-write via NFS for per-student persistent home directories
Local XFCE4 desktop with Firefox-ESR, LibreOffice, VLC, GIMP, Inkscape, etc.
Auth via sssd against the central LLDAP
Intel microcode loaded early (separate iPXE initrd segment, see boot.ipxe below)

To rebuild the client image: nixos-rebuild build --flake .#client on the server, then update the clientSystem store path in hosts/web/configuration.nix and switch.

# boot.ipxe (served by web01) — two initrds: microcode first, then main initrd
#!ipxe
kernel http://172.20.0.21/vmlinuz init=/nix/store/.../init ip=dhcp quiet
initrd http://172.20.0.21/microcode.img
initrd http://172.20.0.21/initrd
boot

4.3 TFTP / iPXE Chainloading

The MikroTik DHCP points clients to the Unidirectional iPXE binary via TFTP (Option 66/67). iPXE then immediately switches to HTTP for the larger kernel and initramfs — much faster and more reliable than TFTP for large files.

Phase 5: Identity & User Management

5.1 LLDAP Container (ct-auth)

LLDAP provides a minimal LDAP server with a clean web UI at http://172.20.90.12:17170. Backend: SQLite (minimal footprint, no PostgreSQL dependency).

# hosts/auth/configuration.nix (mounted as container ctauth)
containers."ctauth" = {
  autoStart = true;
  localAddress = "172.20.90.12";
  config = { pkgs, ... }: {
    services.lldap = {
      enable = true;
      settings.ldap_base_dn = "dc=ngarumavtc,dc=lan";
    };
  };
};

5.2 Three Roles, Three LDAP Groups

The school maps to three roles, each represented by an lldap group:

Role	Group	Who	Login Targets
Admin	`lldap_admin`	Msoya, Ngowi, Ramge	mgmt1 + host SSH (wheel)
Teacher	`lldap_teacher`	Mrosso, Albert, …	mgmt1 (XRDP)
Student	`lldap_student`	~120 apprentices	thin clients only

Built-in lldap groups remain in place:

lldap_admin (id 1, system) gives the LDAP-admin permission inside lldap itself
lldap_strict_readonly (id 3) is used for the readonly bind account that sssd uses

5.2.1 Add `gidNumber` to the group schema (one-off)

Lldap groups have no gidNumber attribute by default, so they aren’t visible as POSIX groups via NSS. Without that, chgrp lldap_teacher and getent group fail. Fix once, via GraphQL:

# Add the schema attribute
curl -sf "$LLDAP/api/graphql" -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query":"mutation{addGroupAttribute(name:\"gidnumber\",attributeType:INTEGER,isList:false,isVisible:true,isEditable:true){ok}}"}'

# Assign GIDs (4001/2/3 chosen to stay out of user UID ranges)
for pair in "1:4001" "5:4002" "4:4003"; do
  gid_id="${pair%%:*}"; gid_num="${pair##*:}"
  curl -sf "$LLDAP/api/graphql" -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d "{\"query\":\"mutation{updateGroup(group:{id:${gid_id},insertAttributes:[{name:\\\"gidnumber\\\",value:[\\\"${gid_num}\\\"]}]}){ok}}\"}"
done

After this, id <user> on sssd-enabled hosts will list the LDAP group, and chgrp lldap_teacher /media/ClassMaterial works.

5.3 Student Provisioning

users/provision-students.sh reads users/students.csv and, per student, creates:

A ZFS dataset homes/student<NNNNN> (mounted at /home/student<NNNNN>)
An lldap user with uidnumber / gidnumber attributes
Group membership in lldap_student (id 4)
An initial password (currently hardcoded; future: per-user random with force-change)

CSV format:

id,nickname,firstNames,lastName,sex,trade,entryDate
13,Vaileth,Vaileth Asia,Mushi,F,ICT,2026-04-01
...

UID is computed as 10000 + id. At end-of-year: snapshot the home dataset, disable the lldap user, eventually zfs destroy the dataset.

5.4 Staff Provisioning (Admins + Teachers)

users/provision-staff.sh mirrors the student script but writes to the high-trust pool. Per staff member:

ZFS dataset tank/data/staff/<username> (under /staff/<username> on the host)
lldap user with uidnumber / gidnumber
Group membership in lldap_admin or lldap_teacher depending on the role column
Random 16-character initial password, printed to stdout at the end of the run for the admin to hand over in person

CSV format (users/staff.csv):

uid,username,fullname,role
1001,ramge,Axel Ramge,admin
2001,msoya,Msoya Lameck,admin
2010,mrosso,Mrosso Boniface,teacher
...

Admins normally also exist as declarative NixOS users (users/<name>/nixos.nix) so they have SSH-key-based root access independent of lldap. Teachers exist only in lldap.

Phase 6: Admin/Teacher Workstation (mgmt1)

The mgmt1 container is where humans actually work. mkuu1 stays a “dumb” host that runs ZFS, NFS, and containers — administrators don’t SSH into it directly except for emergency repair.

6.1 What mgmt1 provides

services.openssh for admin SSH access (key-based)
services.xrdp + XFCE4 for teacher GUI access on port 3389
services.sssd authenticating against lldap, with simple_allow_groups replaced by an ldap_access_filter (see below)
Bind-mounts from the host:
- /home ← /staff (staff homes appear naturally as /home/<user>)
- /home/ramge/sync ← Flake-repo on the host (admins edit and rebuild in-place)
- /media/ClassMaterial ← tank/data/lehrpult (teachers populate class material)
Pre-installed tools: git, gh, claude-code, sops, age, nix, vim, plus Teacher-facing apps firefox-esr, libreoffice, evince, gimp, inkscape

6.2 SSSD access control — two non-obvious requirements

`ldap_access_filter` against `memberOf`

Since lldap groups historically had no gidNumber, simple_allow_groups can’t see membership. Even after adding gidNumber, the cleaner approach is a direct LDAP filter:

[domain/lldap]
access_provider = ldap
ldap_access_order = filter
ldap_access_filter = (|(memberOf=cn=lldap_admin,ou=groups,dc=ngarumavtc,dc=lan)(memberOf=cn=lldap_teacher,ou=groups,dc=ngarumavtc,dc=lan))

Students are rejected even though their LDAP user object resolves cleanly via NSS.

`sssdStrictAccess` on every PAM service

The NixOS default places pam_sss as sufficient in the account phase. If SSSD denies access, the stack continues to pam_unix, which is required and simply confirms the user exists via NSS — net result: the denied user is let in.

security.pam.services.login.sssdStrictAccess     = true;
security.pam.services.xrdp-sesman.sssdStrictAccess = true;
security.pam.services.sshd.sssdStrictAccess      = true;

Symptom if forgotten: sssctl user-checks -a acct -s sshd <denied_user> returns Success despite the filter clearly excluding the user.

6.3 ClassMaterial: Teacher write access via POSIX group

With gidNumber now on lldap_teacher (4002), the shared folder is owned by that group with setgid set so new files inherit the group:

systemd.tmpfiles.rules = [
  "d /media/ClassMaterial 2775 root 4002 -"
];

Teachers write directly to /media/ClassMaterial from mgmt1. (Students don’t currently see this directory at all — exposing it read-only over NFS to the student VLAN is an open item.)

6.4 Verifying mgmt1

ssh ramge@172.20.0.11 → admin shell, can cd ~/sync/gh/NgarumaVTC/nixos, sudo via wheel
xfreerdp /v:172.20.0.11 /u:<teacher> → XFCE4 desktop, writes to /home/<teacher>
sssctl user-checks -a acct -s sshd student10013 → Permission denied
touch /media/ClassMaterial/test as a teacher succeeds; the file inherits lldap_teacher

Phase 7: Thin Client Preparation

7.1 Physical Hardening

For each HP ProDesk / Dell OptiPlex thin client:

[ ] Remove hard drive (no local state — boot is network-only)
[ ] Block unused USB ports with silicon dust caps
[ ] Install DisplayPort to HDMI cable (1m, no extensions)
[ ] Add strain relief (zip tie) to power and display cables
[ ] Label with assigned container number (TC-51 through TC-70)

7.2 BIOS Configuration

[ ] Set Network Boot (PXE) as the first and only boot device
[ ] Disable USB boot
[ ] Set a BIOS password (prevents students from changing boot order)

7.3 Boot Test

[ ] Connect client to student LAN port on the CRS326 switch
[ ] Power on → client should receive DHCP from MikroTik
[ ] iPXE loads kernel + microcode.img + initrd over HTTP → NixOS boots
[ ] XFCE4 login screen appears; sssd auth against LLDAP works
[ ] NFS-mounted home directory is writable and persistent across reboots

Phase 8: Operational Validation

[ ] All 20 thin clients boot and reach their containers simultaneously
[ ] ZFS snapshot schedule active (zfs list -t snapshot)
[ ] NUT shuts down cleanly on battery test (upsmon -c fsd to simulate)
[ ] WireGuard tunnel survives a cellular reconnect
[ ] IPMI accessible from Germany via VPN
[ ] Kiwix serving offline Wikipedia on 172.20.90.22
[ ] ClassMaterial folder: teachers can write, students read-only
[ ] DropBox folder: students can write, cannot read or delete
[ ] Bandwidth cap active: confirm a single client cannot saturate the WAN link

Uh oh!

FilesExpand file tree

implementation.org

Latest commit

History