ilifu Software Catalog

A web catalogue of the software available on the ilifu HPC cluster. Live at https://catalogue.ilifu.ac.za.

Software reaches ilifu users two ways — Lua modules (lmod) and Singularity/Apptainer containers — and this application makes both searchable: which packages exist, which versions are installed, which binaries each version puts on PATH, their man pages, and the R libraries and Python distributions installed alongside them.

That last one is why searching for ggplot2 or numpy works. Nothing on the cluster is named either — they live inside R/4.4.1 and python/3.13.2 — so they are first-class search results that tell you which module or container provides them.

Signed-in users get search, version detail, man pages and a software-request form. Anonymous visitors get a names-only public list. Administrators configure which filesystem roots are scanned.

Architecture

The central constraint: the application runs in Docker, but the data lives on cluster storage. A container cannot meaningfully run module show or apptainer exec against ilifu. So the system is split in two.

  ilifu cluster node                       Docker host
  ┌─────────────────────┐                  ┌─────────────────────────────────┐
  │  ilifu-collector    │                  │  Caddy — TLS, routing           │
  │   · module show     │                  │    ↓                            │
  │   · walk PATH,      │   HTTPS          │  Django + django-ninja          │
  │     MANPATH         │   chunked,  ───► │    ↓ enqueue                    │
  │   · read R/python   │   gzipped        │  Celery worker + beat           │
  │     library trees   │   snapshot       │    ↓                            │
  │   · apptainer exec  │   bearer token   │  Postgres — FTS + pg_trgm       │
  │   · read raw roff   │                  │  Redis — broker, cache, session │
  │   · watchdog        │                  └─────────────────────────────────┘
  └─────────────────────┘

The collector never touches the database. Django never touches cluster storage. The only coupling between them is the payload schema in contracts/, which both import — and a test asserts the collector's dependency closure contains nothing Django-shaped.

Some consequences worth knowing before reading the code:

• Ingest is a full snapshot per scan root, uploaded in chunks. The server diffs it against current state and writes ChangeEvent rows. This is idempotent, self-healing, and detects removals for free — none of which is true of delta-based ingest.
• The collector ships raw roff; the server renders it. So a sanitiser fix can re-render everything without a rescan, ?raw=1 costs nothing, and rendering stays testable in CI.
• Domains are not inferred. Package categories come from an editable keyword rule table plus an Uncategorised bucket for triage. Guessing them was considered and rejected.

Quick start

Requires Docker and uv. Nothing else — the stack is self-contained.

cp .env.example .env          # safe development defaults, no real secrets
docker compose up -d
curl localhost/healthz        # {"status": "ok"}

Six services come up: caddy, web, worker, beat, db, redis. Caddy serves plain HTTP on :80 in development; setting SITE_ADDRESS to a real hostname makes it provision TLS automatically, with no other change. Migrations run themselves on startup — the entrypoint waits for Postgres and applies them under an advisory lock, so web, worker and beat starting together cannot race.

First run

docker compose up -d gives you a working application and an empty catalogue. Four steps fill it. Nothing seeds or bootstraps them for you, and the token step has no management command, so the whole loop is written out here.

1. Sign in. Open http://localhost/ — anonymously that is the public list — and press Sign in. In development the OIDC provider is ilifu_catalog.devauth, a fake in-process issuer offering three fixture identities: a plain researcher, an admin (whose address is the development default of ILIFU_ADMIN_EMAILS), and a third carrying that same address unverified, which is refused admin — the self-promotion guard, reproducible by hand. Pick the admin. (devauth is mounted only when DEBUG is on, and settings.prod hardcodes DEBUG = False, so it cannot exist in production. Against real Keycloak this step is an ordinary redirect and the fixture picker never appears.)

2. Register a scan root at http://localhost/admin/scan — the path the collector will scan, and whether it holds lmod modules or .sif images. Nothing can be ingested for a root the server has not been told about.

3. Issue a collector token and scope it to that root. The plaintext is returned once and never stored; only its SHA-256 is persisted, so there is no way to recover it later.

docker compose exec -T web python manage.py shell -c \
  "from ilifu_catalog.software.models import CollectorToken, ScanRoot; \
   token, raw = CollectorToken.issue(name='login1'); \
   token.allowed_roots.set(ScanRoot.objects.filter(path='/software/modules')); \
   print(raw)"

The scoping is enforced on all three ingest endpoints, not just the first — a token cannot write into, or complete, a snapshot for a root it does not own, even if it learns that snapshot's id. A token scoped to nothing can ingest nothing.

4. Scan. On a cluster node that mounts the trees being scanned (usually a login node) — not in the container, which cannot see cluster storage. Module roots additionally need lmod on that node ($LMOD_CMD/$LMOD_DIR, or lmod on PATH): the collector reads modulefiles through module show rather than parsing Lua itself, and says so plainly if it cannot find it. Container roots need apptainer, and say so just as plainly — a missing binary is refused up front rather than scanned around, because every image failing its exec looks identical to an empty root.

A container scan can also stop on a single image, and this is deliberate. A snapshot is a full replacement, so an image uploaded with empty listings loses every binary, library and man page it had, and an image left out loses its version outright — both are deletions. An image the collector cannot list is therefore represented by the last scan of it that succeeded — a fallback that stays in the cache after it is used, so a second bad night falls back again rather than escalating. A scanned image also contributes its own definition file (apptainer inspect --deffile), shown on the catalogue's provenance tab as the build recipe; an image built without one says so there explicitly, rather than showing nothing. Only with no such scan at all does the run refuse: it uploads nothing, names the image, and leaves the catalogue untouched. Delete the image if it is gone, or rebuild it if it is broken. Two cases never refuse: a pre-SIF legacy image, skipped at discovery, and an image that has actually gone from disk — including one a deploy removes or replaces mid-scan. Both are simply left out, and the catalogue retires them.

uv tool install ./collector
export ILIFU_COLLECTOR_TOKEN=<the token from step 3>
ilifu-collector scan --root /software/modules --kind module \
  --api-base-url https://catalogue.ilifu.ac.za
ilifu-collector watch --root /software/modules --kind module \
  --api-base-url https://catalogue.ilifu.ac.za    # debounced rescan on change

--api-base-url is the site root, not the API prefix — the client appends /api/ingest/... itself. Passing .../api produces 404s on every request.

The collector uploads a full snapshot in batches and prints snapshot <id> accepted. Accepted is all /complete can honestly say: the server stages the snapshot and enqueues the diff, which the worker container applies off the request cycle — a ~1400-package root must never be diffed on a request. So scan returning does not mean the catalogue has changed yet. Watch the worker, or reload the screen a moment later.

To try the pipeline without a cluster, point --root at a directory laid out the way the collector expects: <root>/<name>/<version>.lua for modules, <root>/<name>/<version>.sif for containers — though container discovery is recursive, so a .sif at any depth is found, named after its parent directory (version = filename stem). A module root still needs lmod installed, since lmod is what reads it — whatever the modulefiles say, module show is the thing that resolves them.

Restart the Celery containers after editing anything they run. All three app services bind-mount your working tree, so no rebuild is needed — but Celery has no autoreloader, so docker compose restart worker beat is what makes an edit to tasks, ingest or models take effect. The dev server reloads itself.

Development

uv sync --all-packages              # install every workspace member plus dev tools
uv run pre-commit install           # ruff, mypy, djlint, hygiene, uv lock check
uv run pytest                       # run the suite
uv run pytest -k <pattern>          # a single test or subset
uv run coverage run -m pytest && uv run coverage report

uv, never pip. Note the --all-packages: the workspace root is a virtual project with no dependencies of its own, so a bare uv sync installs nothing at all.

Tests come first. Write the failing test, then the implementation. Test names read test_should_<behavior>_when_<condition>.

Coverage is gated twice in CI. The project as a whole must stay above 95%. A second, narrower gate requires 100% on the modules where a bug is silent rather than loud — the ingest diff engine, the man renderer, search, the category mapper, the OIDC claim mapping and auth backends, the request-body middleware, the collector's parsers, shared directory walker and scan cache, the collector's transport (the code that carries the bearer token), and the async tasks that apply an ingest or notify support (ingest.py, manpages.py, search.py, categories.py, claims.py, backends.py, permissions.py, middleware.py, modules.py, containers.py, lmod.py, libraries.py, walking.py, snapshot_cache.py, client.py, cli.py, tasks.py). Reaching that number with assertion-free tests defeats the point; the gate exists to make those modules trustworthy, not to produce a badge.

The suite needs a real Postgres — SQLite cannot run the FTS or trigram code — and settings/test.py deliberately hardcodes localhost:5432 so an exported DATABASE_URL cannot change what mypy and the tests see. On a machine whose own Postgres already owns that port, run the suite in containers instead; CLAUDE.md has the exact incantation. Install mandoc too: without it the man renderer's one real-subprocess test skips, and the renderer is then only ever tested against a fake.

Repository layout

A uv workspace with three members. The split is load-bearing, not cosmetic — it is what keeps Django off the cluster node the collector runs on.

contracts/    shared Pydantic snapshot schema — the only code both sides import
collector/    cluster-side scanner. httpx, typer, watchdog. Never Django.
catalog/      the Django application
docker/       image definition and entrypoint
design_handoff_ilifu_software_catalog/
              read-only design reference — see below

Install the collector on a cluster node with uv tool install ./collector; it pulls in almost nothing.

Inside catalog/src/ilifu_catalog/, four Django apps and the modules worth knowing by name:

software/     the domain. models, the ingest API (api.py) and its staging
              (snapshots.py), the diff engine (ingest.py), search, category
              rules, the man renderer, Celery tasks, and one view module per
              screen (views_catalog / views_public / views_admin / views_manpage)
accounts/     OIDC backend and claim mapping (claims.py — the admin *decision*;
              the two gates that enforce it live in permissions.py),
              OIDCIdentity (the subject a login is matched on), UserProfile
              (theme), and the revoke_software_admins command
requests_app/ the software request form and its licence gating
devauth/      fake in-process OIDC issuer. DEBUG-only, twice guarded, never prod
settings/     base.py holds the typed Settings model; dev/prod/test build on it
templates/    server-rendered screens; partials/ are the htmx fragment targets
static/       CSS (tokens.css is the theme table), vendored htmx + Alpine,
              self-hosted fonts — nothing loads from an external origin

Configuration

One set of variable names, read identically by Compose, Django settings and CI. They are a typed pydantic-settings model (settings/base.py::Settings), not scattered os.environ reads, so a malformed value fails at startup rather than at first use. See .env.example for the annotated list.

Every variable has a development default, which is why the stack runs on a fresh clone. Production removes the defaults that must not be shared — ProductionSettings makes the secret key, the allowed hosts, all six OIDC fields, the admin allow-list and the mail relay pair (EMAIL_HOST, DEFAULT_FROM_EMAIL) required, so a deployment that omits one fails to boot rather than quietly running on a placeholder.

Variable	Purpose	Required in prod
DJANGO_SETTINGS_MODULE	…settings.dev / .prod / .test	pinned by Compose
DJANGO_SECRET_KEY	Django secret; generate a fresh one per deployment	yes
DJANGO_DEBUG	1 or 0 — ignored by settings.prod, which hardcodes False	no
DJANGO_ALLOWED_HOSTS	comma-separated hostnames	yes
POSTGRES_USER / _PASSWORD / _DB	what the db container initialises Postgres with, on first boot of an empty volume only	yes
DATABASE_URL	Postgres connection string; nothing checks it agrees with the three above	yes
DB_CONN_MAX_AGE	seconds each thread holds its Postgres connection between requests; 0 reverts to a fresh connection per request — the no-deploy escape hatch	no
DB_CONN_HEALTH_CHECKS	ping a held connection at request start, replacing it if broken; meaningless when DB_CONN_MAX_AGE=0	no
REDIS_URL	Django cache and sessions	yes
CELERY_BROKER_URL	Celery broker — its own logical Redis database	yes
INGEST_REDIS_URL	staged ingest snapshots — a third logical database, so a cache flush cannot discard an upload in flight	yes
REDIS_MAXMEMORY	ceiling for the Redis container; sized to hold a full staged snapshot alongside the cache	no
DATA_UPLOAD_MAX_MEMORY_SIZE	largest request body Django will read. Django's 2.5 MB default is far below one ingest batch — see .env.example	no
SITE_ADDRESS	Caddy site address; :80 in dev, real hostname in prod	yes
SUPPORT_EMAIL	shown to users, and the sole recipient of the software-request notification email	no
DOCS_URL	target of every screen's header docs link; the anonymous page's cached HTML picks up a change at the nightly regeneration	no
EMAIL_HOST	SMTP relay for the request notification	yes
EMAIL_PORT / EMAIL_USE_TLS / EMAIL_USE_SSL	relay port and encryption mode; defaults are 587 + STARTTLS	no
EMAIL_HOST_USER / _PASSWORD	relay credentials; empty for a relay in IP-allowlist mode	no
DEFAULT_FROM_EMAIL	the notification's From address — relays reject Django's webmaster@localhost fallback	yes
EMAIL_TIMEOUT	seconds before a hung SMTP send is abandoned; Django's own backend would wait forever	no
OIDC_RP_CLIENT_ID / _SECRET	Keycloak client credentials	yes
OIDC_OP_AUTHORIZATION_ENDPOINT	front-channel: the browser is redirected here	yes
OIDC_OP_TOKEN_ENDPOINT	back-channel: fetched by the app server	yes
OIDC_OP_USER_ENDPOINT	back-channel	yes
OIDC_OP_JWKS_ENDPOINT	back-channel	yes
ILIFU_ADMIN_EMAILS	comma-separated addresses that get the admin screen. The whole authorisation configuration — nothing in Keycloak grants it	yes
IMAGE_REGISTRY / IMAGE_TAG	deploy host only — which image compose.prod.yaml pulls. CI exports both	no

The front-channel/back-channel split matters only against the development issuer, where the two sides see different origins (the browser reaches Caddy on :80; the app server reaches itself on localhost:8000). Against real Keycloak all four URLs are externally reachable and the distinction disappears.

Who can sign in, and who is an admin

Anyone the realm authenticates can sign in, and gets a Django user on first login. No group membership is required or checked.

Admins are configured in this application, not in Keycloak. The ilifu realm sends no groups claim and no realm_access claim, so there is nothing there to read; ILIFU_ADMIN_EMAILS lists the addresses instead. At each login the app compares the email claim against that list and writes the result into the ilifu-software-admin Django group, which is what the two gates in accounts/permissions.py check.

The app stores no email address. The claim decides the group and is then dropped — nothing on the user (stock auth.User keeps its email column; nothing writes it), no hash, no log line. That is why the admin screen names the environment variable rather than listing its contents, why support's request notification tells them to look the requester up in Keycloak, and why revoking an admin is a procedure rather than an edit:

# 1. remove the address from ILIFU_ADMIN_EMAILS, then redeploy, then:
docker compose exec -T web python manage.py revoke_software_admins

Membership is only re-derived at login, so without that command a removed admin keeps /admin/scan until their session expires — two weeks by default. The command empties the group, deletes every session and clears the session cache (all three are needed: cached_db sessions read the cache first, so deleting the rows alone signs nobody out).

Two rules the allow-list follows, both in accounts/claims.py: an address grants admin only if the claim arrives with email_verified: true — otherwise a realm permitting self-service email changes would be a self-promotion route — and matching is exact after case- and whitespace-normalisation, so @ilifu.ac.za grants nobody anything.

compose.prod.yaml pins settings.prod per service, so a stale .env cannot boot the production stack in debug mode.

Deployment

GitHub Actions runs lint, tests and both coverage gates on every pull request. On merge to main it additionally builds the image and pushes it to GHCR tagged with the commit SHA — so the artifact that was tested is the artifact that ships.

Deployment is owned by a separate Ansible repository, which pulls that tag. This repo publishes an image and describes what running it needs; it does not reach a host. See ANSIBLE_DEPLOY.md for the whole contract — image coordinates, the service topology, every environment variable, and what the playbook must not do.

Every action is pinned to a full vX.Y.Z tag rather than a moving major — see the note at the top of .github/workflows/ci.yml for why.

Open

Known gaps, ranked by what to fix first. Each was found by review and deliberately left rather than overlooked.

1. The public list renders duplicate chips for same-named packages. Package.name is not unique (only slug is), so blast as a container and blast as a module are two indistinguishable chips. The catalog screen disambiguates with its mod/sif tag; the public page deliberately has none. Needs a product decision first — on a names-only page, is that one thing the cluster has, or two?
2. rescan all records an intention, not an action. The collector runs on a cluster node and pushes; there is no server-to-collector channel. The screen says so plainly. Closing it needs a poll endpoint or a queue the collector subscribes to.
3. Four admin numbers are unmeasurable under the current snapshot contract: rescan duration, failed jobs, job count, and live watcher liveness. Each is shown as not measured or replaced with something real rather than estimated. Closing them means the collector reporting durations and outcomes, so both sides of contracts/ move.
4. A library never counts as "already installed" on the request form. check_name_exists matches packages and binaries only, so requesting pandas is not told it is vendored inside one Python install. That is sometimes exactly the useful answer and sometimes a way to suppress a legitimate request for a standalone module; it needs an ops decision, not a code change.
5. Unreadable directories are skipped without a count. A directory the collector cannot read costs only its own subtree — walking.py is shared by libraries.py, modules.py and containers.py, so all three log and continue — but the admin screen has nowhere to say "47 directories were unreadable", so a permissions problem narrows the catalogue more quietly than it should. (This item used to list three robustness # TODO:s as well — no apptainer exec timeout, unserialised applies for one root, a digest-skip that never engaged. All three are closed: each exec is bounded by --exec-timeout (60s default), applies serialise on the ScanRoot row lock so a /complete retry's duplicate apply blocks and then no-ops, and the collector caches its last uploaded scan per root to feed the digest-skip — clean payloads only, so an image that scanned degraded is re-probed on the next scan rather than replayed from the cache forever.)
6. A staged snapshot can still be lost if the worker is down for over an hour. Staging lives in Redis with a TTL (COMPLETED_SNAPSHOT_RETENTION_SECONDS), so a snapshot whose apply never gets picked up inside that window is gone, and the collector has already been told 202. tasks.py logs it at ERROR so it is visible rather than silent, and the next scan re-uploads — but closing it properly means staging somewhere durable.
7. A permanently failed support notification has no resend control. The admin queue's ! marker says the notification never went out, but the only way to act on it is a shell (notify_support_of_request.delay(pk)); a resend control on the triage form would close the loop the marker opens. Deliberately deferred: the marker itself was the fix for a silent failure, and a control that re-emails support belongs with a broader decision about what else triage should be able to trigger.
8. docker compose up needs a .env despite the settings defaults. settings/base.py defaults every variable so the stack runs on a fresh clone, but docker/entrypoint.sh reads os.environ['DATABASE_URL'] directly, with no default, so without a .env the web container loops on postgres not ready yet ('DATABASE_URL') until it gives up. cp .env.example .env is the documented first step anyway; the two sources of defaults should still agree.

Design reference

design_handoff_ilifu_software_catalog/ holds the design specification and two HTML prototypes. It is read-only — never edit or reformat it.

• ilifu Software Catalog.dc.html — the hi-fi design. Build from this.
• ilifu Software Catalog Wireframes.dc.html — earlier exploration. Useful for rationale and rejected alternatives; do not build from it.
• README.md — the full specification: every screen, its exact spacing and type, the interaction rules, and the design tokens.

Open either file directly in a browser. CLAUDE.md summarises the decisions that are easy to get wrong when working from them.

Deviations from the specification, flagged for sign-off

Three, each marked with a NOTE: at the code that implements it:

1. Five tabs where the spec gives four (views_catalog.TABS). libraries was added because a library search hit needs somewhere addressable to land.
2. The detail pane's categorise as control (views_catalog.catalog_categorise). The spec gives the pane's first row as the breadcrumb alone and puts triage on the admin screen's CATEGORY MAPPING card. Admin-only, and it does not replace that card — it writes a rule into the same table, one narrowly anchored to the package being categorised, plus the package's own domain so the change shows immediately. The reasoning is that the admin who notices an Uncategorised package is the one browsing the catalogue, and sending them to another screen to retype its name is where triage stops happening.
3. The admin screen's title-row badge reads admin allow-list: ILIFU_ADMIN_EMAILS (views_admin.ADMIN_ALLOW_LIST_VAR) where the spec pins keycloak group: <name>. Admin no longer comes from a Keycloak group — the realm sends no group claim at all — so the spec's copy would be false on the one screen whose readers act on it, and would send an operator asking about their own access to the wrong system. The badge names the variable rather than its contents, because listing the addresses would put the one thing this application deliberately never stores onto a rendered page.