[sjdonado]

What changed

The scheduled Update Parsers workflow has failed on master since October 1. crystal-lang/install-crystal@v1 (v1.9.2) installs libevent-dev without refreshing apt, and the runner image now ships a stale index, so the download returns 404 (crystal-lang/install-crystal#82, fixed upstream but not released). Both workflows now run sudo apt-get update first. The schedule 0 0 1-7,15-21 * 0 also ran every day, because cron matches either the day of month or the weekday, so it is now 0 0 1,15 * *.

The Docker build now uses the official multi-arch crystallang/crystal:1.21.1-alpine image. The previous Alpine 3.23 package was Crystal 1.18.2, so production missed the HTTP::Server request smuggling and request body fixes released in 1.19.2 through 1.21.1. The runtime stage moves to alpine:3.22 to match the libraries the build image links against, and libevent is gone because Crystal uses its own epoll event loop on Linux. shard.yml now requires Crystal 1.21.1.

Kemal moves from 1.12.0 to 1.14.0 for its security fixes, including HEAD requests bypassing GET filters and route keys built from malformed methods. Kemal 1.14 discards any body written before an error, which emptied every API error response. The app exceptions now carry their JSON body, and error handlers in routes.cr print it for 400, 401, 403, 404, 405, 413, 422, and 500.

Micrate pinned db ~> 0.11 and sqlite3 ~> 0.19 and has had no release since 2024. It is replaced by app/lib/migrator.cr, which applies the Up section of pending migrations at boot and rolls back the newest applied one with cli --migrate-down, one BEGIN IMMEDIATE transaction per file. The migrator reads and writes micrate's existing micrate_db_version table, so deployed databases keep their history. This unlocks sqlite3 0.23.0, which keeps bound arguments alive until a statement runs, and db 0.14.0, which fixes multi-threaded pool races. The unused mysql and pg shards disappear with micrate. The benchmark runs migrations in-process instead of requiring the micrate binary.

A new AGENTS.md gives a code map, the constraints that are easy to miss, the check order, and manual verification steps beyond the e2e suite: running the Docker image, a bash walkthrough of every endpoint and its edge cases, and load, throughput, and memory measurement.

Behavior changes

• A request that matches no route returns 404 {"error":"Resource not found"} as JSON. Before, Kemal rendered its own 404 page.
• A request to a known path with the wrong method returns 405 {"error":"Method Not Allowed"} with an Allow header. Before, it returned 404.
• A failing migration now stops the app at boot. Micrate logged the error and the app started anyway.
• SQLite now runs with the settings the database URL always asked for: journal_mode=WAL, synchronous=NORMAL, and foreign_keys=true, plus a new busy_timeout=100. The app, the migrator, and the CLI share them through App::Lib::Database::URL; a key already present in DATABASE_URL wins. crystal-sqlite3 0.19 ignored URL parameters, so master never applied them. On first boot an existing database switches to WAL, and data.db-wal and data.db-shm files appear next to it in the sqlite/ volume.
• Foreign keys are enforced for the first time, so the existing ON DELETE CASCADE rules take effect: deleting a user deletes their links and clicks, and a click recorded after its link was deleted is rejected and logged instead of stored as an orphan.
• Each migration runs with foreign keys off, because with them on, 20240714215409 drops links and the cascade deleted every click of a database upgraded from July 2024 (reproduced: 3 clicks became 0). A migration fails only if it adds a foreign key violation, so orphan rows from the unenforced years do not block boot. One limit predates this PR: a database older than 20250319192003 that holds orphan clicks or links cannot run that migration (NOT NULL constraint failed: clicks_new.link_id).
• cli --migrate-down rolls back the newest applied migration with its Down section and refuses an empty one, so 20250319192003 is irreversible. The current image re-applies a rolled-back file on its next start, so a rollback only sticks when an older image is deployed next.
• A write from another process, such as docker exec ... cli, now waits up to 100 ms for the lock instead of failing with database is locked. The timeout stays short because SQLite's busy wait blocks the single-threaded scheduler, so a long one would stall every request.

Benchmark

Tested on the same laptop in the same session, with 100,000 requests and 125 connections against the redirect endpoint. Both builds recorded every click.

Build	Run	Req/s	Avg latency	p99
master	1	3,435	36.55 ms	82.78 ms
master	2	3,607	34.77 ms	85.84 ms
this branch	1	11,240	11.15 ms	44.44 ms
this branch	2	21,122	5.90 ms	9.83 ms
this branch	3	21,059	5.93 ms	12.23 ms
this branch	4	21,379	5.84 ms	10.89 ms
this branch	5	21,141	5.91 ms	12.04 ms

Apart from the first run after a build, the branch holds at about 21k req/s, roughly 6x master.

The gain comes from SQLite's journal mode, not from dependency code. crystal-sqlite3 0.19 opened the database without reading the URL's query parameters (connection.cr has a TODO for it), while 0.20 and later run them as PRAGMAs on every connection (connection.cr). On master every click insert therefore committed in rollback-journal mode with synchronous=FULL, and those blocking fsyncs stalled the single-threaded server. Two experiments confirm it:

• The unchanged master binary on a database already in WAL mode: 21,894, 21,865, and 21,675 req/s. The same binary on a database in its default mode: 3,725, 3,950, and 3,816 req/s.
• Branch builds with kemal 1.12, or with sqlite3/db pairs from 0.19/0.11 through 0.22/0.14, all landed between 19.7k and 21.5k req/s on a WAL database, the same as the branch.

The README and docs/SETUP.md now show the median run (21,141 req/s, 5.91 ms average latency, 32.14 MiB average memory), and the SETUP run step builds with ENV=production, which is how the numbers were measured.

Checks

• ENV=production shards build --release --no-debug for bit, cli, and benchmark, with no warnings
• ENV=test crystal spec with 61 passing examples, including migrator specs for micrate history, transaction rollback on failure, a July 2024 database upgraded through every real migration with foreign keys on, tolerated orphans and rejected new violations, rollback and the irreversible refusal, and JSON 400, 404, 405, and 413 responses. The July 2024, 413, and foreign key restore specs fail when their fix is removed
• ENV=test crystal spec e2e
• docker build for linux/arm64 and linux/amd64, each smoke tested by creating a link
• The migrator against copies of two local databases micrate had migrated: an up-to-date one was left unchanged, and one a version behind received only 20260808000000_add_pagination_indexes.sql, then rolled back and re-applied it
• Every command in AGENTS.md run against the built image: 32 endpoint status checks matched, GeoIP resolved 8.8.8.8 to US inside the image, and the container used 30 MiB idle and 51 MiB under 100,000 requests

Assumptions

• The migrator applies any pending version in order, even one older than the newest applied version. Micrate refused that case. This would flip if migrations are ever merged out of order on purpose.
• There is no status or scaffold command, and no StatementBegin support. No migration uses them, and the parser raises on any other -- +micrate directive so a future need fails loudly.
• busy_timeout is 100 ms rather than seconds. This would flip if the app runs with a multi-threaded execution context, where a blocked thread no longer stalls every request.
• The update schedule moved from the 1st and 3rd Sunday to the 1st and 15th, since GitHub cron cannot express "nth weekday".
• The GitHub Actions still pinned to Node 20 majors (checkout@v4, docker/*, create-pull-request@v6) keep working today and are left for a separate change.

Refuted evidence

• The crystallang/crystal:1.21.1-alpine image was reported to be based on Alpine 3.24. It is based on Alpine 3.22.6 for both architectures, so the runtime stage uses alpine:3.22 instead of 3.24.

Rejected review findings

• Use the {"errors": ...} shape for the generic 422 fallback: Kemal never raises 422 itself, and the handler exists to print the app's own 422 body.
• Narrow the generic handler list to the statuses Kemal raises: the extra branches are harmless and keep one list for every status the API answers.
• Drop the AGENTS.md endpoint lines that the in-process specs already cover: the walkthrough exists to check every endpoint against the published image over a real network, which the specs do not do.
• Swallow errors from the PRAGMA restore after a failed migration: only callers that pass their own DB::Database (the specs) keep the connection, and hiding the error would leave foreign keys off silently.
• Limit foreign_key_check to files that touch tables with foreign keys: it runs only at boot for pending files, and a full check is the simpler guarantee.
• Skip migrations for cli --update-parsers: the extra migration is harmless and keeps one rule for every CLI command except --migrate-down.
• Load dotenv before the DATABASE_URL default in app/config/env.cr: the ordering predates this PR and production does not use dotenv.