v0.7.2 feat(jobworkerdb): sanitize user provided job data PostgreSQL cannot store

[ungerik]

Jobs carrying user provided data that PostgreSQL cannot store are now persisted with the unstorable characters removed, instead of failing the write and losing the job.

The problem

PostgreSQL stores neither zero bytes nor invalid UTF-8 in a text column. jsonb additionally rejects two escape sequences that are valid JSON:

• the zero-byte escape, which is what encoding/json emits for a zero byte inside a Go string — so a marshalled payload carries no raw zero byte at all, and byte-level stripping is a no-op on it
• unpaired UTF-16 surrogates, which a browser's JSON.stringify emits for a truncated emoji

Both are accepted by json.Valid, so NewJob accepts such a payload and nothing upstream refuses it. Only the INSERT fails, and the job is lost. Verified against PostgreSQL 16 — the zero-byte escape gives unsupported Unicode escape sequence, and a lone high surrogate gives Unicode low surrogate must follow a high surrogate.

What changed

Sanitizing is applied where values are written: job origin and payload, error message, error data and result, and the bundle origin. The in-memory Job is never modified, so a value read back may differ from the one passed in.

Not sanitized: Job.Type and JobBundle.Type. Job.Type is the dispatch key a worker registers for — rewriting it would store a job no worker could ever claim, which is worse than failing the write.

Fails loudly rather than being silently repaired:

• an origin with nothing storable left, which violates the column's non-empty constraint
• a payload with nothing storable left, because notnull.JSON binds nil as the empty object, so it would otherwise be queued and dispatched to a worker as an all-zero payload struct
• an error message that sanitizes to empty is replaced by a placeholder, because an empty error_msg reads back as SQL NULL through nullable.NonEmptyString and would make a failed job report Succeeded() == true

Lookups are never normalized. DeleteJobsFromOrigin and DeleteJobBundlesFromOrigin match their argument verbatim. Sanitizing is not injective, so normalizing a destructive unbounded key could collapse one origin onto another and delete jobs the caller never named, after any authorization performed on the raw string. Deleting too little is recoverable; deleting someone else's jobs is not.

Observability: a WARN names the id, field and byte count whenever sanitizing actually removes something, so a stored value differing from what a producer sent is traceable instead of invisible.

Behaviour change

This moves from fail-closed to silently transformed. Sanitizing deletes rather than replaces, so it joins whatever surrounded a removed character: "a" + zero byte + "b" is stored as "ab".

Validate values after sanitizing, not before — a check that must hold for what a worker acts on belongs in the worker, against what it read back. Sanitizing is also not injective, so two JSON object keys differing only in unstorable characters collapse into a duplicate key and jsonb keeps the last. The jobworkerdb package docs carry the full list of consequences, including that ContextWithSynchronousJobs does not sanitize at all.

Performance

Sanitizing is on the hot path of every job write, and payloads can be large.

1MB payload	ns/op	allocs
already storable (the common case)	~30,000	0
one escape removed	~94,000	1

The clean path scans in place and never copies.

An earlier revision stripped repeatedly until stable. Crafted malformed input can chain removal seams so each round exposes exactly one more, which made that quadratic: 108ms for a 30KB payload, extrapolating to minutes for 1MB, with a transaction held open when the write came through AddJobBundle. Now bounded at two extra rounds: 151us for the same input. Input still unstorable after the cap is malformed JSON that PostgreSQL rejects anyway, so it fails the write instead of burning CPU. TestSanitizeJSONBoundedOnChainedRemovals asserts the work stays linear.

Tests

sanitize.go is at 100% statement coverage. TestPostgresRejectsUnstorableStrings is a negative control that pins what PostgreSQL actually refuses, so the sanitizing tests cannot pass vacuously — including the assertion that Go's json.Valid accepts inputs jsonb rejects.

Covered: both escape classes, valid surrogate pairs kept intact, uppercase hex spelling, escaped-backslash parity, truncated escapes, escapes formed by byte removal, the allocation-free contract (pointer identity), per-type NULL binding (notnull vs nullable), the WARN log's field/id/byte-count labels, and end-to-end DB assertions for each write path including a bundled job's num_jobs_stopped counter.

Review

Reviewed by /review (5 specialists + red team), /simplify (4 agents), and adversarial passes from both Claude and Codex. Findings fixed along the way included:

• a state confusion where a failed job could report Succeeded() == true while SQL saw it as errored, and DeleteFinishedJobs never reaped it
• a full payload copy on every write
• a zero-length jsonb bind that would roll back a bundle's counter increment and wedge the bundle forever
• the quadratic stripping loop above

Releases v0.7.2.