Repository navigation
Local-development distribution for PGX (CLI, installer, Docker, tests) - #1
Merged
Merged
Conversation
…so.16, report the probe
PGRun beta finding: the row-count idiom xpath(..., query_to_xml(...)) failed
on a PGX host with "could not dlopen libxml2.so.2" while every other check
matched PostgreSQL, and the compatibility gate had no way to see it.
Root cause: libxml2 2.14 renamed its SONAME from libxml2.so.2 to
libxml2.so.16. Ubuntu 25.10+ (26.04 LTS) ship only .so.16, and the bare
libxml2.so symlink belongs to the -dev package a runtime host does not
install. The loader probed .so.2 / .so / .2.dylib only, so a host that had
libxml2 still failed every xpath, xmltable, xml_is_well_formed and xml
input call. The failure surfaced as a bare backend panic reported as an
internal error.
- adt_xml::libxml: probe libxml2.so.16 before .so.2 and .so (macOS: the
system libxml2.2.dylib, then libxml2.16.dylib and the Homebrew keg);
record which name loaded and libxml2's xmlParserVersion.
- A host with no loadable libxml2 now raises C's NO_XML_SUPPORT shape:
SQLSTATE 0A000 "unsupported XML feature" with the same DETAIL, plus a
HINT carrying the probe failure and the package to install. The session
survives; non-XML statements are unaffected.
- pgrust_runtime_status() gains "libraries": libxml2 {available, library,
version, error} and icu {available, unicode_version}, so pgx-agent's
Status() can exclude a misconfigured host before it serves a branch.
- docs: compatibility-gate.md records the finding as host rule
host_library_missing (a host rule, not a source rule); pgrun-interface.md
documents the field; README names the run-time libraries and packages.
Verified on Ubuntu 26.04 arm64 (libxml2 2.15.2, libxml2.so.16 only, no
.so.2): xpath/xmltable/xml input/query_to_xml all work and the status
reports library "libxml2.so.16", version "21502". With the library hidden
the same statements return 0A000 with the hint, the session continues, and
the status reports available=false with the probe message. macOS loads the
system libxml2.2.dylib as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
dist/assemble.sh builds pgx-<version>-<os>-<arch>/ (bin/pgx, bin/postgres,
share/{timezone,timezonesets,tsearch_data,extension}, share/pgx/base-cluster
from a stock PostgreSQL 18 initdb, dist.json, licenses) plus a tarball and
its sha256. dist/test/assemble_test.sh checks the layout in dist/layout.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
New workspace crate crates/bin/pgx: argument parsing with the stable exit codes (0 ok, 1 failure, 2 usage, 3 not ready, 4 conflict, 5 not found, 6 in use, 7 corrupt install), installation layout discovery relative to the binary, $PGX_HOME resolution, and config.json (0600) round-tripping. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Startup over the Unix socket with trust auth, simple queries with text results, ErrorResponse decoding, identifier and literal quoting. Tested against a fake duplex stream; the live path is covered by the black-box suites. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
pgx init copies the bundled base cluster into $PGX_HOME/data (0700), writes postgresql.conf / pgx.conf / pg_hba.conf (socket trust, loopback scram-sha-256), picks a free port, probes block cloning to choose file_copy_method, starts the engine socket-only to set a random superuser password, stops it, and records config.json (0600). Refuses long socket paths (PGX_RUN_DIR escape hatch), busy ports, and a second init without --force. pgx version reports the distribution's dist.json. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
start spawns the engine detached in its own session with the stack limit raised, regenerates pgx.conf from config.json, clears stale postmaster.pid files (pid dead, or alive but not our engine), refuses a busy port with exit 4, and serialises concurrent starts with a lock file so exactly one engine runs. stop is a fast shutdown (SIGINT) with --force escalating to SIGKILL; status prints the state in text or JSON and exits 3 when stopped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Database names are validated (^[A-Za-z_][A-Za-z0-9_-]{0,62}$, system
databases refused as targets) and always quoted. drop maps 55006 to exit 6
with a --force hint and WITH (FORCE) when asked; clone is CREATE DATABASE
TEMPLATE with timing and the copy method; ls and url have JSON forms and
the URL is verified to work with psql over TCP (scram) and the socket.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Rust black-box tests for pgx clone (isolation both ways, in-use source is exit 6, missing source exit 5, existing target exit 4, failed clones leave no base/<oid> directory, 8 concurrent clones, 20 clone/drop cycles leak no files). tests/pgx/lib: a standard-library wire client (trust, md5, SCRAM, extended protocol, COPY) and a harness around the pgx CLI. tests/pgx/clone_consistency.py clones a checksummed source ten times under a concurrent writer and verifies each clone is identical. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Checks the installation layout, dist.json, that the engine runs, the data directory and config.json (mode), socket path length, runtime state and stale pid files, port ownership, free disk, the hard stack limit, the filesystem and block cloning, libxml2/ICU when running, and the log for FATAL/PANIC. Exit 1 when any check fails; runs before init too. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
tests/pgx/test_sql.py runs 19 content-checked cases against a pgx-managed runtime over real connections: DDL/DML, joins and windows, constraint SQLSTATEs, index use, JSONB, arrays, sequences/identity, views, plpgsql, prepared statements over the extended protocol, COPY in/out, error recovery, isolation levels, row locks, ten concurrent writers, scram over TCP and both URL forms. tests/pgx/run.sh runs the Python suites and keeps logs under tests/results/<timestamp>/. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
tests/compat: PGRun's behavioural compatibility suite (minitest + pg), run.sh starts a throw-away PGX and a PostgreSQL 18 control and compares rows and SQLSTATEs step by step (22 runs, 53 assertions, 0 failures). Regression baseline on 1fa2878 with the PGX profile: 223/231 byte-exact, 6 plan-only, 2 profile-induced (synchronous_commit off, no parallel workers); artifacts under docs/pgx/local-dev/validation/. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…han and invalid-db handling Found and fixed by adversarial testing: - pgx clone now issues CREATE DATABASE ... STRATEGY = FILE_COPY so the engine's file_copy_method applies: a 281 MB clone consumes ~0 MB (copy-on-write via clonefile/reflink) at ~42 ms, instead of a full 280 MB WAL_LOG copy. - pgx start retries its port probe for up to 5 s so a socket lingering from a just-killed engine is not mistaken for a real conflict (fixes intermittent exit 4 after SIGKILL); the engine binds with SO_REUSEADDR and never hit it. - databases left invalid by an interrupted DROP (datconnlimit = -2) are flagged by pgx ls and are droppable; pgx doctor gains read-only orphan base/<oid> detection (conservative: runtime up, numeric names, >2 min old) and never deletes anything. Suites (macOS ARM64, against the assembled distribution): - test_crash.py: SIGTERM/INT/KILL at rest and during insert/commit/create/ clone/drop/shutdown, storage exhaustion, permission and bad-config starts, stale pid/socket. 5 runs ephemeral + 1 full, randomized order. full keeps every acknowledged commit (0 lost of 21436); ephemeral loses only a clean prefix tail under synchronous_commit=off (documented), never a partial batch. Disk-full raises 53100 then PANICs the runtime (one crash domain); recovers on restart with data intact. - test_isolation.py: RC/RR/serializable, write skew (40001), deadlock (40P01), lock/statement/idle timeouts, advisory locks, drop --force (57P01), concurrent create/drop of one name. 12 tests. - test_storage.py: 150+ clone/drop cycles leak nothing, clone disk cost matches the backend, clone isolation, permissions, many-database restart, low-disk PANIC+recovery. storage_zfs.sh covers ZFS on a host that has it (COVERAGE GAP otherwise; never mocked). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_concurrency.py: 10 and 100 concurrent clients, 16 parallel creates, 20 concurrent CLI reads, connect/disconnect churn; every invariant checked after the storm. stress.py: the acceptance workloads (1000 create/drop, 1000 clone/drop, N clients for a fixed duration, start/stop cycles, optional soak) while sampling the engine's RSS, file-descriptor and thread counts, and fails on fd/thread growth that signals a leak. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Each fixture drives a real client against a database on the assembled PGX distribution and checks results, not just exit codes: psql (DDL/DML/\copy/ rollback/JSONB), node-postgres (pool, prepared, 23505, JSONB), psycopg 3 (executemany, COPY, server-side cursor, 42P07), jackc/pgx v5 (CopyFrom, batch, rollback, unique violation), ActiveRecord 8.1 (migrations up/down, seed, savepoint rollback, RecordNotUnique, InvalidForeignKey, JSONB, 8 parallel threads, schema dump, cleanup). run_all.sh creates a database per app with pgx, runs it, drops it; missing drivers are reported as coverage gaps, never silent passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- dist/install.sh: one-line installer. Detects OS/arch, downloads the release tarball and .sha256, verifies the checksum before installing, installs atomically into PGX_INSTALL_DIR, links pgx into PGX_BIN_DIR, verifies the binary runs, no root. dist/test/install_test.sh proves download, verify, relocate and refusal of a tampered tarball end-to-end over local HTTP. - pgx serve: run the engine in the foreground (init if needed), the Docker and systemd entrypoint. init now reads PGX_PASSWORD, PGX_PORT, PGX_DURABILITY and PGX_LISTEN (widening pg_hba with scram for non-loopback addresses). - Dockerfile (multi-stage) + entrypoint: ubuntu:24.04 runtime with libxml2 and ICU, PGX_HOME on a volume; dist/test/docker_test.sh builds, serves over TCP and checks persistence across a restart (COVERAGE GAP when no daemon). - .github/workflows: ci.yml (fmt/clippy, unit + black-box, installer, Docker), release.yml (linux-amd64/linux-arm64/darwin-arm64 artifacts + checksums as a DRAFT, lifecycle suite against the packaged artifact), nightly.yml (crash, isolation, concurrency, storage, stress, compat, regression, app fixtures). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…veness) Found by running the assembled distribution on real Linux arm64 (Ubuntu 26.04): - dist/assemble.sh: Debian/Ubuntu build PostgreSQL --with-system-tzdata, so the share dir has no timezone/ tree. Add --tzdir with a /usr/share/zoneinfo fallback (recorded as tzdata_source in dist.json). macOS/Homebrew unchanged. - A PGX runtime detaches the engine on 'pgx start'; if its parent is gone and PID 1 does not reap (a bare container), the engine becomes a zombie on exit. kill(pid,0) reports a zombie as alive, so stop/status hung and restart was blocked by the engine's own socket lock. Fixes: pid_alive() treats a Linux zombie (/proc/<pid>/stat state Z) as not alive; pgx start clears a socket lock whose PID is dead (zombie-aware); the Python harness filters zombies from engine_pids(). Deployments with an init (systemd, docker run --init) or the image's 'pgx serve' (engine is PID 1) never hit this. - test_crash: ephemeral async-commit loss is reported, not bounded by an arbitrary time window (documented synchronous_commit=off behaviour; the prefix and no-partial-batch invariants still assert no corruption). Linux arm64 results against the assembled dist: lifecycle, 19 SQL tests, clone, restart-persistence, 12 isolation tests, and the 10 crash tests all pass. macOS ARM64 unchanged: 90 cargo tests, fmt and clippy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
README Quickstart now leads with the install one-liner and the pgx CLI (build-from-source kept below). docs/pgx/local-dev/ adds install, CLI reference, integrations (Rails/Node/Python/Go/psql), storage (COW vs portable), troubleshooting, architecture and benchmark methodology, plus validation-report.md: the full suite matrix, per-test regression differences with causes, the durability and crash-recovery results, the macOS/Linux/ZFS/Docker coverage breakdown, risks, and the GO / NO-GO / BLOCKED decision (GO for macOS and Linux arm64 local dev; Docker, Linux x86-64 and ZFS BLOCKED on infrastructure, gated by CI). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…L encoding, install safety) Fixes from the final fresh-context review: - conf.rs: validate every PGX_LISTEN entry (IP / CIDR / special token) and reject whitespace, quotes and '#', closing a pg_hba injection (a templated listen value could have written a 'trust' rule) and the malformed-CIDR breakage; pg_hba now emits a CIDR as-is instead of appending /32. - process.rs: is_our_engine matches the '-D <data>' launch flag, not the data path as a bare substring, so a backup/editor touching ~/.pgx cannot be taken for the engine; init --force refuses while the recorded PID is alive at all, so a mis-identified live cluster is never deleted. - cmd/db.rs: the password in a connection URL is percent-encoded as userinfo (escaping / ? # among others), so a PGX_PASSWORD with special characters no longer yields a malformed postgres:// URL. - dist/install.sh: move an existing install aside before renaming the new one in, so an interruption can never leave no install (brief: avoid partial installs). Deferred (Minor): the wire-protocol parser can panic on a malformed message from the local trusted engine (clean abort, not an out-of-bounds read; never fires in normal operation). Full suite: 95 pgx tests, fmt and clippy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lockfile entries for the new crates/bin/pgx workspace member (libc, serde_json and their existing transitive deps; no new external dependency trees). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 'Unit tests' step ran `cargo test -p pgx --release` with PGX_DIST="", which also pulls in the integration suites in tests/ (clone/lifecycle/…) that require an assembled distribution — so they panicked 'PGX_DIST= lacks bin/postgres' and failed the job before the black-box step could run with PGX_DIST set. Scope that step to `--bin pgx` (the crate's own unit tests). Also drop `docker build -q` in docker_test.sh so a build failure shows the builder's output (e.g. assemble.sh's own error) instead of only buildkit's summary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two latent failures that only surface now that the Unit tests step is scoped
correctly (the integration suites never actually ran in CI before):
- CI assembled the dist with --version ci-<sha>, so `pgx version` reported a
non-0. version and version_prints_engine_and_compat failed its
starts_with("0.") check. Assemble with 0.0.0-ci.<sha> (a valid-looking
prerelease) and match it in the Export PGX_DIST glob.
- free_port() used bind(:0), so under parallel tests two harnesses could be
handed the same recently-closed ephemeral port and race to bind it in
`pgx init` (exit 4). Hand out a distinct, pid-offset, bind-verified port per
call instead.
Neither assertion is weakened. clone/lifecycle/databases/doctor pass locally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Docker image build failed with 'initdb: error: cannot be run as root': assemble.sh bakes the base cluster with initdb, which refuses root, and the builder stage runs as root. Create a 'builder' user and run assemble as it. - The harness execs a freshly-copied pgx binary; under CI parallelism the spawn can transiently fail (EAGAIN / ETXTBSY). Retry a spawn *failure* (not a non-zero exit, which is a real result) a few times. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The container ran as root, and the PGX engine (like PostgreSQL) refuses "root execution of the PostgreSQL server" — so `pgx serve` exited status 1 and the container never accepted TCP. Add an unprivileged `pgx` user, own the data/run dirs, COPY the distribution --chown=pgx (so the bundled base cluster, mode 700, is readable), and USER pgx. The named volume inherits pgx ownership. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Local-development distribution for PGX
Makes PGX installable and usable locally with no system PostgreSQL: a self-contained, relocatable PostgreSQL-18-compatible runtime plus a
pgxCLI, an installer, a Docker image, and an extensive automated test suite.What lands
crates/bin/pgx— thepgxCLI:init / start / stop / restart / status / create / drop / clone / ls / url / doctor / serve / version.dist/— relocatable distribution (assemble.sh), one-line installer (install.sh), Docker image +pgx serveentrypoint, and the CI/release workflows.tests/pgx,tests/apps,tests/compat— black-box SQL, crash recovery, transaction isolation, storage/clone, concurrency and stress suites; real app fixtures (psql/node/python/go/rails); the ported compatibility suite.fix: libxml2 soname probe(b2ac5558e8) — the one engine change: XML functions on hosts with libxml2 ≥ 2.14.docs/pgx/local-dev/.Verification (already run locally)
Not covered here (tracked separately)
Release-branch model
mainis the public development + GitHub-Release branch. The stable engine PGRun deploys lives onprod(pinned at913630e11d=pgx-v0.1-beta.1). See the stacked PR for the CI gating that enforces "public artifacts frommainonly".🤖 Generated with Claude Code