diff --git a/AGENTS.md b/AGENTS.md index d3f1ba7..cb2f1bf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,7 +29,7 @@ Table-driven tests sit beside the source as `*_test.go`; filesystem tests use `t The e2e suite in `test/e2e` carries `//go:build e2e`. It needs Docker and a host with the `amneziawg` module loaded, so it runs on a Linux host, not on macOS. It uses the published -`awg-grpc` image, builds usher from the tree and runs the production interval constants: +`awg-grpc` image, builds usher from the tree and keeps the production intervals: ```sh GOOS=linux GOARCH=amd64 go test -c -tags e2e -o e2e.test ./test/e2e @@ -69,7 +69,8 @@ scripts/format.sh ``` Run before presenting changes. It checks gofmt and runs `go vet` without tags and with the `e2e` -tag. `gofmt -w .` fixes formatting. The pre-commit hook and the CI `check` job run the same script. +tag. `gofmt -w .` fixes formatting. The CI `check` job runs the same script; the pre-commit hook +runs it only when a `.go` file is staged. After a change to a shell script or a workflow, also run: @@ -96,41 +97,28 @@ the notes of the annotated tag. ## Conventions -- A private key, a preshared key or a `client_params` value is never logged, and an error message - never quotes one. A public key without an entry is logged as base64. -- A Reality `privateKey` or `mldsa65Seed`, a `certificateFile` or `keyFile`, a PEM key, `sockopt`, - a server `fallbacks` entry and a drift value never reach `clients/`, a log line or an error. A - drift reason names the differing part only. -- usher never writes the Xray server config. It owns the users of the managed inbounds and - `clients/`. -- `state/users.json` is the only source of secrets. The kernel and `clients/` derive from it, and - usher saves it before any backend's `Apply`. -- A failure of one interface never stops the daemon; the pass logs it and goes on. +- The secret rules and the ownership rules are the Locked decisions in + `docs/architecture/README.md`. Read them before touching logging, errors or `clients/`. - No interface without a second implementation, no exported surface only for tests. ## Layout ``` cmd/usher/ entry point: environment, signals, run loop wiring -internal/config/ usher.yml loader and validation -internal/state/ users.json load and save -internal/atomicfile/ atomic file replace shared by state and the clients mirror -internal/pass/ settings, the Backend/Session seam, the reconcile pass -internal/awg/ awg backend: enrollment, address allocation, .conf rendering -internal/xray/ Xray backend: config reader, drift guard, users, mirror, rendering, stale socket -internal/clients/ client file paths and the clients/ mirror -internal/watch/ file poll, refill ticker and the trigger loop -deploy/ example compose file, example Xray config, e2e compose files -test/e2e/ awg end-to-end test through docker compose (tag e2e) -test/e2e/xray/ Xray end-to-end test (tag e2e) +internal/ packages, one per component of docs/architecture/overview.md +deploy/ example compose file, usher.yml and Xray config; e2e compose files and configs +test/e2e/ awg end-to-end test (tag e2e); xray/ holds the Xray end-to-end test scripts/format.sh the Format chain (see Format) -scripts/git-hooks/ pre-commit hook: branch guard, then scripts/format.sh +scripts/git-hooks/ pre-commit hook: branch guard, then scripts/format.sh for staged .go files scripts/ci/module.sh builds and loads the pinned kernel module in the CI e2e job +docs/ operator docs (xray.md, operations.md); architecture/ and plans/ (see Docs) .github/workflows/ ci.yml and release.yml Dockerfile Go build stage and the runtime image ``` ## Docs -Stable design is in `docs/architecture/`, starting at `overview.md`. Dated plans are in -`docs/plans/`, completed ones in `docs/plans/completed/`. +`README.md` is the front page for operators; `docs/xray.md` holds the Xray server config contract +and `docs/operations.md` the operator procedures. Stable design is in `docs/architecture/`, +starting at its `README.md`, which lists the locked decisions. Dated plans are in `docs/plans/`, +completed ones in `docs/plans/completed/`. diff --git a/README.md b/README.md index 84a9a48..4b1f088 100644 --- a/README.md +++ b/README.md @@ -8,186 +8,140 @@ User management for the VPN services of one node. You list users and the interfaces each may use -in one file; within seconds usher creates their keys, puts them into the running services and -writes a ready client config per user and interface. Addresses and keys stay internal: you type -names only. An interface is an AmneziaWG interface or an inbound of your Xray config. +in one file; a few seconds after you save it, usher creates their keys, puts them into the running +services and writes a ready client config per user and interface. Addresses and keys stay +internal: you type names only. An interface is an AmneziaWG interface or an inbound of your Xray +config. usher holds no capabilities and is never in the packet path. Every VPN service runs in its own container with a small management API on a unix socket; usher calls that API and nothing else, so a crash or an upgrade of usher leaves every session running. AmneziaWG is supported through [awg-grpc](https://github.com/mrcsin/awg-grpc), Xray through its own gRPC API. +## How it fits together + +```mermaid +flowchart TD + subgraph docker[Docker services] + usher[usher] + awg[awg-grpc] + subgraph xray[Xray] + vless[inbound vless-reality] + hy2[inbound hy2] + end + end + subgraph os[OS network interfaces] + awg0[awg0] + awg1[awg1] + end + usher -- "USHER_AWG_SOCKET, gRPC" --> awg + usher -- "USHER_XRAY_SOCKET, gRPC HandlerService" --> xray + awg -- "awg set" --> awg0 + awg -- "awg set" --> awg1 +``` + +usher drives one awg-grpc and one Xray: each variable names one socket. One awg-grpc carries any +number of AmneziaWG interfaces, one Xray any number of inbounds, and every name you put in +`usher.yml` is one of them. + +The socket is the only control channel, and it has no authentication: any process that can open +it controls every peer or user of that service. Each service creates its socket with mode `0660`, +so usher must belong to the socket's group to open it. The usher image runs as uid and gid 1000, +and the example compose file gives both sockets that group. + +Through the sockets usher owns every interface awg-grpc reports and every Xray inbound with a tag +and a supported protocol: it removes any peer or user that `usher.yml` does not list. + ## Users +usher keeps everything under `/srv/usher` in its container; the example compose file mounts the +three directories: + +``` +/srv/usher/ +├── config/usher.yml you write it, usher only reads it +├── state/users.json usher writes it, back it up +└── clients//. usher writes it, hand the files to the users +``` + `config/usher.yml`: ```yaml alice-phone: [awg0] -alice-laptop: [awg0, awg1] +alice-laptop: [awg0, vless-reality] bob: [] ``` -- One name per device: a WireGuard key cannot serve two devices at once. +- Give each device its own name. +- A user name has up to 32 lowercase letters, digits, `_` and `-`, and starts with a letter or a + digit. - Removing a name, or setting it to `[]`, switches the user off. The keys stay, and putting the name back restores the same config. -- A broken file changes nothing: usher keeps the last valid one and logs the error with its line. -- A service that restarts empty is refilled at the next refill tick (every 30 s; Xray measured - 28 to 32 s, bounded at 45 s in the e2e) without any change to the file. +- usher checks the file at every change: YAML syntax, the name rules, no user or interface listed + twice, and at least one user with an interface, so a truncated file cannot switch everyone off. + A file that fails changes nothing: usher logs the error with its line and keeps the last valid + file in memory, through any number of failed saves in a row. It never writes `usher.yml`. After + a restart there is no earlier valid file, so usher logs and does nothing until the file is + valid. +- awg-grpc and Xray hold no user state: a restart of either starts it empty. usher holds all of + it in `state/users.json` and puts the users back within 30 s. + +For this file `clients/` holds: + +``` +clients/ +├── alice-phone/ +│ └── awg0.conf +└── alice-laptop/ + ├── awg0.conf + ├── vless-reality.json + └── vless-reality.txt +``` -Client files appear in `clients//`: `.conf` for AmneziaWG, `.json` and `.txt` -for Xray. Secrets live in `state/users.json`, the only copy of every key: keep the state volume in -the backup set. +`state/users.json` is the only copy of every key, so keep the state volume in the backup set. +[`docs/operations.md`](docs/operations.md) covers client isolation, DNS and new keys for a user. ## Running [`deploy/compose.example.yml`](deploy/compose.example.yml) runs usher next to awg-grpc and Xray. -Images are `ghcr.io/mrcsin/usher:`, one per release. Drop the service you do not use. +With one backend, delete the other service and its socket variables from the file. Images are +`ghcr.io/mrcsin/usher:`, such as `v0.2.0`. -| Variable | Example | Meaning | -| -------------------- | ------------------------ | ---------------------------------------------------- | -| `USHER_HOST` | `203.0.113.10` | IPv4 address clients connect to; no host names | -| `USHER_AWG_SOCKET` | `/run/awg-grpc/awg.sock` | awg-grpc socket; enables the AmneziaWG backend | -| `USHER_DNS` | `1.1.1.1,1.0.0.1` | DNS servers in AmneziaWG configs; needs the awg socket | -| `USHER_XRAY_SOCKET` | `/run/xray/api.sock` | Xray API socket; enables the Xray backend | -| `USHER_XRAY_CONFIG` | `/etc/xray/config.json` | Xray server config, read-only; needs the Xray socket | +| Variable | Required | Example | Meaning | +| ------------------- | ------------------------ | ------------------------ | ------------------------------------------------ | +| `USHER_HOST` | yes | `203.0.113.10` | IPv4 address clients connect to, not a host name | +| `USHER_AWG_SOCKET` | one of the two sockets | `/run/awg-grpc/awg.sock` | awg-grpc socket; enables AmneziaWG | +| `USHER_DNS` | with `USHER_AWG_SOCKET` | `1.1.1.1,1.0.0.1` | DNS servers in AmneziaWG configs | +| `USHER_XRAY_SOCKET` | one of the two sockets | `/run/xray/api.sock` | Xray API socket; enables Xray | +| `USHER_XRAY_CONFIG` | with `USHER_XRAY_SOCKET` | `/etc/xray/config.json` | Xray server config, read-only | -`USHER_HOST` is always required. Set at least one socket. Publish each interface port 1:1, and give -usher the socket group of the service containers. +`USHER_AWG_SOCKET`, `USHER_XRAY_SOCKET` and `USHER_XRAY_CONFIG` are absolute paths inside the +usher container. -To issue new keys for a user, stop usher, delete its `/` entry from -`users.json` and start it again. The Xray entries are under `xray` and named `/`. If you -change an inbound between a protocol with a UUID (VLESS, VMess) and one with a password (Trojan, -Hysteria2), the stored credentials no longer fit: usher reports "stored credential does not fit" -and keeps the old client files until you stop usher, delete the `/` entries of that tag -and start it again. +`USHER_HOST` is an address because a client resolves a host name through its DNS server, which +sits inside the tunnel. When the tunnel drops, the client cannot resolve the name and does not +reconnect. -### Upgrading +Besides the sockets, every deployment gives usher read access to the Xray config file and +publishes each interface port on the same host port, because client configs carry the port of the +interface. usher itself listens on no port. The comments in the example compose file carry the +rest of the wiring. -- `USHER_AWG_SOCKET` is required for the AmneziaWG backend; earlier releases hardcoded the path. - Set it to the awg-grpc socket (`/run/awg-grpc/awg.sock` in the example) before you upgrade. -- The first save writes `users.json` as version 2 and copies the old file to `users.json.v1`. To - roll back to an earlier image, stop usher and rename `users.json.v1` back to `users.json`. +## AmneziaWG -## Xray +An interface is a `.conf` file of awg-grpc; usher manages its peers only. A user gets the +lowest free address of the interface's IPv4 subnet, an X25519 key pair and a preshared key. The +client `.conf` carries that address, `USHER_DNS`, the obfuscation parameters of the interface and +`USHER_HOST` with the interface port as the endpoint. An interface holds at most 1000 users. -You write the Xray server config and usher fills its inbounds with users. An inbound tag is the -interface name in `usher.yml`, up to 32 characters. usher reads the config file and puts users -into the running inbounds through Xray's `HandlerService`. It never writes the config. Xray keeps -users in memory only, so usher refills them at the next refill tick after a restart. - -Per user and inbound usher writes `clients//.json`, a full client config with a local -socks proxy on 127.0.0.1:10808 and an http proxy on 127.0.0.1:10809, and `clients//.txt`, -a share link. - -### Server config - -- One JSON file; comments are allowed. No `-confdir`. -- Mount the directory that holds the file into both containers, not the file. usher then sees a - replaced file at its next refill tick. -- `api.services` is `["HandlerService"]` and `api.listen` is a unix path with `,0660`, such as - `/run/xray/api.sock,0660`. The Xray process group must equal usher's group. -- The socket directory is a bind mount that usher can write. After a hard kill Xray leaves a stale - socket and fails to start on it. usher unlinks the socket at its next tick, and Xray's restart - policy (`restart: unless-stopped`) starts it again. usher unlinks a socket only when no process - holds the `.lock` file that Xray keeps beside it, or, when that file is gone (a failed - start removes it), when a connection to the socket is refused. A live Xray keeps its socket. -- Keep certificates and key files outside the directory that usher mounts. usher does not read - them, and Xray reloads them itself. The example mounts `./xray-certs` into the Xray container - only. -- The Xray image follows `xray-core` in `go.mod` (now `ghcr.io/xtls/xray-core:26.10.10`). Another - version can change how Xray builds an inbound and put every inbound in drift. -- Every managed inbound has a tag, a single integer `port`, no unix `listen` and an empty - `clients` list. A Hysteria2 inbound also has an empty `hysteriaSettings.auth`; usher gives each - user an auth. VLESS `decryption` is `none`. - -[`deploy/xray/config.json`](deploy/xray/config.json) is an example with placeholders that Xray -refuses to start with. - -### Supported shapes - -| Part | Values | -| --------- | --------------------------------------------------------------- | -| Protocol | VLESS, Hysteria2, VMess, Trojan | -| Transport | raw (header none), ws, grpc, httpupgrade, xhttp, hysteria | -| Security | none, tls, reality | -| Mask | salamander (Hysteria2, without `packetSize`), `quicParams` | - -Shadowsocks and WireGuard inbounds are ignored. VMess and Trojan are covered by unit tests only. -An inbound outside this table fails; the log names the field. Its existing client files stay. - -### The x-usher block - -Add `x-usher` to an inbound to tell usher what clients need. Xray ignores the key. Every field is -optional, and an unknown key fails the inbound. - -```json -"x-usher": { - "address": "203.0.113.10", - "port": 443, - "sni": "example.com", - "fingerprint": "chrome", - "trust": "ca", - "pin_sha256": "…", - "hop_ports": "20000-30000" -} -``` +## Xray -| Field | Default | Meaning | -| ------------- | ------------------------------------ | ---------------------------------------------------- | -| `address` | `USHER_HOST` | address clients dial | -| `port` | the inbound `port` | port clients dial | -| `sni` | `tlsSettings.serverName`, or the first Reality `serverNames` entry without `*` | name clients send; on Reality one of the `serverNames` entries | -| `fingerprint` | `chrome` | uTLS fingerprint; Reality allows `chrome`, `firefox`, `safari` | -| `trust` | `ca` | `ca` or `pin`, see below | -| `pin_sha256` | hash of the inline certificate | hex SHA-256 of the leaf; only with `trust: pin` | -| `hop_ports` | none | Hysteria2 port range `from-to`, from < to | - -A TLS inbound with neither `sni` nor `tlsSettings.serverName` fails. Reality matches the name by -exact lookup in `serverNames`, so on Reality an `sni` that is not literally one of its entries -fails the inbound, and so does a config with only wildcard names. The privateKey and the -`mldsa65Seed` must be base64url keys of the right size, and the seed must differ from the key. - -### TLS trust - -- `ca`: the client verifies the server certificate against its CA roots and gets only the - server name, the ALPN list and the fingerprint. Use a certificate that a public CA issued for - your domain. -- `pin`: the client also gets `pinnedPeerCertSha256`, from `pin_sha256` or from an inline - `certificates[].certificate`. An inbound with neither fails. A Hysteria2 pin inbound gets no - `.txt`, because v2rayNG and Hiddify do not read the pin from a `hysteria2://` link. - -Reality needs no certificate. usher derives the public key from `privateKey`, takes `shortIds[0]` -and sets `spiderX` to `/`. With `mldsa65Seed` it adds `mldsa65Verify`. Private keys, seeds, key -files and fallbacks never reach `clients/` or the log. - -### Client compatibility - -Measured on 2026-10-14 against Xray 26.10.10. - -- Reality links work in Xray-core clients only. sing-box 1.14.3 fails the server's - `X25519MLKEM768` requirement. Only `chrome`, `firefox` and `safari` carry it. -- xhttp is not in upstream sing-box. -- VLESS ws, Trojan, VMess and Hysteria2 with salamander work in sing-box. -- A user in `usher.yml` that loses an interface loses new connections at once. An open Hysteria2 - QUIC connection stays up until the client reconnects. -- `hop_ports` needs a DNAT on the host from the range to the inbound port. usher does not set it. - -### Config edits - -usher compares each inbound with the running Xray at every pass. After you edit a managed inbound -in the file, restart Xray. Until then usher logs a drift reason that names the part (port, proxy -settings, stream settings), keeps the old client files and still removes users who left -`usher.yml`. Certificate files that the config names are reloaded by Xray itself and cause no -drift. When the running inbound is missing, runs another protocol or cannot be compared with the -file, usher changes nothing on it, because an account of the wrong type crashes Xray; restart Xray. - -An inbound with an `x-usher` error, or one whose credential or client file cannot be built, keeps -its old client files, and usher still removes the users who left `usher.yml` from it, once the -drift check has compared the inbound with the running Xray and found it usable. It adds no -user and changes no credential until you fix the file. An inbound that fails any other check (port, -`listen`, non-empty `clients`, transport, Reality keys, a tag that is not a valid interface name -or that the file defines twice) is left alone: usher cannot trust the file entry, so it adds and -removes no user on that tag and keeps its old client files. +You write the Xray server config, and usher fills its inbounds with users. An inbound tag is the +interface name in `usher.yml`. usher puts users into the running inbounds through Xray's +`HandlerService` and never writes the config. The `.json` client config runs a local socks proxy +on 127.0.0.1:10808 and an http proxy on 127.0.0.1:10809; the `.txt` file is a share link. +[`docs/xray.md`](docs/xray.md) covers the server config contract, the supported shapes, the +`x-usher` block, TLS trust, client compatibility and config edits. ## License diff --git a/deploy/compose.example.yml b/deploy/compose.example.yml index da2cadb..f79ca57 100644 --- a/deploy/compose.example.yml +++ b/deploy/compose.example.yml @@ -11,6 +11,7 @@ services: # awg gets one pipe per preshared key; 1000 peers come close to the default soft limit of 1024. ulimits: nofile: { soft: 65536, hard: 65536 } + # The host port equals the interface port: client configs carry it. ports: - "51820:51820/udp" volumes: @@ -27,6 +28,7 @@ services: # Must equal usher's group, so that usher can open the API socket. user: "0:1000" command: ["run", "-config", "/etc/xray/config.json"] + # The host port equals the interface port: client configs carry it. ports: - "443:443/tcp" - "8443:8443/udp" @@ -39,7 +41,7 @@ services: # Owned by uid 1000, not a named volume: usher unlinks a stale API socket after a crash. - ./xray-sock:/run/xray usher: - image: ghcr.io/mrcsin/usher:v0.1.0 + image: ghcr.io/mrcsin/usher:v0.2.0 restart: unless-stopped cap_drop: [ALL] read_only: true diff --git a/docs/architecture/README.md b/docs/architecture/README.md new file mode 100644 index 0000000..ed4f42f --- /dev/null +++ b/docs/architecture/README.md @@ -0,0 +1,20 @@ +# Architecture + +- [overview.md](overview.md): purpose, components, the pass, keys and addresses, the Xray backend, + state. + +## Locked decisions + +| Area | Decision | +|---|---| +| Scope | AmneziaWG over IPv4 and the Xray shapes in [docs/xray.md](../xray.md#supported-shapes). No subcommands besides `usher run`, no central management, no migration of existing peers. | +| Ownership | usher owns the peers of every interface awg-grpc reports and the users of every managed Xray inbound; an unreferenced interface gets an empty list. It never writes the Xray config. | +| Broken `usher.yml` | Never changes state. The last valid file is kept in memory only. A file without any user and interface pair is invalid, so a truncated file cannot wipe a node. | +| Failures | A failure of one interface never stops the daemon and never blocks the others. An interface without an IPv4 prefix fails only when a user references it. | +| `clients/` | Files of an interface that no backend reports are removed, unless a backend is down. | +| `USHER_HOST` | An IPv4 address, not a name: a client whose DNS points into the tunnel cannot resolve the name again to reconnect. | +| Secrets | `state/users.json` is the only source of secrets. No private key, PSK or `client_params` value reaches a log line or an error message. No Reality `privateKey` or `mldsa65Seed`, key file, PEM key, `sockopt`, `fallbacks` entry or drift value reaches `clients/`, a log line or an error message; a drift reason names the differing part only. A public key without an entry is logged as base64. | + +## Decision records + +None. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 9c3c01f..58afce0 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -8,11 +8,11 @@ operator's Xray config. The operator types names only. Addresses and keys are in ## Stack -Go, one binary: `usher run`. It talks gRPC to `awg-grpc` (`github.com/mrcsin/awg-grpc` v0.1.0) -over the socket in `USHER_AWG_SOCKET`, and to Xray's `HandlerService` over the socket in -`USHER_XRAY_SOCKET`. At least one is set. It runs in a container with no capabilities, a read-only -root file system and uid 1000. Config parsing uses `gopkg.in/yaml.v3`. The Xray backend imports -`xray-core` (`infra/conf`, `app/proxyman/command`) and `libxray/share`. +Go, one binary: `usher run`. It talks gRPC to awg-grpc over the socket in `USHER_AWG_SOCKET`, and +to Xray's `HandlerService` over the socket in `USHER_XRAY_SOCKET`. At least one is set. It runs in +a container with no capabilities, a read-only root file system and uid 1000. The Xray backend +builds inbounds with `xray-core` itself and renders share links with `libxray`, so the Xray image +must match the `xray-core` version in `go.mod`. ## Components @@ -20,7 +20,7 @@ root file system and uid 1000. Config parsing uses `gopkg.in/yaml.v3`. The Xray |---|---| | `cmd/usher` | Reads the `USHER_*` environment, builds the backends, wires signals, the watcher and the pass. | | `internal/config` | Loads and validates `usher.yml`. | -| `internal/state` | Loads and saves `state/users.json`, version 2. | +| `internal/state` | Loads and saves `state/users.json`. | | `internal/atomicfile` | Replaces a file atomically for `state` and the `clients/` mirror. | | `internal/pass` | Settings, the `Backend` and `Session` seam and the reconcile pass. | | `internal/awg` | The awg backend: enrollment, address allocation, `.conf` rendering. | @@ -32,37 +32,40 @@ root file system and uid 1000. Config parsing uses `gopkg.in/yaml.v3`. The Xray A backend is a `Backend{Name, Suffixes, Open}`. `Open` returns a `Session` with `Interfaces`, `Prepare` (enroll users, render files), `Apply` and `Close`. `Suffixes` lists the file suffixes -the backend owns (`.conf`; `.json` and `.txt`). A backend that fails to open is down for the pass: -usher keeps every file with one of its suffixes. An interface name that two backends report fails -in both. The steps below run per backend; the text names the awg calls. +the backend owns. + +| Step | awg | Xray | +| ----------- | ------------------------------------------------ | ----------------------------------------------------------- | +| `Open` | `GetStatus` | read the config, remove a stale socket, `ListInbounds`, drift check | +| `Interfaces`| every interface awg-grpc reports | every managed inbound tag of the config | +| `Prepare` | fails an interface that is not present | returns the structural error of an inbound | +| `Apply` | `ApplyPeers`, `allow_empty` with no users | `GetInboundUsers`, then `AlterInbound` per changed user | +| Suffixes | `.conf` | `.json`, `.txt` | 1. Load `usher.yml`. When it is invalid, use the last valid file of this process. With none, log and stop. Load `users.json`. On error, log and stop. -2. Call `GetStatus`. On error, log and stop. -3. For each reported interface: fail it when `present=false`. Otherwise enroll every user that - references it and render their configs. An enrollment or render error fails the interface. -4. Save `users.json` when step 3 created or changed an entry. -5. For each interface that did not fail: call `ApplyPeers` with one peer per referencing user - (`address/32`), with `allow_empty` when no user references the interface. An error fails the +2. `Open` every backend. A backend that fails to open is down for this pass: usher keeps every + file with one of its suffixes, and the other backends go on. +3. For each interface of each open backend: fail it when two backends report its name. Otherwise + `Prepare` it: enroll every user that references it and render their files. An error fails the interface. +4. Save `users.json` when step 3 created or changed an entry. On error, log and stop. +5. `Apply` each interface that did not fail. An error fails the interface. 6. When no backend is down, log every user and interface pair that no backend reports ("interface is not reported by any backend"). 7. Make `clients/` equal to the rendered files of applied interfaces plus the current files of - failed ones. + failed interfaces and down backends. -Two triggers start a pass: a change of `usher.yml` and a 30 s refill ticker. One goroutine runs the -watcher loop and the passes, with the poll ticker, the settle timer and the refill ticker as cases -of one `select`. A tick that fires during a pass waits in its ticker, which buffers one tick, so -each source yields at most one following pass. A refill tick and a file change during one pass give -two following passes. File polling follows `reproxy` (`app/discovery/provider/file.go`): `os.Stat` -every 3 s, and a new modification time counts once it has stayed unchanged for 500 ms. The -connection is opened per pass and every call has a 10 s deadline. +A pass runs at start, after a change of `usher.yml` and at every 30 s refill tick. One goroutine +runs the watcher loop and the passes, with the poll ticker, the settle timer and the refill ticker +as cases of one `select`. A tick that fires during a pass waits in its ticker, which buffers one +tick, so each source yields at most one following pass. The watcher calls `os.Stat` every 3 s, and +a new modification time counts once it has stayed unchanged for 500 ms. Each backend opens its +connection per pass, and every call has a 10 s deadline. -## Keys and addresses +## AmneziaWG keys and addresses -- `state/users.json` is the only source of secrets. Entries are keyed `interface/user` under the - `awg` group. The kernel and `clients/` derive from it. usher saves it (temp file, fsync, rename, - directory fsync, mode 0600) only when an entry changed, and always before any backend's `Apply`. +- Entries are keyed `interface/user` under the `awg` group of `users.json`. - A new entry gets an X25519 key pair (`crypto/ecdh`), 32 random PSK bytes and an address. - The address is the lowest host address of the interface's first IPv4 prefix that is not the network or broadcast address, not an interface address and not held by another entry of that @@ -70,44 +73,10 @@ connection is opened per pass and every call has a 10 s deadline. changes the interface address, is reassigned and logged, and the config is rewritten. - Switching a user off removes the peer and the client file and keeps the entry, so the same name gets the same key and address back. usher never deletes entries. - -## Locked decisions - -- The scope is AmneziaWG over IPv4 and the Xray shapes in the README. -- usher owns every interface the wrapper reports, and an unreferenced interface gets an empty - list. -- A broken `usher.yml` never changes state. The last valid file is kept in memory only. -- A file without any user and interface pair is invalid, so a truncated file cannot wipe a node. -- A failure of one interface never stops the daemon and never blocks the others. -- An interface without an IPv4 prefix fails only when a user references it. -- `clients/` files of an interface that no backend reports are removed. -- `USHER_HOST` is an IPv4 address, not a name: a client whose DNS points into the tunnel cannot - resolve the name again to reconnect. -- No private key, PSK or `client_params` value is logged. A public key without an entry is logged - as base64. -- No Reality `privateKey` or `mldsa65Seed`, key file, `sockopt`, `fallbacks` entry or drift value - is logged or written to `clients/`. -- No subcommands besides `usher run`, no central management, no migration of existing peers. - -## Operations - -- `clients/` belongs to usher: anything not rendered from the applied state is removed. Files are - mode `0600`, uid 1000. -- Changing an interface's address, key, port or obfuscation values rewrites every client config of - that interface; those configs are handed out again. - Addresses are never freed: switched-off users keep theirs. A full subnet fails the whole interface, refill included. awg-grpc accepts at most 1000 peers per interface. -- `users.json` sits in the `usher-state` volume (mode `0600`, uid 1000). To re-key or delete a user: - `docker compose stop usher`, then - `docker run --rm -it -u 1000:1000 -v _usher-state:/state alpine vi /state/users.json`, - delete the `/` entry, `docker compose start usher`. A user still in `usher.yml` - gets new keys. -- `USHER_HOST` is an address, not a name: a client that re-resolves a name while its DNS points - into the tunnel cannot reconnect. -- Client isolation and DNS redirection belong to the service config, not to usher. For AmneziaWG: - `FORWARD` drops between clients in `PostUp`; a resolver on the tunnel address (for example - cloudflared with a DoH upstream in the `awg` network namespace) with `USHER_DNS` set to that - address, and optionally a `DNAT` of port 53 in `PostUp`. +- A change of the interface key, port or `client_params` rewrites every client config of that + interface; those configs are handed out again. ## Xray backend @@ -116,32 +85,32 @@ connection is opened per pass and every call has a 10 s deadline. - The operator owns one Xray server config: inbounds, outbounds, routing, `api`, certificates, Reality keys, and a per-inbound `x-usher` block with client hints. Xray reads it at start. usher reads the same file read-only at every pass. -- usher owns the users inside the inbounds, through the API, and everything under `clients/`. Xray - keeps users in memory, so the refill tick restores them after a restart. -- A managed inbound has a tag that matches the interface name pattern (32 characters), one integer - port, no unix `listen`, empty `clients` and a protocol of vless, hysteria, vmess or trojan. Other - inbounds are skipped. An inbound that fails validation carries its error. A structural error - (port, listen, clients, decryption, transport, Hysteria auth, masks, xhttp, Reality keys, a decode - failure, an invalid or duplicate tag) is returned by `Prepare`, so the pass skips `Apply` and - touches no user of the tag; `Apply` also returns it if called. An `x-usher` hint error, like an - enroll or render error, is kept by `Prepare`, which returns none, so the pass still calls `Apply`. - `Apply` removes the users who left only when the drift check found the running inbound usable, - then returns the error, which keeps the old files. `validate` runs the hint check last, so a hint - error is the only reason left and `checkDrift` still compares the inbound. A tag defined twice is - reported once, and its entry carries the error. +- usher owns the users inside the managed inbounds, through the API, and everything under + `clients/`. Xray keeps users in memory, so the refill tick restores them after a restart. +- A managed inbound has a tag and a protocol of vless, hysteria, vmess or trojan. Other inbounds + are skipped. + +### Errors of an inbound + +`validate` puts every problem of an inbound into one of two classes: + +- A structural error (port, `listen`, `clients`, decryption, transport, security, Hysteria + fields, masks, xhttp, Reality keys, a decode failure, an invalid or duplicate tag). `Prepare` + returns it, so the pass skips `Apply` and touches no user of the tag. +- A hint error (`x-usher`), an enroll error or a render error. `Prepare` keeps it and returns + none, so the pass calls `Apply`, which removes the users who left and then returns the error. + `validate` runs the hint check last, so `checkDrift` still compares an inbound with a hint + error. + +A tag defined twice is reported once, and its entry carries the error. ### Reading the config `readConfig` reads the file with Xray's comment-aware reader into `infra/conf` types plus raw maps. It normalizes the aliases (`method`, `splithttp`, `websocket`, `tcp`, `splithttpSettings`, `tcpSettings`, Hysteria `users`) before validation and mirroring. Every error names the field and -never a value. `x-usher` decodes with unknown keys rejected. Reality matches names by exact lookup, -so there an explicit `x-usher.sni` must be one of the `serverNames` entries, and the config needs an -entry without a wildcard when `sni` is unset. A Reality `privateKey` and `mldsa65Seed` must decode, -and the seed must differ from the key. `hop_ports` requires from < to. An inbound without -`streamSettings` stays without them in `Raw`, so it builds as Xray builds the file. readConfig -registers a discard handler for Xray's global logger once, because `Build()` prints deprecation -warnings. +never a value. An inbound without `streamSettings` stays without them in `Raw`, so it builds as +Xray builds the file. ### Users @@ -154,8 +123,6 @@ warnings. logs `users changed`. The next pass converges from the listed state. - When the running inbound is missing, runs another protocol or could not be compared, `Apply` changes nothing: an account of the wrong type makes Xray's validator panic and kills the process. -- Removal stops new connections. An open Hysteria2 QUIC connection stays until the client - reconnects. ### Drift guard @@ -165,20 +132,19 @@ name a file are dropped on both sides, because Xray reloads those files itself. runtime counts as drift. The proxy type of the running handler is compared with the file's protocol before any `Build()`; a mismatch, or a file entry that does not decode or build, marks the drift unusable. The reason names the differing parts; an error of `Build()` becomes a fixed text, because -Xray quotes the keys and short ids it rejects. `Apply` reconciles users first and then returns the -drift error, so revocation works while the old client files stay. The exception is an unusable -drift: a missing inbound, another proxy type or a failed comparison (see Users). A running managed -inbound that the file does not define is logged each pass. +Xray quotes the keys and short ids it rejects. On a usable drift `Apply` reconciles users and then +returns the drift error, so the old client files stay. On an unusable drift it changes nothing. A +running managed inbound that the file does not define is logged each pass. ### Stale socket A killed Xray leaves its API socket, and a new start fails on it. Xray holds `flock(LOCK_EX)` on `.lock` while it listens. Before dialing, usher opens the lock read-only and tries `LOCK_EX|LOCK_NB`. When it gets the lock, no Xray listens: usher unlinks the socket and reports Xray -down for this pass. The restart policy starts Xray at its next attempt. The socket directory is a -bind mount that usher can write. A failed Xray start removes the lock file, so for a socket without -one usher probes the socket instead: only a refused connection counts as stale, and a live Xray, or -a socket that a new Xray replaced meanwhile, stays. A path that is not a socket is an error. +down for this pass. The restart policy starts Xray at its next attempt. A failed Xray start removes +the lock file, so for a socket without one usher probes the socket instead: only a refused +connection counts as stale, and a live Xray, or a socket that a new Xray replaced meanwhile, stays. +A path that is not a socket is an error. ### Client mirror @@ -191,12 +157,9 @@ a socket that a new Xray replaced meanwhile, stays. A path that is not a socket - `tlsSettings` is rebuilt: server name, ALPN, fingerprint and, with `trust: pin`, `pinnedPeerCertSha256`. - `realitySettings` is rebuilt: public key from `privateKey` (`crypto/ecdh`), `shortIds[0]`, - `spiderX` `/`, `mldsa65Verify` from `mldsa65Seed`. The fingerprint is `chrome`, `firefox` or - `safari`. -- Hysteria2: the client `auth` is the user's password. From salamander only `password` is copied; - `packetSize` fails the inbound. `hop_ports` adds a client `udphop` mask (mode - `intervalLocal,intervalRemote`, interval 30, in libxray's shape). The range needs a host DNAT to - the inbound port. + `spiderX` `/`, `mldsa65Verify` from `mldsa65Seed`. +- Hysteria2: the client `auth` is the user's password. From salamander only `password` is copied. + `hop_ports` adds a client `udphop` mask with mode `intervalLocal,intervalRemote` and interval 30. ### Outputs @@ -209,7 +172,12 @@ a socket that a new Xray replaced meanwhile, stays. A path that is not a socket ## State -`state/users.json` is version 2: the `awg` group keyed `interface/user` and the `xray` group keyed -`tag/user`. An `xray` entry holds exactly one of `id` and `password`. `Load` accepts versions 1 -and 2. Before the first save of a version 1 file usher writes `users.json.v1` beside it, so a -rollback is a file rename. +`state/users.json` is the only source of secrets; the kernel, Xray and `clients/` derive from it. +It is version 2: the `awg` group keyed `interface/user` and the `xray` group keyed `tag/user`. An +`xray` entry holds exactly one of `id` and `password`. usher saves it (temp file, fsync, rename, +directory fsync, mode 0600) only when an entry changed, and always before any backend's `Apply`. +`Load` accepts versions 1 and 2. Before the first save of a version 1 file usher writes +`users.json.v1` beside it, so a rollback is a file rename. + +`clients/` belongs to usher: anything not rendered from the applied state is removed. Files are +mode `0600`, uid 1000. diff --git a/docs/operations.md b/docs/operations.md new file mode 100644 index 0000000..437c5f0 --- /dev/null +++ b/docs/operations.md @@ -0,0 +1,20 @@ +# Operations + +This page covers what usher leaves to the services and how to issue new keys to a user. + +## Isolation and DNS + +usher does not configure the services. For AmneziaWG, the interface config does it: a `FORWARD` +drop between clients in `PostUp`, and a resolver on the tunnel address (for example cloudflared +with a DoH upstream in the awg-grpc network namespace) with `USHER_DNS` set to that address. + +## New keys for a user + +`state/users.json` has two groups: `awg` with entries `/` and `xray` with entries +`/`. usher never deletes an entry. To issue new keys: + +1. `docker compose stop usher` +2. `docker run --rm -it -u 1000:1000 -v _usher-state:/state alpine vi /state/users.json` +3. Delete the entries of the user, then `docker compose start usher` + +A user still in `usher.yml` gets new keys, and every client file of that user changes. diff --git a/docs/xray.md b/docs/xray.md new file mode 100644 index 0000000..f1e20d1 --- /dev/null +++ b/docs/xray.md @@ -0,0 +1,130 @@ +# Xray + +This page covers the server config contract, the supported shapes, the `x-usher` block, TLS +trust, client compatibility and config edits. + +## Server config + +- Xray starts with one config file (`-config`), and usher reads the same file. Xray's `-confdir` + mode, which merges several files, is not supported. The file may contain `//`, `/* */` and `#` + comments. +- Mount the directory that holds the file into both containers, not the file itself. An editor + replaces the file, and a bind mount of a single file keeps showing the old one. +- `api.services` is `["HandlerService"]`, and `api.listen` is a unix socket with mode `0660`, such + as `/run/xray/api.sock,0660`. +- The socket directory is a bind mount that usher can write. A hard kill leaves a stale socket, + and Xray then fails to start. usher removes that socket, and the restart policy + (`restart: unless-stopped`) starts Xray again. usher never removes the socket of a running Xray. +- Keep certificates and key files in a directory that only the Xray container mounts + (`./xray-certs` in the example): usher needs no access to private keys. Xray reloads changed + certificate files itself. +- Use the Xray image of the example compose file. usher compares every inbound with the running + Xray, and another Xray version can build an inbound differently, so every inbound reports a + drift (see [Config edits](#config-edits)). +- Every managed inbound has a tag, one integer `port`, no unix `listen` and an empty `clients` + list. A tag has up to 32 letters, digits, `_`, `=`, `+`, `.` and `-`, not starting with `=`, + `+`, `.` or `-`. VLESS has `decryption: none`. Hysteria2 has `security: tls`, `version: 2` in both + `settings` and `hysteriaSettings`, and an empty `hysteriaSettings.auth`; usher gives each user an + auth. + +[`deploy/xray/config.json`](../deploy/xray/config.json) is an example. Replace its +`REPLACE_WITH_*` placeholders; Xray does not start with them. + +## Supported shapes + +| Part | Values | +| --------- | ---------------------------------------------------------- | +| Protocol | VLESS, Hysteria2, VMess, Trojan | +| Transport | raw (header none), ws, grpc, httpupgrade, xhttp, hysteria | +| Security | none, tls, reality; Hysteria2 needs tls | +| Mask | salamander, on Hysteria2 only and without `packetSize` | + +usher manages only inbounds with a tag and one of these protocols; it skips all others. An +inbound outside this table fails; the log names the field, and its existing client files stay. +Other `finalmask` keys, such as `quicParams`, stay on the server: usher does not copy them to +clients. The end-to-end tests cover VLESS (Reality, xhttp, ws) and Hysteria2; the other +combinations have unit tests only. + +## The x-usher block + +Add `x-usher` to an inbound to tell usher what clients need. Xray ignores the key. Every field is +optional, and an unknown key fails the inbound. An example for a Hysteria2 inbound: + +```json +"x-usher": { + "address": "203.0.113.10", + "port": 443, + "sni": "vpn.example.com", + "fingerprint": "chrome", + "trust": "ca", + "hop_ports": "20000-30000" +} +``` + +| Field | Default | Meaning | +| ------------- | ------------------------------------ | ---------------------------------------------------- | +| `address` | `USHER_HOST` | address clients dial | +| `port` | the inbound `port` | port clients dial | +| `sni` | `tlsSettings.serverName`, or the first Reality `serverNames` entry without `*` | name clients send; on Reality one of the `serverNames` entries | +| `fingerprint` | `chrome` | uTLS fingerprint; Reality allows `chrome`, `firefox`, `safari` | +| `trust` | `ca` | `ca` or `pin`, see [TLS trust](#tls-trust) | +| `pin_sha256` | hash of the inline certificate | 64 hex characters, SHA-256 of the leaf; only with `trust: pin` | +| `hop_ports` | none | Hysteria2 only: port range `from-to`, from < to | + +A TLS inbound with neither `sni` nor `tlsSettings.serverName` fails. Reality matches the name by +exact lookup in `serverNames`, so on Reality an `sni` that is not literally one of its entries +fails the inbound, and so does a config with only wildcard names. The `privateKey` and the +`mldsa65Seed` must be base64url keys of the right size, and the seed must differ from the key. + +`hop_ports` needs a DNAT on the host from the range to the inbound port. usher does not set it. + +## TLS trust + +- `ca`: the client verifies the server certificate against its CA roots and gets only the + server name, the ALPN list and the fingerprint. Use a certificate that a public CA issued for + your domain. +- `pin`: needs `security: tls`. The client also gets `pinnedPeerCertSha256`, from `pin_sha256` or + from an inline `certificates[].certificate`. An inbound with neither fails. A Hysteria2 pin + inbound gets no `.txt`, because v2rayNG and Hiddify do not read the pin from a `hysteria2://` + link. + +Reality needs no certificate. usher derives the public key from `privateKey`, takes `shortIds[0]` +and sets `spiderX` to `/`. With `mldsa65Seed` it adds `mldsa65Verify`. Private keys, seeds, key +files and fallbacks never reach `clients/` or the log. + +## Client compatibility + +Checked against Xray 26.10.10 and sing-box 1.14.3. + +- Reality links work in Xray-core clients only: the server requires an `X25519MLKEM768` key + share, which sing-box does not send. Of the uTLS fingerprints, only `chrome`, `firefox` and + `safari` send it. +- xhttp is not in upstream sing-box. +- VLESS ws, Trojan, VMess and Hysteria2 with salamander work in sing-box. + +## Removing a user + +A user who loses an interface in `usher.yml` cannot open new connections on it. An open Hysteria2 +QUIC connection stays up until the client reconnects. + +## Config edits + +After you edit a managed inbound in the file, restart Xray. Until then the inbound in the file +differs from the running one, and usher logs a drift that names the differing part: proxy type, +proxy settings, port, stream settings or receiver settings. Certificate files that the config +names are reloaded by Xray itself and cause no drift. + +If you change an inbound between a protocol with a UUID (VLESS, VMess) and one with a password +(Trojan, Hysteria2), the stored credentials no longer fit. usher logs "stored credential does not +fit" and keeps the old client files until you delete the `/` entries of that tag as in +[New keys for a user](operations.md#new-keys-for-a-user). + +When an inbound has a problem, usher keeps its old client files and treats its users by the first +row that matches: + +| Problem | Users | +| ----------------------------------------------------------------------------- | -------------------------------------------------------- | +| The inbound fails a check of the file (port, `listen`, `clients`, transport, security, masks, Hysteria2 fields, Reality keys, a tag that is invalid or defined twice) | unchanged | +| The running inbound is missing, runs another protocol or cannot be compared | unchanged, because an account of the wrong type crashes Xray | +| An `x-usher` error, or a credential or client file cannot be built | users who left `usher.yml` are removed; none are added | +| A drift | in step with `usher.yml` |