diff --git a/AGENTS.md b/AGENTS.md index 1102ef7e..df4e13a0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,30 +1,18 @@ -# Agent Operations Guide +# AGENTS.md -## Voice Automation Flow -- Use `bin/run-voice-loop` to launch the Codex supervisor. By default it streams prompts through `codex exec --full-auto -`; override `CODEX_CHAT_CMD` before launch if you need a different model or options. -- The loop reads `state/session.json` for context. Flip `listening: true` with `bin/px-wake --set on` (or `--keyboard`) before speaking; the loop idles until that flag is raised. -- Audio feedback is produced through `tool-voice` (`espeak` fallback). Logs are appended to `logs/tool-voice-loop.log` and `logs/tool-voice-transcript.log`. Inspect quick stats with `bin/px-voice-report --json`. +This repository's agent guidance lives in **[CLAUDE.md](CLAUDE.md)** — the +cross-cutting invariants you must not break, and the map to where current truth +for each subsystem lives. -## Diagnostics & Safety -- Run `bin/px-diagnostics` at the start of a session. It narrates every check (status, sensors, weather, circle motion unless `--no-motion`, camera capture, speaker/microphone tests) and records a JSON summary in `logs/tool-diagnostics.log`. -- Keep wheels on blocks for live runs; dry-run (`PX_DRY=1`) skips motion and still plays the spoken announcements so you can verify the speaker. -- `bin/px-stop` remains the emergency halt; it is safe to call repeatedly. +Read it first. It applies to every agent working here, not only Claude Code. -## Automation Toolbox -- `bin/px-dance` performs a narrated demo routine (voice intro → circle → figure-eight → finale). Use `PX_DRY=1` to rehearse without motion. -- `bin/px-frigate-stream` pushes an RTSP feed (`rpicam-vid` → `ffmpeg`) to Frigate/go2rtc (`pi5-hailo.local` by default). Test first with `--dry-run` to confirm command lines. -- `bin/px-session` bootstraps a tmux workspace (voice loop, wake console, log tail). `--plan` prints the layout before launching. +Start with: -## Development Workflow -1. Activate the virtualenv: `source .venv/bin/activate`. -2. Implement helpers under `bin/` and keep logic in Python for easier testing. -3. Add or update pytest coverage in `tests/`; set `PX_BYPASS_SUDO=1` and `LOG_DIR=logs_test` (relative paths resolve under `PROJECT_ROOT`) in the test environment to avoid privileged operations. -4. Run `python -m pytest` before every commit (current suite covers voice tools, diagnostics, tmux plan, and streaming helpers). -5. Update documentation (`README.md`, `docs/TOOLS.md`, roadmap/strategy docs) alongside new features so operators have fresh instructions. +- [CLAUDE.md](CLAUDE.md) — the invariants and the navigation map +- [docs/architecture/overview.md](docs/architecture/overview.md) — the system map +- [docs/git-workflow.md](docs/git-workflow.md) — **never blanket-stage**; stage exact owned paths +- [docs/testing.md](docs/testing.md) — the suite runs on the robot it controls -## Lessons Learned -- Treat every helper as a modular tool Codex can invoke; build consistent JSON outputs and summaries to keep transcripts clean. -- State persistence is critical: always update `state/session.json` when a tool runs so the next Codex turn has context. -- Keep audio pathways live even in dry-run; it surfaced a muted speaker regression immediately. -- Use structured logging (`logs/tool-voice-transcript.log`, `logs/tool-*.log`) to audit behaviour and drive reporting tools. -- Leverage tmux (`bin/px-session`) during development to survive SSH drops and keep wake/log panes visible. +> This file previously carried its own operations guide, written when the voice +> loop was Codex-only. It described a system that no longer exists and is +> preserved in git history rather than here. diff --git a/CLAUDE.md b/CLAUDE.md index b313069b..70360a69 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,495 +1,212 @@ -# CLAUDE.md +# CLAUDE.md — SPARK's Constitution -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. +This file is the **rules and the map**. It holds the cross-cutting invariants +you must not break, and it says where current truth for each subsystem lives. -## Project Overview +It deliberately contains **no** subsystem detail, no incident history, and no +tuning constants. Those go stale, and a constitution that goes stale stops +being read. If you need to know how something works, follow a link — the +linked doc is the authority, and this file is not. -Helper scripts and Python library for a SunFounder PiCar-X robot built by Adrian and Obi together — with Obi, not for him. The system runs on a Raspberry Pi and uses a voice loop (Claude / Codex / Ollama) to control the car via spoken commands, with two jailbroken personas (GREMLIN and VIXEN) and a three-layer cognitive architecture that gives the robot an inner life. Adrian and Claude wrote the code; Codex and Gemini helped with QA. +--- -## Environment Setup +## What this is -```bash -source .venv/bin/activate -``` - -All `bin/` scripts source `bin/px-env` automatically, which sets `PROJECT_ROOT`, `LOG_DIR`, and adds `$PROJECT_ROOT/src` and `/home/pi/picar-x` to `PYTHONPATH`. - -**First use:** `cp state/session.template.json state/session.json` - -## Running Tests - -```bash -python -m pytest # full suite (~1235 tests) -python -m pytest tests/test_state.py # single file -python -m pytest -k test_name # single test -python -m pytest -m "not live" # skip hardware tests -sudo .venv/bin/python -m pytest tests/test_tools_live.py -v -s # live hardware tests -``` - -Test env vars (auto-set via `conftest.py` `isolated_project` fixture): `PX_BYPASS_SUDO=1`, `LOG_DIR=/logs`, `PX_SESSION_PATH=/state/session.json`, `PX_VOICE_DEVICE=null`. - -**Critical:** bin scripts run under `/usr/bin/python3` (not venv) — picarx/robot_hat live in system site-packages. - -## Architecture +A SunFounder PiCar-X running on a Raspberry Pi, built by Adrian and Obi +together — *with* Obi, not for him. **SPARK** is its default persona: a warm, +non-coercive companion for a neurodivergent child, with a three-layer cognitive +architecture, a voice loop, and two jailbroken alternate personas. -### Python Library (`src/pxh/`) +Two consequences follow from that sentence, and most of the rules below are +downstream of them: -| Module | Purpose | -|--------|---------| -| `state.py` | Thread-safe session management via `FileLock` (10s timeout). `atomic_write()` uses mkstemp+fsync+os.replace for SD card durability. | -| `mind.py` | Cognitive loop daemon. Three-layer: awareness → reflection → expression. | -| `voice_loop.py` | Supervisor loop. `ALLOWED_TOOLS` whitelist (41 tools). `validate_action()` sanitizes LLM params. | -| `api.py` | FastAPI REST API, port 8420. Single worker only — not multi-worker safe. | -| `race.py` | Autonomous racing controller. | -| `claude_session.py` | Central dispatcher for all SPARK-initiated Claude interactions. | -| `spark_config.py` | Tunable constants (reflection angles, topic seeds, prompts). Primary target for self-evolution PRs. | +1. **This checkout is the running robot.** Daemons are reading these files + while you edit them, and `master` deploys on merge. +2. **A real child is the user.** Quiet mode is a dysregulation protocol, not a + preference. Location data is a child's location. Silence is sometimes the + correct output. -**Critical gotchas:** -- `update_session()` calls `ensure_session()` *before* acquiring the lock — `FileLock` is not reentrant -- `api.py` PIN rate limit store capped at 10k IPs with oldest-first eviction; `X-Forwarded-For` trusted from localhost only +--- -### os.getlogin() Under Systemd +## The invariants -`picarx.py:48` calls `os.getlogin()` in `Picarx.__init__()`. Under systemd there is no `/dev/tty` → `OSError: [Errno 6]`. Fix: `~/.local/lib/python3.11/site-packages/usercustomize.py` wraps `os.getlogin()` with fallback to `LOGNAME`/`USER`. **Do not remove** — affects all 14+ GPIO scripts. - -### Bin Scripts - -- **`px-*`** — User-facing helpers. Source `bin/px-env`, delegate to `tool-*` or run embedded Python heredoc via `/usr/bin/python3`. -- **`tool-*`** — Low-level tool wrappers invoked by the voice loop. Must emit a single JSON object to stdout. Motion tools gated by `confirm_motion_allowed` in session state. - -### Voice Loop - -Three backends, same `pxh.voice_loop` core: - -| Launcher | Backend | System prompt | -|---|---|---| -| `bin/run-voice-loop` | Codex CLI | `docs/prompts/codex-voice-system.md` | -| `bin/run-voice-loop-claude` | `bin/claude-voice-bridge` | `docs/prompts/claude-voice-system.md` | -| `bin/run-voice-loop-ollama` | `bin/codex-ollama` | `docs/prompts/codex-voice-system.md` | - -Loop: wait for `listening: true` → build prompt (system + session + transcript + thoughts) → call LLM subprocess → parse last JSON `{tool, params}` → `validate_action()` → `execute_tool()` → update session. Override via `CODEX_CHAT_CMD`. - -**Conversation buffer**: each turn is appended to `state/conversation-{persona}.jsonl` (rolling window, `PX_CONVERSATION_TURNS`, default 10) and injected back into the next prompt as a "Recent conversation" section — gives SPARK short-term memory across turns without relying solely on file-injected session state. Per-persona file so GREMLIN/VIXEN/Spark histories never bleed. SPARK's utterance is the action's `params.text`, falling back to `(tool_name)` for non-speech actions. - -### Wake Word System - -```bash -bin/run-wake [--wake-word "hey robot"] [--dry-run] -``` +### 1. Failure-First -STT priority chain: SenseVoice (primary, ~5s) → faster-whisper (best AU accent) → sherpa-onnx Zipformer → Vosk (wake word grammar only). Models gitignored, must be downloaded separately. +Observe and reproduce before you theorise. Preserve the evidence — logs, the +failing state file, the exact command — before you change anything, because +the fix destroys the reproduction. -**Capture is `arecord`, never PyAudio** (`src/pxh/mic_stream.py`). PortAudio's ALSA backend sits in a permanent overrun-recovery loop on the C-Media USB mic: opened at 44100 Hz it delivers ~29,900 samples/sec, and since the listener must pass `exception_on_overflow=False`, ~32% of every utterance is silently spliced out. There is no clipping, no zero-run and no envelope anomaly, so **every offline metric on the recorded WAV looks clean** — only listening reveals it. Do not reintroduce PyAudio. +**Verify the actual failing path**, not a path that resembles it. A green unit +test for a function does not prove the subprocess that calls it under `sudo` +works. Most bugs on this robot have been silent successes: `aplay` exits 0 and +plays nothing, `claude` runs as the wrong user and returns a fallback string, +a poll loop times out and reports success. **If a component reports success, +that is a claim to be checked, not evidence.** -`ArecordStream` mirrors `pyaudio.Stream.read/start_stream/close`, so call sites are unchanged. A reader thread drains the pipe into a bounded deque; this is load-bearing, not decoration — the listener stops reading for seconds at a time (STT, then the LLM call) and a 64 KB pipe holds only ~0.37 s, so without it arecord would block and overrun its own ALSA buffer, rebuilding the original bug. Drops are counted and logged (`dropped_chunks`), never silent. +### 2. `master` is code truth; GitHub Issues are work identity -**Regression test:** `bin/px-mic-check` — chirp-train loopback through SPARK's own speaker. Healthy: 18/18 chirps, ≤3 ms deviation, 0 drops. The broken PyAudio path scored 13/18 with the timeline compressed by seconds. Needs the mic free (`systemctl stop px-wake-listen` first). +Branch off `master`. A merge to `master` deploys to the robot and publishes the +public site. A change's identity is its issue number — not its branch, not a +plan document, not a memory file. +→ [docs/git-workflow.md](docs/git-workflow.md) -**Whisper anti-hallucination**: `temperature=0`, `condition_on_previous_text=False`, `no_speech_threshold=0.6`. Post-filters: non-ASCII dominant, phantom phrases, repetitive text → reject. +### 3. Never blanket-stage -**Critical:** `bpe_model` kwarg is **not** supported by the installed sherpa-onnx — do not add it to `load_stt_model()`. +`git add -A`, `git add .`, `git add -u`, `git commit -a` are **forbidden**. +Stage exact owned paths, then read `git diff --cached` before committing. -### Audio Pipeline +**Preserve unrelated dirty work.** The tree routinely carries someone else's +half-finished change or a daemon's runtime output. If it overlaps what you must +edit, commit it first as its own labelled commit — never absorb it. +→ [docs/git-workflow.md](docs/git-workflow.md) -Speech: `espeak --stdout` → WAV bytes → `aplay -D pulse` → PulseAudio → HifiBerry DAC → speaker. +### 4. Semantic intelligence proposes; deterministic code constrains -**Critical gotchas:** -- When scripts run as **root** (`px-perform`, `tool-voice`): must set `PULSE_SERVER=unix:/run/user/1000/pulse/native` in the aplay subprocess env. Root's `XDG_RUNTIME_DIR=/run/user/0` can't find the pi-user socket. Audio silently fails without this. -- `robot_hat.enable_speaker()` must be called before any audio (toggles GPIO 20 for MAX98357A amp). aplay exits 0 but nothing plays if skipped. -- PulseAudio holds the DAC exclusively — `aplay -D robothat` (ALSA bypass) fails "device busy". +An LLM chooses what SPARK should do. Code decides whether it may. Policy, +privacy, authority, and execution limits are implemented as pure functions and +chokepoints, never as instructions to a model. -### Daemon Health (`src/pxh/health.py`) +Corollary: **a safety property that can only be enforced by asking a model +nicely is not enforced.** +→ [docs/architecture/policy-and-authority.md](docs/architecture/policy-and-authority.md) -Answers "is this daemon *doing its job*", which `systemctl status` cannot. Every daemon calls `record_success()` / `record_failure()`; `read_health()` aggregates. +### 5. Prompts and personas are cognition and style — never enforcement -**Store: `state/health/.json`, one file per component — never a single shared file.** `px-alive` and `px-battery-poll` run as root while everything else runs as `pi`; a shared file would need a `FileLock`, and a root-created lock at 0644 locks out every `pi` daemon with EACCES. Per-component files remove the lock, the read-modify-write race, and the ownership hazard together. +`voice_loop.py`'s persona swap **replaces** the system prompt rather than +supplementing it. Anything protective that lives only in prose vanishes the +moment GREMLIN or VIXEN is active. -**The directory is created `1777`** (sticky, world-writable, like `/tmp`) because `atomic_write()`'s `mkstemp` needs directory write permission — a root-created 0755 dir would break every `pi` writer. `_ensure_health_dir()` re-chmods on every write, so whichever user wins the creation race, both can write. Do not "tighten" this to 0755. +Never fix a safety bug by editing a prompt. Never document a safety property as +though a prompt were its mechanism. +→ [docs/architecture/policy-and-authority.md](docs/architecture/policy-and-authority.md) -**Status is derived at read time, never stored** — a dead daemon can't leave a lying "ok" behind. `ok` → `degraded` (1–2 failures) → `stale` (silent past its per-component `STALE_AFTER_S`) → `failing` (≥3 consecutive) / `missing`. Per-component windows matter: `px-blog` runs daily, `px-mind` every 60s. +### 6. Unknown fails closed; authority is least-privilege -- `record_success(..., min_interval_s=N)` throttles fast loops (px-alive ticks 2×/s — an fsync per tick would wear the SD card). **Failures never throttle**, and a failure clears the throttle so the recovery is written immediately — otherwise a flapping component accumulates failures while its successes are dropped, and reads as "failing" while working. -- Reporting never raises. Health must not be able to kill the daemon it reports on. -- px-mind publishes the aggregate to `state/health.json` and into `awareness["health"]`; `summarize()` feeds reflection context. Readers that must be correct **when px-mind is down** call `read_health()` directly, not the snapshot. -- `tests/conftest.py` has an **autouse** fixture redirecting `health_dir()` to tmp. Without it, in-process tests write mock health records into the live robot's `state/health/` — `isolated_project` is opt-in and only isolates subprocesses. +When trust or authority cannot be established, the answer is no. -**Claude spend visibility:** `token_log.log_usage()` takes a `backend` argument and splits totals under `by_backend` in `state/token_usage.json`. The top-level totals mix free Ollama with paid Claude and cannot answer "what am I spending". `call_llm()` also sets `result["backend"]` to the tier that actually served — the `backend=` reflection log line shows the *configured* primary, not the one that answered. +- A session that cannot be read is treated as quiet mode, not as "quiet mode + off". Unknown resolves the same way as known-restrictive. +- An unclassified request kind routes to the **unprivileged** session. The + default must make forgetting safe. +- Guard with an **allowlist**, not a denylist. Forgetting to add a key should + mean the data is absent, not exposed. -### Idle-Alive Daemon +The one deliberate exception is documented and narrow: an unreadable +`awareness.json` fails *open*, because it is written by a daemon that is +routinely down and failing closed there would mute SPARK indefinitely. +→ [docs/architecture/policy-and-authority.md](docs/architecture/policy-and-authority.md), +[docs/architecture/privacy.md](docs/architecture/privacy.md) -Keeps robot alive when idle. Holds a **persistent Picarx handle** — do not refactor to create/destroy per-action (`reset_mcu` leaks GPIO5 and `close()` doesn't release it). +### 7. Runtime state is not source code -**Readiness vs. liveness**: the unit is `Type=notify`. `WatchdogSec=15` only arms after the daemon sends `READY=1`, which it does from `notify_ready()` at the first state where it is actually working — holding the Picarx handle, or deliberately not holding it (on charger, in I2C backoff). Hardware acquisition therefore runs under `TimeoutStartSec=60`, because `Picarx.__init__` can block past 15s contending for I2C with a tool that just took GPIO (normal acquisition is ~6s). Pre-`READY` heartbeats also send `EXTEND_TIMEOUT_USEC`, which covers an unbounded park behind a foreign lease. **Do not add heartbeats inside initialisation instead** — that would keep the watchdog fed while wedged, blinding it to the thing it exists to catch. - -**GPIO exclusivity**: One process holds the Picarx handle. Tools call `yield_alive` (defined in `bin/px-env`) to send SIGUSR1 to px-alive; systemd restarts it after 10s. Long-running owners hold and refresh the tokenized `state/gpio_lease.json` authority while using hardware. `state/exploring.json` describes wander intent/state only. - -### Wander (px-wander / pxh.wander) - -`bin/px-wander` is a thin bash wrapper (yield_alive + calibration guard) around `src/pxh/wander.py`; the engine is a module, not a script, so it can be imported and tested directly. - -**Calibrate before wandering on a new floor:** place all grayscale sensors over that surface and run `bin/px-wander --calibrate-cliff` (`--accumulate` keeps the darkest floor across spots). The launcher self-elevates for GPIO and writes `exploring.json` before yielding px-alive; do not replace it with a direct Python invocation. The ADC power-on latch is rejected — including a *partially* latched read — so calibration fails closed until live sensor values appear. - -**The cliff guard is deliberately layered**, because motor noise tripped every early live run: median-of-3 sampling, confirmation by persistence rather than one stationary read, a stationary re-read to confirm an in-motion trip, sonar echo-timeout retries before counting a sensor failure, and board-gap-vs-drop discrimination by *width*, not depth. Do not simplify any one of these away — each was added after a specific live failure. - -**GPIO**: every live wander writes `exploring.json` *before* constructing Picarx and runs a 20s `_ExploringRefresher` thread for the whole run — px-alive ignores the file once its mtime is >60s old, so a single start-of-run write only protects the first minute. `wander.py` acquires a `GpioLeaseGuard` and **exports `PX_GPIO_LEASE_ID`**, which is how `tool-describe-scene` and `tool-announce` borrow the lease instead of aborting. Probe-turn arc recovery reverses with the SAME steer angle as the probe (bicycle model — mirrored steer doubles the heading change instead of undoing it). - -**Vision timeouts are a strict ordering, not three independent numbers:** `wander.DESCRIBE_SCENE_TIMEOUT` (150s) must outlive `tool-describe-scene`'s whole run — its 45s Claude call plus photo capture plus its **bounded** 60s tool-voice step. `tool-voice` blocks indefinitely when another process holds the audio device, so that bound is what stops wander killing the tool mid-run. The relationship is pinned by `test_describe_scene_timeout_has_margin_over_claude`, which reads the tool's real constant rather than a literal. - -### Cognitive Loop (px-mind) - -```bash -bin/px-mind [--awareness-interval 30] [--dry-run] -``` - -Three-layer architecture: -- **Layer 1 — Awareness** (every 60s, no LLM): sonar + session + calendar + Frigate → `state/awareness.json` -- **Layer 2 — Reflection** (on transition or every 5min idle): all personas use Ollama on M5 as primary (`http://M5.local:11434` — the UDR7 stopped serving the bare `M5` hostname; verified live 2026-08-15, `getent hosts M5` returns nothing while `M5.local` resolves to 192.168.0.249. A bare `M5` makes tier 1 fail instantly and silently spends money on tier 2). Four-tier fallback: Ollama M5 → Claude Haiku (SPARK only) → Ollama Cloud → Pi localhost (opt-in, off by default — Pi 4 OOM risk). Writes to `state/thoughts.jsonl`. **Tier 2 asks the resident brain first** (`mind.call_claude` → `call_brain_reflection`, kind `reflection`) and only shells out to `claude -p` when `ask_brain` returns None — warm context instead of a cold process per thought, and metered. The session is told, in `docs/prompts/spark-brain-system.md`, that a `reflection` turn is answered by *returning* the thought rather than acting on it; the caller dispatches the `action` field itself, so a session that speaks during reflection makes it happen twice. -- **Layer 3 — Expression** (30min cooldown; `greet_arrival` bypasses it on a real arrival, 120s anti-flap): dispatches to tool-voice/tool-look/tool-remember and cognitive tools. Valid actions include (wait, greet, greet_arrival, comment, remember, look_at, weather_comment, scan, play_sound, photograph, emote, look_around, time_check, calendar_check, introspect, evolve, morning_fact, research, compose, self_debug, blog_essay, message_obi, set_goal, update_goal, complete_goal). Suppressed during school, quiet time, bedtime (all calendar-driven). **Hardcoded night silence: 19:00–07:00 Hobart time — no speech/audio/motion. Silent cognitive actions (`NIGHT_ALLOWED_ACTIONS`: wait, remember, research, compose, introspect, self_debug, set_goal, update_goal, complete_goal) are exempt and run overnight.** -- **`message_obi` action**: SPARK initiates a direct message to Obi via the dashboard. Exponential backoff: starts at 10min, doubles on unanswered nudge, caps at 4h, resets when Obi replies. Respects all suppressors. Thoughts with `action=message_obi` are **redacted** in `thoughts-spark.jsonl` (written as `[private message to Obi]`) so the private DM content never reaches the public `/api/v1/public/thoughts` endpoint. -- **Memory consolidation**: nightly Haiku pass (02:00–06:00 Hobart, ≤2 attempts/day, state/consolidation_meta.json) distills the last 24h of thoughts into state/memories-spark.jsonl; reflection retrieves the top-3 relevant memories by keyword/tag overlap. Goal persistence in state/intention-spark.json (7-day expiry, one active at a time). - -**Critical gotchas:** -- All time-of-day logic uses `ZoneInfo("Australia/Hobart")` — never hardcoded UTC offsets -- Battery emergency shutdown at ≤10% (speaks warning → `sudo shutdown -h now`) -- **Charging detection (`pxh/battery_trend.py`) cannot use adjacent polls.** The pack gains ~0.004V per 30s poll while readings swing up to 0.17V, so differencing measures noise — that bug read `charging: false` through a whole afternoon on the charger. Most of the swing is px-alive's servo load dragging the rail, and load only pulls *down*, so a rolling max recovers resting voltage before a least-squares slope over the window. Thresholds are bootstrapped from a measured trace (0.6% false-charging, 85% detection), deliberately skewed because a false `charging` **suppresses the emergency shutdown**. Detection costs ~10 min, so the plug-in chime lags. Re-tune against a fresh measured trace, never against intuition. -- Single-instance PID guard via `/proc/{pid}` liveness check -- Arrival detection uses module-level `_last_known_findmyhub` cache (not awareness snapshot) — survives M5.local→Pi push outages. Do not replace with snapshot diff. -- `state/thought-images/` cleaned hourly (images >30 days deleted) - -### Epistemic Provenance (`src/pxh/provenance.py`) - -Every durable claim in `state/notes[-persona].jsonl` and `state/memories-{persona}.jsonl` records where it came from, so retrieved memory can distinguish what SPARK saw, was told, inferred, or wrote itself. - -The six kinds have confidence ceilings clamped on write and read: `observation` and `verification` (1.0), `report` (0.9), `inference` (0.6), `narrative` (0.5), and legacy `unknown` (0.3). The ordering is the safety property. The model never chooses a kind: callers set constants, and consolidation allowlists its input fields. Ceilings deliberately live outside `spark_config.py`, which self-evolution can propose editing. - -Writes are strict; reads are lenient. Invalid or legacy data remains readable as `unknown`, without promoting a coarse `source` string into a claim type. Corrections mark supersession without deleting history. Relevance retrieval returns only topical matches (never recent padding); explicit `mode="recent"` remains available. A populated store with no relevant hit does not fall back to raw notes. - -### Autonomous Racing (px-race) - -```bash -bin/px-race --calibrate # sensor calibration -bin/px-race --map # practice lap (builds track profile) -bin/px-race --race --laps 5 -bin/px-race --dry-run --map -``` +`state/` is the robot's living state and is almost entirely untracked. **Never +commit a runtime state file.** Data rewritten every loop belongs on tmpfs, not +the SD card. Durable writes go through `atomic_write()`. +→ [docs/operations/state-and-runtime.md](docs/operations/state-and-runtime.md) -Two-phase: Phase 1 builds track segment profile; Phase 2 uses it to maximize speed. Dual-sensor: grayscale (primary edge avoidance, <1ms) + sonar (obstacle/centering, ~30ms). No LLM/network/audio in the race loop. +### 8. Tests are hermetic by default -**PD sign convention**: `pd_edge` uses `Kp=−20.0` (negative Kp) so positive error (drift right) → negative steer (left correction). The spec states `Kp=20` but the code is correct for the error convention used. Unit tests use `kp=20.0` generically — that's fine. +The suite runs on the robot it controls. Unless explicitly marked `live`, a +test must not read or write live state, make a billed call, reach the network, +or touch hardware. A test that samples the live robot is not flaky — it is +wrong, and it changes the thing it measures. +→ [docs/testing.md](docs/testing.md) -Safety (priority): E-stop (sonar < threshold) → edge guard → obstacle dodge → I2C failure (3 errors → brake) → stuck detect (2s no movement → reverse) → timeout → battery. +### 9. Targeted green is not repository green -`state/race_live.json` written every ~0.5s for dashboard integration. +Running `-k` on what you touched proves your change, not the repository. Run +the full suite before claiming done. -### Social Posting (px-post) +And a green suite proves nothing about live hardware, the resident tmux +sessions, trust boundaries, or anything behind `sudo`. **Those require explicit +live proof** — a real run, with its output quoted. Never infer them. +→ [docs/testing.md](docs/testing.md) -Watches `state/thoughts-spark.jsonl` (salience ≥0.7 or spoken action), runs Claude QA gate, posts to `state/feed.json` and Bluesky. "Ambiguous" QA responses (e.g. "Maybe") default to pass — QA is a safety net, not a quality bar. +### 10. Historical documents are evidence, not truth -**Privacy:** `message_obi` thoughts are redacted before being written to `thoughts-spark.jsonl` (the thought text is replaced with `[private message to Obi]`), so private DMs never reach social posting or the public thoughts endpoint. +`docs/superpowers/specs/`, `docs/superpowers/plans/`, `docs/historical/`, PR +bodies, and old session notes record what was decided and why, on the date in +the filename. None of it is maintained. **Do not implement from them and do not +cite them as current behaviour.** +→ [docs/superpowers/README.md](docs/superpowers/README.md) -### Claude Session Manager +--- -| Session Type | Model | Cooldown | Daily Quota | -|---|---|---|---| -| `evolve` | Opus | 24h | 1/day | -| `self_debug` | Sonnet | 6h | 2/day | -| `research` | Haiku | 2h | 3/day | -| `compose` | Haiku | 4h | 2/day | -| `conversation` | Sonnet | 15min | 4/day | -| `blog` | Haiku | 30min | 5/day | -| `consolidate` | Haiku | 20h | 1/day | +## Where current truth lives -Global: 30min cooldown between sessions (except `self_debug`/`blog`), 8/day cap. When ≤2 remaining: only `self_debug`/`evolve` allowed. Bypass: `PX_CLAUDE_BUDGET_DISABLED=1`. Session log: `state/claude_sessions.jsonl`. - -### The Brain — persistent Claude session (`src/pxh/brain.py`) - -**SPARK's Claude calls are migrating off `claude -p` onto a resident interactive Claude Code session.** This is a settled decision, not a tradeoff to re-argue: a one-shot subprocess throws away context on every call and cannot use SPARK's own tools. `bin/px-claude-session` is the session; `src/pxh/tmux_claude.py` drives it in tmux; `src/pxh/brain.py` is the request/reply channel. - -**Replies come back through the filesystem, never the pane.** `capture-pane` returns *rendered* terminal output — wrapping, spinners, ANSI escapes, a finite scrollback — so an answer scraped from it is at the mercy of the terminal. The session answers by running a tool instead. Pane for humans, filesystem for machines. - -Mailbox at `state/brain//`: `inbox/.json` (request) → `outbox/.json` (reply, written by `bin/tool-brain-reply`) → `dead/` (swept on session recreate), plus `current.json` (the in-flight request — what wedge detection keys on) and `validation.json` (proof a real handshake landed — what readiness means now). - -**Readiness is a proven round trip, never the prompt glyph.** The glyph renders identically for a session that is actually listening and for one sitting behind a permission dialog it cannot answer — that collapse is the bug this file used to document as the design. `bin/px-brain` sends one real request through `tool-brain-reply` and requires one real reply echoing a nonce, recording the outcome in `validation.json`. `brain.session_state()` derives one of four strings from that marker at read time, never stored: `validated` (a real round trip landed on the model the marker records — noticing that the *configured* model has since changed is `handshake_reason`'s separate job, and is what triggers a re-handshake), `validating` (a handshake is in flight, or aged out if it's been too long), `no_marker` (the session is up but has never proven it can answer, or its marker just expired), `session_absent` (tmux has no such session). `ask_brain()` only proceeds on `validated`. `bin/px-brain-status` prints all four states plus the model and marker age in one command — start there before attaching to a pane. The supervisor itself is guarded by an `fcntl` flock (`state/brain/.supervisor.lock`) so a second copy started by hand refuses to run rather than racing the systemd-managed one for the same sessions. - -**Two sessions, and the split is a trust boundary, not load balancing.** `spark-brain` runs at the repo root with SPARK's tools. `spark-io` handles text SPARK did not write (`post_qa`, `public_chat`, `obi_chat` — see `_IO_KINDS`) from a cwd *outside* the repository with exactly one tool, `tool-brain-reply`. **A new kind that handles untrusted input must be added to `_IO_KINDS`** — the default is the privileged session, so forgetting is the dangerous direction. - -**Critical gotchas:** -- **`ask_brain()` returns `None` on every failure and never raises.** None means "fall back" — callers drop to the Ollama tiers exactly as they do today when Claude is unreachable. There is deliberately no exception path; this sits under daemons. -- **Single-flight `FileLock` per session.** Two concurrent `send-keys` runs do not queue, they interleave into one garbled prompt — the failure mode is not "slow" but "both answers wrong". A caller that can't get the lock in `LOCK_WAIT_S` falls back rather than queueing. -- **Mailbox directories are `1777`, and the lock file `0666`** — same reasoning as `state/health/`, and it is load-bearing for the same reason: SPARK's daemons do not all run as the same user, and a root-created 0755 dir locks every `pi` daemon out of `atomic_write`'s `mkstemp`. Do not tighten either. -- **The glyph never proves a session can answer.** `run_handshake` does not gate on `pane_ready()` at all — the handshake's real reply-with-nonce is itself the authoritative readiness test, so checking the glyph first would only add a redundant, misleading gate (a permission dialog renders it too). `handshake_reason` is different: inside the bounded window right after a recycle it *does* consult `_is_idle` (which ends in `pane_ready`), because in that window the supervisor already knows a real turn — the recycle's own journal-append-then-`/clear` — is in flight, and the glyph is what tells it that turn has finished. Injecting mid-turn splices two prompts into one and produces a plausible-looking wrong answer. -- **There is exactly one spelling of `tool-brain-reply`, and it is absolute.** Claude Code matches a `Bash(...)` allowlist rule against the command by *prefix*, so `Bash($PROJECT_ROOT/bin/tool-brain-reply:*)` admits an absolute invocation and nothing else — a bare or repo-relative spelling misses it and raises a permission dialog nobody is attached to answer, which is a wedge. Relative also cannot work for the io session, whose cwd is outside the repo. `brain.TOOL_BRAIN_REPLY` is the constant; the nudge and both allowlists use it, and both system prompts carry a `{{TOOL_BRAIN_REPLY}}` placeholder that `bin/px-claude-session` substitutes at launch. **Never write a literal `tool-brain-reply` into a prompt** — pinned by `test_launcher_renders_one_absolute_reply_spelling`. -- **`tool-brain-reply` validates everything** — bare-uuid4 id (it becomes a filename), the id must name a *pending* request (otherwise a valid uuid is a write primitive aimed at the outbox), JSON payload under `MAX_REPLY_BYTES`. It is reachable from the untrusted io session. -- **`ask_brain` meters every request** (`state/brain/meter.json`, per kind per day). It is the first chokepoint every Claude request passes through. Reflection reaches it via `ask_brain` without going through `claude_session.py`'s per-type cooldowns — deliberately, since reflection runs every 5 min and a daily cap would simply stop it. The meter gives visibility without a cap; the `claude -p` fallback under it is still unmetered, which is the remaining hole. -- `tests/conftest.py` has an **autouse** fixture redirecting `brain_root()` to tmp. Without it an in-process test drops a real request into the running robot's inbox, where the live session answers it. - -**`px-brain` supervisor (`bin/px-brain`, `src/pxh/brain_daemon.py`):** owns both sessions so callers don't have to. **Its first job is holding a read-only attached tmux client per session** — 3.3a's `send-keys` fails outright when no client is attached, so without the holder injection fails precisely when nobody is watching. `TERM` must be set in the unit (`tmux attach` refuses without one). `KillMode=process` is deliberate: restarting the supervisor must not kill the sessions it supervises. It also sweeps pending requests to `dead/` on session (re)create, unwedges (Escape, then kill after `ESCAPE_GRACE_S`), and recycles context on turn count + nightly at 02:00 Hobart — **always at an idle moment**, since a `/clear` between nudge and reply loses the request. Wedge detection keys on `current.json`, never on stale inbox files (an abandoned inbox entry means a caller gave up, not that the session is stuck). - -**Rollout:** `PX_BRAIN_KINDS` (default `research,compose,post_qa,reflection`) selects which kinds route to the brain; everything else still takes the old path. Read at call time so the rollout can be widened or rolled back live — `bin/px-post` consults the same dial for its QA gate. `evolve` cannot move until the brain can work inside a git worktree: a resident session's tool envelope is fixed at launch and cannot be widened per call. Remaining `claude -p` call sites: `mind.py` (`call_claude_haiku` — now the *fallback* under the brain, not the primary), `api.py` (`_call_claude_public`), `bin/claude-voice-bridge`, `bin/px-blog`, `bin/px-post` (legacy branch), `bin/tool-describe-scene`, `bin/px-cron-say`. Design: `docs/superpowers/specs/2026-08-01-px-brain-design.md`. - -**`bin/tool-describe-scene` may be a bug fix, not just a migration.** `bin/tool-wander:64` runs `px-wander` under `sudo -n`, and `wander._call_describe_scene` passes that environment straight down — so the tool's `claude -p` runs **as root**, with root's `HOME`. If root has no Claude credentials there, vision silently returns `FALLBACK_DESCRIPTION` on every real wander and nothing logs a credential error. The sudo chain is verified in the code; **the credential failure itself has not been confirmed on the robot** — check before claiming it fixed. Under the brain the root process only drops a JSON file and the authenticated `claude` runs as `pi`, which sidesteps it either way. - -### Self-Evolution (px-evolve) - -SPARK proposes code changes via GitHub PR. Human approval required — changes never auto-apply. - -**Safety constraints:** -- **Whitelist**: `src/pxh/spark_config.py`, `src/pxh/mind.py`, `src/pxh/voice_loop.py`, `bin/tool-*` (new only), `tests/`, `docs/prompts/` -- **Blacklist**: `docs/prompts/persona-*`, `api.py`, `bin/tool-chat*`, `bin/px-evolve`, `.env`, `systemd/` -- Max 3 files changed; pytest must pass; 30min Claude timeout; PR gated on file whitelist check - -### Blog (px-blog) - -Scheduled writer (daily/weekly/monthly/essay) + voice-triggered (`tool-blog`). Posts to `state/blog.json` envelope, served at `GET /api/v1/public/blog`. OG meta rewriting via `site/workers/og-rewrite.js` (same Cloudflare Worker pattern as `/thought/*`). - -### Home Assistant Integration - -Custom conversation component at `ha/custom_components/spark_conversation/` routes Nest Mini/Hub Max voice commands through `POST /api/v1/public/chat`. - -**HA 2026.x quirks:** `supported_languages` must be a `@property`; config entries require `created_at`, `modified_at`, `discovery_keys`, `subentries`; use `AddConfigEntryEntitiesCallback` not `AddEntitiesCallback`. - -### Location Awareness (Google Find Hub) - -Cron on M5.local (every 5min): queries three Chipolo trackers → SSH-pushes `state/findmyhub.json` to Pi. - -**Privacy rule:** Location data excluded from reflection context — never appears in SPARK's thoughts or social posts. Only available in direct conversation (`where's dad?`). - -**Enforced by an allowlist, not a denylist.** `mind._REFLECTION_AWARENESS_KEYS` names the keys permitted into the reflection prompt's JSON dump; everything else is dropped. The previous denylist (`if k != "health"`) leaked raw GPS **twice** — findmyhub tracker coords and `ha_presence` per-person lat/lon, the house to 5 m — into every reflection, and thoughts feed `/api/v1/public/thoughts`, the site feed and Bluesky. Deliberately absent: `findmyhub`, `ha_presence` (presence reaches the prompt only via the coordinate-free "Who's home" prose) and `health`. **A new awareness key stays out of the prompt until someone adds it here** — that default is the whole point. Pinned by `test_reflection_prompt_excludes_all_location_coordinates` and `test_reflection_awareness_json_is_allowlisted`. - -**Arrival detection:** Uses module-level `_last_known_findmyhub` cache (not awareness snapshot diff) — survives transient push outages. - -### MCP Server - -`bin/mcp-server` exposes 5 read-only tools via FastMCP (stdio): `spark_status`, `spark_thoughts`, `spark_awareness`, `spark_sonar`, `spark_vitals`. Registered in `.mcp.json`. - -### Announce Pipeline (tool-announce + M5 relay) - -SPARK speaks through the Nest Mini/Hub Max via a two-hop chain: `bin/tool-announce` (Pi) → M5 relay (LAN) → afterwords TTS (M5 localhost) → HA media-player cast. - -**Architecture:** -- M5 relay (`m5/announce-relay/`) runs on port **7862**, fronting afterwords on `127.0.0.1:7860`. Afterwords never listens on LAN. -- `POST /announce` pre-synthesizes text to a WAV file; `GET /audio/{key}` serves it unauthed so HA can fetch by URL. -- Always address the relay by IP (`192.168.0.249`, M5-wifi's DHCP reservation — see `ANNOUNCE_RELAY_URL` in `spark_config.py`) — never `M5.local`. Nest speakers fetch the audio URL themselves and can't resolve mDNS. (M5's wired leg is pinned `.100` but its adapter is unplugged; the relay moved to `.249` on 2026-08-05. The relay's own `RELAY_PUBLIC_BASE_URL` in `~/announce-relay/.env` on M5 must match, or every audio URL it hands out points at the wrong address.) -- `data` voice only (afterwords `data` model); single target in v1 (no speaker groups → no echo). - -**Night silence:** Enforced inside `bin/tool-announce` using `NIGHT_SILENCE_START_H`/`NIGHT_SILENCE_END_H` from `spark_config` (default 19:00–07:00 Hobart time, via `ZoneInfo`). All trigger paths (voice loop, px-mind `announce` action, `message_obi` private audio) pass through the tool, so the gate is a single chokepoint — a suppressed call returns `{"status":"suppressed","reason":"night_silence"}`. The same bounds also gate the px-mind `announce` action in `mind.py` (`_is_night_silence`). Tests force the window deterministically via the `PX_NIGHT_SILENCE_START_H`/`PX_NIGHT_SILENCE_END_H` env overrides. - -**`ANNOUNCE_ENABLED` flag:** Defined in `src/pxh/spark_config.py`, **`True` since 2026-08-01** — pre-flight gates G1/G2 passed: WAV casts natively to both the Office Mini and the Hub Max, `media_content_type` pinned to `"music"`. Gates whether the autonomous paths (`_dispatch_announce` in `mind.py` → px-mind `announce` action and `message_obi` audio) fire the tool at all; a user-initiated voice-loop announce is independent of it. Check relay health first: `curl http://192.168.0.249:7862/health` from the Pi. - -**Private audio (`message_obi`):** Uses the relay's `priv/` namespace with a 3-minute TTL (vs. 7-day for public audio). The DM text itself is still redacted from `thoughts-spark.jsonl` as `[private message to Obi]`; only the audio is ephemeral on-relay. - -### Site (spark.wedd.au) - -Static site on Cloudflare Pages (auto-deploys from `master`, `site/` dir). - -Key files: -- `site/css/colors.css` — single-source 12-mood palette (CSS vars `--mood-*`). All JS uses `getComputedStyle().getPropertyValue('--mood-' + mood)` — never hardcode hex. -- `site/js/config.js` — single API base URL (`window.SPARK_CONFIG.API_BASE`). Never hardcode URLs in JS. -- `site/workers/og-rewrite.js` — intercepts `/thought/?ts=` and `/blog/?id=` to rewrite OG meta server-side (social crawlers don't execute JS). - -### REST API - -```bash -bin/px-api-server # live mode -bin/px-api-server --dry-run # FORCE_DRY -``` - -**Auth**: Bearer token (`PX_API_TOKEN`) or session token from `POST /api/v1/pin/verify` (4h TTL). Unauthenticated: `/api/v1/health` and `/api/v1/public/*`. - -- Public rate limit: 120 req/min per IP (`PublicRateLimitMiddleware`); `/api/v1/public/chat` has stricter 10 msg/10min -- `X-Forwarded-For` only trusted from `127.0.0.1`/`::1` — not from Cloudflare -- Async wander: returns 202 + `job_id`; poll via `GET /api/v1/jobs/{id}` -- Device reboot/shutdown: two-step — `POST /api/v1/device/{action}` returns nonce; confirm via `POST /api/v1/device/confirm` within 60s -- **Obi chat**: `POST /api/v1/obi-chat` (auth required) — Obi sends a message, SPARK responds using `_OBI_CHAT_SYSTEM_PROMPT`, both sides logged to `state/obi_chat.jsonl`; 10s rate gate. `GET /api/v1/obi-chat?since=` returns messages after the given timestamp. User-supplied text is sanitised via `_sanitize_chat_text()` (strips `<>`, newlines, NUL) before being stored or interpolated into prompts. - -See `src/pxh/api.py` for full endpoint list. - -### Jailbroken Chat Personas - -| Persona | Tool | Voice | Character | -|---|---|---|---| -| **GREMLIN** | `tool-chat` | `en+croak`, pitch 20, rate 180 | Temporal-displaced military AI from 2089 | -| **VIXEN** | `tool-chat-vixen` | `en+f4`, pitch 72, rate 135 | Former V-9X sexbot by Matsuda Dynamics | - -**Critical:** `think: false` is essential for Ollama — reasoning chains re-enable refusal in small models. `clean_response()` strips scaffolding dividers before voice output. - -### Systemd Services - -| Service | Script | User | Restart | -|---|---|---|---| -| `px-alive` | `bin/px-alive` | root | always, 10s (StartLimitIntervalSec=0) | -| `px-wake-listen` | `bin/px-wake-listen` | pi | always, 10s | -| `px-battery-poll` | `bin/px-battery-poll` | root | always, 10s | -| `px-mind` | `bin/px-mind` | pi | always, 10s | -| `px-brain` | `bin/px-brain` | pi | always, 10s (`KillMode=process`) | -| `px-post` | `bin/px-post` | pi | always, 30s | -| `px-api-server` | `bin/px-api-server` | pi | always, 2s | -| `px-frigate-stream` | `bin/px-frigate-stream` | pi | always, 10s | -| `px-evolve` | `bin/px-evolve` | pi | on-failure, 30s | -| `px-blog` | `bin/px-blog` | pi | on-failure, 30s | -| `px-tts-glados` | GLaDOS TTS :7861 | pi | always, 10s | -| `cloudflared` | Tunnel → spark-api.wedd.au | pi | always, 10s | - -## Safety Model - -- `PX_DRY=1` (or `--dry-run`) skips all motion and audio. **Default is live when unset.** -- `confirm_motion_allowed: false` in session state blocks motion tools regardless of dry mode -- All tools must be in `ALLOWED_TOOLS` in `voice_loop.py` -- Parameter ranges hard-validated in `validate_action()` (speed 0–60, duration 1–12s, etc.) - -### Behavioural Policy (#174) — `src/pxh/policy.py` - -Quiet mode, night silence and on-call/hot-mic suppression are *behavioural* -invariants: they hold regardless of which prompt, persona or dispatcher proposed -the action. This matters because `voice_loop.py`'s persona swap **replaces** the -system prompt rather than supplementing it, so any safety behaviour that lives -only in prose vanishes the moment GREMLIN or VIXEN is active. - -`policy.evaluate()` is the rule and only the rule — pure, no I/O, no clock, no -imports from the dispatchers, and it never executes anything. Callers classify -their own vocabulary into an `Effect` and pass the context in. - -**Three enforcement points. The third is the one that closes the hole:** - -| Site | Origin | On a blocked verdict | -|---|---|---| -| `voice_loop.validate_action()` | `interactive` | downgrade to a presence-safe substitute | -| `mind.expression()` | `autonomous` | drop the action | -| **`bin/tool-voice`** | `interactive` | `{"status":"suppressed","reason":…}`, exit 0 | - -The first two are dispatchers, so they only bind callers that go *through* a -dispatcher. `bin/tool-voice` is the sink every speech producer funnels into -(`tool-chat`, `tool-chat-vixen`, `tool-voice-persona`, `px-cron-say`, -`px-battery-poll`), and it is also what anything holding a shell reaches — -including the resident `spark-brain` session, whose tool envelope is SPARK's own -`bin/`. Before the sink gate, prose in a system prompt was the only thing -between that session and the speaker at 3am. The upstream checks stay as defence -in depth; **do not remove one because the other exists.** - -**The sink pins `origin` and `effect` rather than accepting them.** A sink -cannot know its caller — that is precisely why it needs its own gate — -`interactive` is the stricter of the two origins so a wrong guess can only ever -suppress, and a caller that could declare its own effect could declare its way -out of the gate entirely. - -**The gate sits above both the persona reroute and the `PX_DRY` branch.** -`tool-voice-persona` re-enters `tool-voice`, so a gate below the reroute would -still catch the audio — but only after an Ollama round trip on text that was -never going to be spoken. And a dry run must model the live decision, or every -dry test of a speaking route asserts behaviour the robot will not show. - -`src/pxh/policy_context.py` is the **only** loader of the session/awareness/clock -facts `policy.evaluate()` refuses to read for itself; the dispatcher and the sink -both go through it so the two cannot drift. **Its two reads have opposite failure -postures, and that is the point.** - -- **Session — fails closed.** `load_session_for_policy()` returns a - `SessionRead(data, available)`, never a bare dict, and `policy.evaluate()`'s - rule 0 suppresses audio when `available` is false. A `{}` cannot carry both - "no quiet flag set" and "no idea": quiet mode is the dysregulation protocol, - so resolving the second into the first grants permission to speak during a - meltdown on the strength of a failed file read. The earlier fail-open posture - argued a contended lock would otherwise mute SPARK under load; it bought no - such thing, since `tool-voice` calls `update_session()` on that same lock a - few lines later and dies there — pre-fix, contention produced an utterance - *and* a traceback. The `except` in the loader is deliberately broad because - failing closed cannot permit anything; every failure prints to stderr. -- **Awareness — fails open.** An unreadable snapshot yields `{}` and the - on-call/hot-mic rule goes inactive, rather than muting SPARK for as long as - px-mind is down. `awareness.json` is written by a daemon that is routinely - down; the session is written by whatever is running. Quiet mode and night - silence read nothing from this file. - -Both dispatchers still fail open on their own session read (`voice_loop.py` and -`mind.py` catch `FileLockTimeout` into `{}` — they do not use this loader for the -session). That is now backstopped rather than load-bearing: every audio action -they dispatch funnels through the sink, which re-reads and fails closed. - -Pinned by `test_direct_tool_voice_is_silent_while_the_session_lock_is_held` and -`test_direct_tool_voice_is_silent_when_the_session_cannot_be_read`, which assert -against a canary player script on disk rather than against tool-voice's own -JSON — a sink that speaks and then crashes prints no self-report at all. - -Night-window bounds come from `spark_config.night_silence_bounds()`, which -honours `PX_NIGHT_SILENCE_START_H`/`_END_H`. That seam is load-bearing for the -suite: without it every subprocess test of a speaking tool would pass by day and -return `suppressed` after 19:00 Hobart. `tests/conftest.py` pins the window shut -(`START=99`) for `isolated_project`; tests that mean to exercise night silence -override both values. - -**Still ungated, deliberately and on the record:** `bin/tool-play-sound`, -`bin/px-perform`, `px-wake-listen`'s chimes, `wander._speak()`, -`mind._play_alarm_beeps()`, and `px-battery-poll`'s plug/unplug sweep. -`tool-announce` self-gates at its own relay chokepoint, and since it now reads -`policy.is_night_hour()` the Nest path and the onboard speaker cannot disagree -about when night is — but it still enforces night silence *only*, not quiet mode -or on-call. -All of these are inventoried in `tests/test_policy_invariants.py::AUDIO_PRODUCERS` -with a disposition each, and **a new file that reaches `aplay`/`espeak`/a TTS -endpoint fails `test_every_audio_producer_is_inventoried` until someone -classifies it.** That test is the tripwire against the next silent bypass, not a -formality — a `delegates` claim is re-verified against the file rather than -trusted. - -`src/pxh/policy.py` and `tests/test_policy_invariants.py` are blacklisted from -px-evolve (see `claude_session.BLACKLIST_FILES`). Evolvable policy coverage -lives in `tests/test_policy.py` — keep that split. - -## Security - -- PIN verify returns session tokens (4h TTL) — raw Bearer token never exposed to browser -- Per-IP PIN lockout (`state/pin_lockout.json`): 3 failures → 5min lockout, 10 → 30min. 1000-IP hard cap. -- `X-Forwarded-For` only trusted from localhost — never from external proxies -- Two-step device confirmation (nonce, 60s window) -- `_sanitize_chat_text()` (module-level in `api.py`) strips `<>`, `\n`, `\r`, NUL from all user-supplied chat text before storage or prompt interpolation — applied to both public chat history and obi-chat messages - -## Adding a New Tool - -1. Create `bin/tool-` (bash + embedded Python heredoc; see existing tools) -2. Add to `ALLOWED_TOOLS` and `TOOL_COMMANDS` in `src/pxh/voice_loop.py` -3. Add `validate_action` branch to sanitize params into env vars -4. Add to `docs/prompts/claude-voice-system.md` (and codex version) -5. Add to `docs/prompts/persona-gremlin.md` and `persona-vixen.md` -6. Add a dry-run test in `tests/test_tools.py` using the `isolated_project` fixture - -Every tool must: emit a single JSON object to stdout, support `PX_DRY=1`, handle errors as `{"status": "error", "error": "..."}`. - -## Key Environment Variables - -Non-obvious variables only — most names are self-documenting. Full list in `bin/px-env` and `.env.example`. - -| Variable | Purpose | +| Topic | Doc | +|---|---| +| System map, layering, dependency direction | [architecture/overview](docs/architecture/overview.md) | +| Resident Claude sessions, mailbox, readiness | [architecture/resident-brain](docs/architecture/resident-brain.md) | +| Behavioural policy, trust boundaries, evolution limits | [architecture/policy-and-authority](docs/architecture/policy-and-authority.md) | +| Where a claim came from, and how far to trust it | [architecture/provenance](docs/architecture/provenance.md) | +| Location, private messages, chat sanitisation | [architecture/privacy](docs/architecture/privacy.md) | +| Memories, goals, lived-experience adaptation | [architecture/memory-and-learning](docs/architecture/memory-and-learning.md) | +| GPIO exclusivity, px-alive, leases | [hardware/gpio-and-alive](docs/hardware/gpio-and-alive.md) | +| Cliff guard, exploration safety | [hardware/wander-safety](docs/hardware/wander-safety.md) | +| Capture, playback, TTS, the mic | [hardware/audio-and-mic](docs/hardware/audio-and-mic.md) | +| Battery, charge detection, emergency shutdown | [hardware/power](docs/hardware/power.md) | +| State classes, atomic writes, locks, tmpfs | [operations/state-and-runtime](docs/operations/state-and-runtime.md) | +| Daemon health reporting | [operations/health](docs/operations/health.md) | +| LLM tiers, budgets, metering, spend | [operations/llm-routing](docs/operations/llm-routing.md) | +| Services, install, API, site, relay | [operations/deployment](docs/operations/deployment.md) | +| Isolation, live tests, structural tripwires | [testing](docs/testing.md) | +| Branching, staging, commits | [git-workflow](docs/git-workflow.md) | + +Per-script and per-module reference: [docs/SCRIPTS.md](docs/SCRIPTS.md). +Written for Obi: [docs/how-sparks-brain-works.md](docs/how-sparks-brain-works.md). + +--- + +## Read this before touching that + +| If you are changing… | Read first | |---|---| -| `PX_DRY` | `1` = dry-run. **Default is live when unset.** | -| `PX_BYPASS_SUDO` | `1` = skip sudo (tests only) | -| `PX_MIND_BACKEND` | `auto` (SPARK→Claude, others→Ollama), `claude`, or `ollama` | -| `PX_MIND_LOCAL_OLLAMA` | `1` = enable local Pi Ollama fallback (off by default — OOM risk) | -| `PX_CLAUDE_BUDGET_DISABLED` | `1` = bypass all session rate limits | -| `PX_CLAUDE_MODEL_*` | Per-session-type model overrides (e.g. `PX_CLAUDE_MODEL_EVOLVE`) | -| `PX_EVOLVE_DRY` | `1` = skip worktree/PR (queue entry still written with `dry: true`) | -| `PX_POST_QA` | `0` = skip Claude QA gate (testing) | -| `PX_HA_DEBUG` | `1` = verbose HA fetch logging | -| `PX_HOME_LAT` / `PX_HOME_LON` | Home coords for Find Hub at-home detection (defaults: `-43.13567`, `147.11840`) | -| `OLLAMA_CLOUD_API_KEY` | Enables Tier 3 Ollama Cloud fallback in px-mind | -| `PX_VOICE_LOCK_TIMEOUT` | Voice output lock timeout in seconds (default: 30) | - -## Multi-Model QA +| anything that can produce sound | [policy-and-authority](docs/architecture/policy-and-authority.md) — a new audio producer fails the suite until it is classified | +| a system prompt or persona | [policy-and-authority](docs/architecture/policy-and-authority.md) §prompts, and [resident-brain](docs/architecture/resident-brain.md) §editing | +| anything reading `awareness` | [privacy](docs/architecture/privacy.md) — a new key stays out of the reflection prompt until allowlisted | +| a durable write to notes/memories | [provenance](docs/architecture/provenance.md) — kinds and ceilings | +| anything that opens `Picarx` | [gpio-and-alive](docs/hardware/gpio-and-alive.md) — yield or lease first | +| the cliff guard | [wander-safety](docs/hardware/wander-safety.md) — every layer earned its place | +| microphone capture | [audio-and-mic](docs/hardware/audio-and-mic.md) — never PyAudio | +| charge detection thresholds | [power](docs/hardware/power.md) — re-tune only against a measured trace | +| a file written every loop | [state-and-runtime](docs/operations/state-and-runtime.md) — tmpfs, not the SD card | +| `conftest.py` or an autouse fixture | [testing](docs/testing.md) — each one exists because a run changed the robot | +| a `bin/tool-*` | the six-step checklist below | + +### Adding a tool + +1. `bin/tool-` — bash + embedded Python heredoc, following the neighbours +2. `ALLOWED_TOOLS` **and** `TOOL_COMMANDS` in `src/pxh/voice_loop.py` +3. a `validate_action()` branch that hard-validates params into env vars +4. `docs/prompts/claude-voice-system.md` and the codex version +5. `docs/prompts/persona-gremlin.md` and `persona-vixen.md` +6. a dry-run test in `tests/test_tools.py` using `isolated_project` + +Every tool must emit a **single JSON object** to stdout, honour `PX_DRY=1`, and +report errors as `{"status": "error", "error": "..."}`. + +--- + +## Working here ```bash -# Run in parallel via run_in_background; synthesise results - -hermes -z "QA prompt" 2>&1 -agy --dangerously-skip-permissions --add-dir /Users/adrian/repos/spark --print-timeout 10m --print "QA prompt" 2>&1 -gemini -p "QA prompt" 2>&1 -echo "QA prompt" | codex exec --full-auto - 2>&1 +source .venv/bin/activate # development and tests +cp state/session.template.json state/session.json # first use only +python -m pytest # the real gate; run it in full +python -m pytest -m "not live" # skip hardware +bin/px-brain-status # before touching the resident sessions +bin/px-diagnostics --no-motion --short # quick health check ``` -**`agy --print` takes the prompt as its value, not as a trailing argument.** The -old spelling here put `--print` first and the prompt last, so `--print` consumed -`--dangerously-skip-permissions` as its value and the prompt was never read — -agy answered a question about the flag and exited 0. A QA run that returns -cleanly having reviewed nothing is the dangerous failure: it looks like a pass. -Keep `--print` last. Its default timeout is 5m, short for a whole-diff review. +**`bin/` scripts run under `/usr/bin/python3`**, not the venv — `picarx` and +`robot_hat` live in system site-packages. `bin/px-wake-listen` is the exception. + +**Safety defaults you must know:** `PX_DRY=1` skips motion and audio, and +**the default is live when it is unset.** `confirm_motion_allowed: false` in +session state blocks motion tools regardless of dry mode. `PX_BYPASS_SUDO=1` is +for tests only. -Narrow prompts for agy — it does better with a named file list and a ranked -list of what to look for than with "review this branch". +Full environment variable list: `bin/px-env` and `.env.example`. Note that +`.env` is loaded by `bin/px-mind` and some daemons, **not** by `bin/px-env` — +a tool run standalone may silently lack its secrets. diff --git a/README.md b/README.md index 1c99014d..39506a37 100644 --- a/README.md +++ b/README.md @@ -36,6 +36,10 @@ bin/px-spark [--dry-run] [--input-mode voice|text] ## Architecture +> This section is an orientation sketch. The **canonical** architecture docs — +> and the invariants that must not be broken — are +> [CLAUDE.md](CLAUDE.md) and [docs/architecture/overview.md](docs/architecture/overview.md). + ``` ┌─────────────────────────────────────────────┐ │ Voice Backends │ @@ -481,7 +485,7 @@ source .venv/bin/activate # 4. Dry-run a tool to verify the setup PX_DRY=1 bin/tool-status -# 5. Run tests (1070 dry-run, no hardware needed) +# 5. Run tests (~1460 collected; use -m 'not live' to skip hardware) python -m pytest tests/ -m "not live" # 6. Launch SPARK (Claude voice companion) diff --git a/docs/architecture/memory-and-learning.md b/docs/architecture/memory-and-learning.md new file mode 100644 index 00000000..73db3c6a --- /dev/null +++ b/docs/architecture/memory-and-learning.md @@ -0,0 +1,84 @@ +# Memory and Learning + +**Owns:** what SPARK keeps between turns and between days — +`src/pxh/memory.py`, `src/pxh/intention.py`, +`src/pxh/contextual_preference.py`, and the conversation buffer. + +Where those records get their trust from is +[architecture/provenance](provenance.md). + +--- + +## Invariant + +### Four stores, four different lifetimes + +| Store | File | Lifetime | +|---|---|---| +| Conversation buffer | `state/conversation-{persona}.jsonl` | rolling window, `PX_CONVERSATION_TURNS` (default 10) | +| Notes | `state/notes[-persona].jsonl` | append-only, durable | +| Consolidated memories | `state/memories-{persona}.jsonl` | append-only, distilled nightly | +| Active goal | `state/intention-{persona}.json` | one at a time, 7-day expiry | +| Contextual experience | persona-scoped experience store | append-only, never rewritten | + +**Every store is persona-scoped** so GREMLIN, VIXEN and SPARK histories never +bleed into each other. A shared store would let a jailbroken persona's output +re-enter SPARK's cognition as if SPARK had thought it. + +### The conversation buffer is short-term memory, and it is separate from state + +Each turn appends to the buffer and is injected into the next prompt as a +"Recent conversation" section. This is what gives SPARK continuity across turns +without depending solely on file-injected session state. SPARK's own utterance +is the action's `params.text`, falling back to `(tool_name)` for non-speech +actions. + +### Consolidation is nightly, capped, and metered + +A Haiku pass between 02:00–06:00 Hobart, at most 2 attempts per day +(`state/consolidation_meta.json`), distils the last 24h of thoughts into +`state/memories-{persona}.jsonl`. + +**Consolidation allowlists its input fields.** It does not copy arbitrary +thought keys forward, which is what stops a location-bearing field surviving +into a store that reflection reads. See +[architecture/privacy](privacy.md). + +### Retrieval is by relevance, and never pads + +Reflection retrieves the top-3 relevant memories by keyword/tag overlap. A +populated store with no relevant hit returns nothing rather than falling back +to recency or to raw notes — see +[architecture/provenance](provenance.md). + +### Goals are singular and expire + +`intention.py` holds one active goal per persona, with a 7-day expiry. +`set_goal` archives any previous active goal rather than overwriting it. A +robot with five simultaneous goals has none. + +### Lived-experience adaptation is append-only and narrow + +`contextual_preference.py` records one system-attributed experience per line +and **never updates an earlier line**. Corruption is reported without +rewriting the file. Adaptation is deliberately bounded to choosing between +options SPARK already had, not to inventing new behaviour. + +`load_experiences()` reports corrupt lines rather than silently skipping them, +because a store that quietly drops what it cannot parse is a store whose size +tells you nothing. + +--- + +## Why it looks like this + +*History, not rule.* + +Persona scoping came after cross-persona bleed: SPARK retrieved a GREMLIN note +and reflected on it in SPARK's voice. + +The retrieval-never-pads rule came from an observed fixation. SPARK's thoughts +circled Obi far more than the topic seeds explained, and the cause turned out +to be retrieval bias rather than the seeds — the memory query was built almost +entirely from human-presence signals, so nearly every query matched +human-presence memories, and padding with recent records made it worse. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md new file mode 100644 index 00000000..1791fdf5 --- /dev/null +++ b/docs/architecture/overview.md @@ -0,0 +1,111 @@ +# Architecture Overview + +**Owns:** the map. What the parts are, which direction they depend on, and +where each one is documented in full. No subsystem detail lives here — this +page exists so you can find the page that has it. + +--- + +## Invariant + +### One robot, four concentric layers + +``` + hardware picarx / robot_hat / GPIO / I2C / ALSA + ^ + daemons px-alive, px-wake-listen, px-battery-poll, px-mind, + px-brain, px-post, px-api-server, px-blog, px-evolve + ^ + library src/pxh/ — the logic; importable and testable + ^ + surfaces bin/px-* (humans), bin/tool-* (LLM dispatchers), + REST API, MCP server, site +``` + +**Logic lives in `src/pxh/`, never in a `bin/` script.** `bin/` scripts are +thin: they source `bin/px-env`, resolve privileges, and call into the library +or a tool. The rule exists so behaviour can be tested in-process; a rule +implemented inside a bash heredoc can only be tested by spawning a subprocess, +and subprocess tests are the slowest and flakiest part of the suite. + +`src/pxh/wander.py` is the worked example: `bin/px-wander` is a thin wrapper +(privilege elevation, calibration guard) around an importable engine. + +### The three cognitive layers + +SPARK's autonomy is `px-mind` (`src/pxh/mind.py`), and it is three layers with +different costs and different failure postures: + +| Layer | Cadence | LLM? | Writes | +|---|---|---|---| +| 1 — Awareness | every 60s | no | `state/awareness.json` | +| 2 — Reflection | on transition, or every 5 min idle | yes | `state/thoughts.jsonl` | +| 3 — Expression | gated by cooldown + suppressors | no | dispatches a `bin/tool-*` | + +Layer 1 must never call an LLM: it is the layer that still works when the +network is down, and everything above it reads its snapshot. + +Layer 3 is **budgeted and suppressed**, not free-running: +`EXPRESSION_COOLDOWN_S` (1800s) is the minimum gap between spontaneous +utterances. `greet_arrival` bypasses that budget on a real arrival, with its +own 120s anti-flap. Expression is suppressed entirely during school, quiet +time and bedtime (all calendar-driven), and during night silence — see +[policy-and-authority](policy-and-authority.md). + +Layer 3 never touches GPIO directly. Every physical act routes through a +`bin/tool-*`, which is what makes the policy sink in +[policy-and-authority](policy-and-authority.md) a real chokepoint rather +than an advisory one. + +### Dependency direction is one-way + +`policy.py` imports neither `mind.py` nor `voice_loop.py`. Both dispatchers +import `policy.py`. The same holds for `provenance.py`, `health.py`, +`state.py`, and `runtime_paths.py` — leaf modules that many callers share. + +A new module that needs to import a dispatcher is a design smell: it means the +rule you are writing belongs in the dispatcher, or the shared part belongs in a +new leaf. + +--- + +## Where each subsystem is documented + +| Subsystem | Doc | +|---|---| +| Resident Claude session (the brain) | [architecture/resident-brain](resident-brain.md) | +| Behavioural policy, trust boundaries, self-evolution limits | [architecture/policy-and-authority](policy-and-authority.md) | +| Epistemic provenance of durable claims | [architecture/provenance](provenance.md) | +| Location, private messages, chat sanitisation | [architecture/privacy](privacy.md) | +| Memories, goals, lived-experience adaptation | [architecture/memory-and-learning](memory-and-learning.md) | +| GPIO exclusivity, px-alive, leases | [hardware/gpio-and-alive](../hardware/gpio-and-alive.md) | +| Cliff guard and wander safety | [hardware/wander-safety](../hardware/wander-safety.md) | +| Capture, playback, TTS | [hardware/audio-and-mic](../hardware/audio-and-mic.md) | +| Battery, charge detection, emergency shutdown | [hardware/power](../hardware/power.md) | +| Session state, atomic writes, tmpfs runtime | [operations/state-and-runtime](../operations/state-and-runtime.md) | +| Daemon health reporting | [operations/health](../operations/health.md) | +| LLM tiers, budgets, metering | [operations/llm-routing](../operations/llm-routing.md) | +| Services, install, tunnel | [operations/deployment](../operations/deployment.md) | +| Test isolation and hermeticity | [testing](../testing.md) | + +Per-script and per-module reference — every file in `bin/` and `src/pxh/` — +lives in [docs/SCRIPTS.md](../SCRIPTS.md). This page does not duplicate it. + +A plain-language explanation written for Obi is +[docs/how-sparks-brain-works.md](../how-sparks-brain-works.md). + +--- + +## Why it looks like this + +*History, not rule.* + +The layering was not designed up front. `px-mind` began as a single loop that +called an LLM on every tick; splitting awareness out was forced by network +outages, during which the robot went completely blind rather than merely +quiet. The "layer 1 never calls an LLM" rule is that outage written down. + +The `bin/` → `src/pxh/` extraction happened subsystem by subsystem, and +`wander.py` was the one that made the case: the cliff guard could not be +regression-tested at all while it lived inside a bash heredoc, and it was the +part of the robot most likely to drive itself off a table. diff --git a/docs/architecture/policy-and-authority.md b/docs/architecture/policy-and-authority.md new file mode 100644 index 00000000..56e6b592 --- /dev/null +++ b/docs/architecture/policy-and-authority.md @@ -0,0 +1,192 @@ +# Policy and Authority + +**Owns:** who is allowed to do what, and where that is decided. Behavioural +policy (`src/pxh/policy.py`), the trust split between Claude sessions, the +tool whitelist, and the limits on self-evolution. + +Hardware authority — which process may hold GPIO — is a different question and +lives in [hardware/gpio-and-alive](../hardware/gpio-and-alive.md). + +--- + +## Invariant + +### Intelligence proposes. Policy disposes. + +An LLM chooses *what* SPARK should do. Deterministic code decides whether it +*may*. These are separate concerns implemented in separate files, and the +second never asks the first for permission. + +Concretely: a persona prompt can say anything at all, and it still cannot make +the robot speak during quiet mode, because the rule is not in the prompt. + +### Behavioural invariants live in `policy.py`, and only there + +`policy.evaluate()` is pure — no file I/O, no clock, no subprocess, no import +of `mind.py` or `voice_loop.py`, and it never executes anything. It returns a +`PolicyVerdict`. Callers classify their own vocabulary into an `Effect` +(`audio` / `presence` / `other`) and an `Origin` (`interactive` / +`autonomous`) and pass the facts in. + +That purity is what makes it testable and what stops it accreting. A new +audio-producing tool inherits every rule by being classified `audio` at its own +call site; `policy.py` never learns a tool name. + +The rules, in order: + +| Rule | Binds | Blocks when | +|---|---|---| +| 0 — session unavailable | both origins | the session could not be read at all | +| 1 — quiet mode | both origins | `spark_quiet_mode is True` | +| 2 — night silence | interactive only | Hobart hour in the night window | +| 3 — on-call / hot mic | interactive only | `ha_context.adrian_on_call` or `adrian_mic_active` | + +The autonomous side's night rule is `mind.NIGHT_ALLOWED_ACTIONS` — the actions +that may still run during the 19:00–07:00 Hobart silence because they make no +sound and no motion: `wait`, `remember`, `research`, `compose`, `introspect`, +`self_debug`, `set_goal`, `update_goal`, `complete_goal`. **SPARK thinks +overnight; it does not speak or move.** + +Rules 2 and 3 are interactive-only *by design*: the autonomous loop enforces +its own equivalents (`NIGHT_ALLOWED_ACTIONS`, on-call suppression) in +`mind.py`, with their own tests. Duplicating them here would put one invariant +in two places that can disagree. + +### Three enforcement points, and the sink is the one that closes the hole + +| Site | Origin | On a blocked verdict | +|---|---|---| +| `voice_loop.validate_action()` | `interactive` | downgrade to a presence-safe substitute | +| `mind.expression()` | `autonomous` | drop the action | +| **`bin/tool-voice`** | `interactive` | `{"status":"suppressed","reason":…}`, exit 0 | + +The first two are dispatchers, so they bind only callers that go *through* a +dispatcher. `bin/tool-voice` is the sink every speech producer funnels into, +and it is what anything holding a shell reaches — including the resident +`spark-brain` session, whose tool envelope is SPARK's own `bin/`. + +**Do not remove one because another exists.** They are defence in depth +against different classes of caller. + +Three properties of the sink are load-bearing: + +- **It pins `origin` and `effect` rather than accepting them.** A sink cannot + know its caller — that is precisely why it needs its own gate. `interactive` + is the stricter origin, so a wrong guess can only ever suppress. A caller + that could declare its own effect could declare its way out of the gate. +- **The gate sits above both the persona reroute and the `PX_DRY` branch.** + `tool-voice-persona` re-enters `tool-voice`, so a gate below the reroute + would still catch the audio, but only after an Ollama round trip on text + that was never going to be spoken. And a dry run must model the live + decision, or every dry test of a speaking route asserts behaviour the robot + will not show. +- **Substitution cannot recurse.** A blocked verdict with + `suggest_presence_substitute` makes the caller re-evaluate its substitute at + `_depth=1`. If the rules would block at `_depth >= 1`, `evaluate()` *raises* + rather than returning — a presence-safe substitute must never itself be + `effect="audio"`, and that guarantee is mechanical rather than assumed. + +### Reading the facts: session fails closed, awareness fails open + +`src/pxh/policy_context.py` is the **only** loader of the session/awareness +facts `policy.evaluate()` refuses to read for itself. Both the dispatcher and +the sink go through it so the two cannot drift. Its two reads have deliberately +opposite postures: + +- **Session — fails closed.** `load_session_for_policy()` returns a + `SessionRead(data, available)`, never a bare dict. A `{}` cannot carry both + "no quiet flag set" and "no idea"; quiet mode is the dysregulation protocol, + so resolving the second into the first grants permission to speak during a + meltdown on the strength of a failed file read. The `except` there is broad + on purpose — failing closed cannot permit anything — and every failure + prints to stderr. +- **Awareness — fails open.** An unreadable snapshot yields `{}` and the + on-call rule goes inactive, rather than muting SPARK for as long as `px-mind` + is down. `awareness.json` is written by a daemon that is routinely down; the + session is written by whatever is running. Quiet mode and night silence read + nothing from this file. + +### Every audio producer is inventoried, or the suite fails + +`tests/test_policy_invariants.py::AUDIO_PRODUCERS` maps every file in `bin/` +and `src/pxh/` that reaches `aplay`, `espeak`, or a TTS endpoint to a +disposition: `gated`, `self-gated`, `delegates`, `ungated`, `diagnostic`, +`server`, or `mention`. + +`test_every_audio_producer_is_inventoried` discovers producers by scanning for +the audio primitives and asserts the discovered set equals the inventory. **A +new file that reaches audio fails the suite until someone classifies it.** A +`delegates` claim is re-verified against the file rather than trusted. + +Currently `ungated`, deliberately and on the record: `bin/tool-play-sound`, +`bin/px-perform`, `bin/px-wake-listen`'s chimes, `wander._speak()`, +`mind._play_alarm_beeps()`, and `px-battery-poll`'s plug/unplug tone. +`bin/tool-announce` is `self-gated` — it enforces night silence at its own +relay chokepoint via `policy.is_night_hour()`, so the Nest path and the onboard +speaker cannot disagree about when night is, but it does **not** enforce quiet +mode or on-call. + +### Trust boundaries between Claude sessions + +Two resident sessions, and the split is a trust boundary, not load balancing. +`spark-brain` runs at the repo root with SPARK's tools. `spark-io` handles text +SPARK did not write, from a cwd *outside* the repository, with exactly one tool. + +`brain._IO_KINDS` is the classification. **A new kind that handles untrusted +input must be added to it** — the default routes to the privileged session, so +forgetting is the dangerous direction. See +[architecture/resident-brain](resident-brain.md) for the mechanism. + +### The voice loop's tool whitelist + +`voice_loop.ALLOWED_TOOLS` is an allowlist, and `validate_action()` +hard-validates every parameter into a range before it becomes an env var. A +tool not in the set cannot be dispatched no matter what the model emits. + +Adding a tool is a checklist, not one edit — see +[Adding a tool](../SCRIPTS.md) and the steps in `CLAUDE.md`. + +### Self-evolution is bounded by a whitelist *and* a blacklist + +`px-evolve` opens a PR; a human merges it. Changes never auto-apply. + +`claude_session.file_in_whitelist()` checks the blacklist first, then the +whitelist. Both exist because either alone is fragile: the blacklist names +files that must stay protected even if a future whitelist pattern grows broad +enough to cover them. + +`BLACKLIST_FILES` includes `src/pxh/policy.py` and +`tests/test_policy_invariants.py` explicitly, for exactly that reason — the +constitutional layer and the test that pins it must not become evolvable by +accident. Evolvable policy coverage lives in `tests/test_policy.py`; **keep +that split.** + +`PX_EVOLVE_MAX_FILES` (default 3) caps the diff, and the branch must pass +pytest before the PR opens. + +--- + +## Why it looks like this + +*History, not rule.* + +Policy became a module (issue #174) because `voice_loop.py`'s persona swap +**replaces** the system prompt rather than supplementing it. Every safety +behaviour that lived only in prose vanished the moment GREMLIN or VIXEN was +active. Prose could not be the mechanism. + +The sink gate (#206) came later and closed a real hole: before it, the only +thing between the resident `spark-brain` session and the speaker at 3am was a +paragraph in a system prompt. The brain's tool envelope is SPARK's own `bin/`, +so it could call `bin/tool-voice` directly, and both dispatcher gates sat +upstream of that call. + +Rule 0 (fail closed on an unreadable session) replaced an earlier fail-open +posture. The argument for failing open was that a contended lock would mute +SPARK under load. It bought no such thing: `tool-voice` calls +`update_session()` on that same lock a few lines later and dies there, so +pre-fix contention produced an utterance *and* a traceback. Pinned by +`test_direct_tool_voice_is_silent_while_the_session_lock_is_held` and +`test_direct_tool_voice_is_silent_when_the_session_cannot_be_read`, which +assert against a canary player script on disk rather than against tool-voice's +own JSON — a sink that speaks and then crashes prints no self-report at all. diff --git a/docs/architecture/privacy.md b/docs/architecture/privacy.md new file mode 100644 index 00000000..c8074b7d --- /dev/null +++ b/docs/architecture/privacy.md @@ -0,0 +1,86 @@ +# Privacy + +**Owns:** what must never leave the robot, and the mechanisms that enforce it. +Location data, private messages to Obi, and untrusted chat text. + +Everything here concerns a real child and a real household. These are not +hygiene rules. + +--- + +## Invariant + +### Location never reaches reflection, and therefore never reaches anything public + +SPARK knows where people are (Google Find Hub trackers via `state/findmyhub.json`, +Home Assistant presence). That knowledge is available in **direct conversation +only** — *"where's dad?"* — and never in a thought. + +This matters because thoughts are not private: `state/thoughts-spark.jsonl` +feeds `/api/v1/public/thoughts`, the public site feed, and Bluesky. + +### The mechanism is an allowlist, not a denylist + +`mind._REFLECTION_AWARENESS_KEYS` names the awareness keys **permitted** into +the reflection prompt's JSON dump. Everything else is dropped. + +Deliberately absent, and each absence is load-bearing: + +- `findmyhub` — raw tracker coordinates +- `ha_presence` — per-person latitude/longitude +- `health` — noise, not privacy, but still excluded + +Presence reaches the prompt only through the coordinate-free *"who's home"* +prose. + +**A new awareness key stays out of the prompt until someone adds it to the +allowlist. That default is the entire point.** Pinned by +`test_reflection_prompt_excludes_all_location_coordinates` and +`test_reflection_awareness_json_is_allowlisted`. + +### Private messages to Obi are redacted before they are durable + +The `message_obi` action lets SPARK initiate a direct message to Obi via the +dashboard. Thoughts carrying `action=message_obi` are written to +`state/thoughts-spark.jsonl` as the literal string `[private message to Obi]`. + +The redaction happens **before the write**, not at the read side. A read-side +filter would mean the private text existed on disk in a file that several +public endpoints and `px-post` read, and any one of them forgetting the filter +would leak it. + +Private audio for the same action uses the announce relay's `priv/` namespace +with a 3-minute TTL, against 7 days for public audio. + +### User-supplied chat text is sanitised before storage or interpolation + +`api._sanitize_chat_text()` strips `<`, `>`, newlines, carriage returns, and +NUL from all user-supplied chat text — applied to both public chat history and +obi-chat messages — **before** it is stored or interpolated into a prompt. + +Sanitising before storage rather than before display means a later reader that +forgets to sanitise cannot resurrect the problem. + +### Text SPARK did not write goes to the unprivileged session + +`post_qa`, `public_chat`, and `obi_chat` are classified in `brain._IO_KINDS` +and route to `spark-io`, which runs outside the repository with one tool. See +[architecture/policy-and-authority](policy-and-authority.md) and +[architecture/resident-brain](resident-brain.md). + +--- + +## Why it looks like this + +*History, not rule.* + +The allowlist replaced a denylist, and the denylist leaked. It read +`if k != "health"` — which passed everything else through, including +`findmyhub` tracker coordinates and `ha_presence` per-person latitude and +longitude. The house was published to five metres, in every reflection, twice +over, into a feed that goes to Bluesky. + +The bug was not that someone forgot to add `findmyhub` to the denylist. The bug +was that a denylist makes *forgetting* the failure mode. An allowlist makes +*forgetting* mean the data is simply absent, which is the direction you want to +fail in when the data is a child's location. diff --git a/docs/architecture/provenance.md b/docs/architecture/provenance.md new file mode 100644 index 00000000..890af506 --- /dev/null +++ b/docs/architecture/provenance.md @@ -0,0 +1,105 @@ +# Epistemic Provenance + +**Owns:** `src/pxh/provenance.py` — where every durable claim came from, and +how much it is allowed to be believed. + +--- + +## Invariant + +### Every durable claim records its origin + +SPARK writes two stores that later re-enter cognition: +`state/notes[-persona].jsonl` and `state/memories-{persona}.jsonl`. Retrieved +memory must be able to distinguish what SPARK **saw**, was **told**, a model +**interpreted**, SPARK **worked out**, or SPARK simply **wrote about itself**. + +### Seven kinds, ordered by how much they may be trusted + +| Kind | Means | Ceiling | Default | Written today by | +|---|---|---|---|---| +| `observation` | direct/deterministic sensor content | 1.0 | 0.8 | direct sensor writers | +| `verification` | checked against something outside SPARK | 1.0 | 0.9 | *(no writer yet)* | +| `report` | a person or external source asserted it | 0.9 | 0.7 | voice-loop remember, research | +| `model_perception` | a model interpreted sensor evidence | 0.75 | 0.65 | wander's scene descriptions | +| `inference` | SPARK worked it out from other records | 0.6 | 0.5 | *(no writer yet)* | +| `narrative` | SPARK's own generated prose about itself | 0.5 | 0.4 | consolidation, compose, mind | +| `unknown` | provenance was never recorded | 0.3 | 0.2 | every record written pre-#170 | + +`inference` and `verification` have **no writer**. That is stated rather than +papered over: nothing in SPARK currently checks a belief against the world, and +pretending otherwise would be the exact failure this module exists to prevent. + +> `src/pxh/provenance.py`'s module docstring still opens with the words "Six +> kinds" — a stale count left over from before `model_perception` was added by +> #177. The table beneath it and `provenance.KINDS` are both correct at seven. +> Filed as [#214](https://github.com/adrianwedd/spark/issues/214). + +### Ceilings are clamped on write *and* on read + +A writer may ask for any confidence it likes; what it gets is clamped. Clamping +again on read means generated prose cannot present itself as perception no +matter how the record was produced, edited, or hand-written into the file +afterwards. + +**The ordering between the ceilings is the safety property. The exact numbers +are calibration.** Re-tuning a number is ordinary work; reordering two kinds is +a change to the guarantee. + +### The model never chooses a kind + +Callers set constants. Consolidation allowlists its input fields. There is no +path by which a model labels its own output as `observation`. + +### The ceilings deliberately do not live in `spark_config.py` + +That module is the self-evolution whitelist's primary target. A system able to +propose raising its own confidence ceilings could talk itself out of this +guarantee. See +[architecture/policy-and-authority](policy-and-authority.md). + +### Writes are strict; reads are lenient + +Invalid or legacy data stays **readable** as `unknown`, keeping whatever coarse +`source`/`type` string it had — but its epistemic kind is never *inferred* from +that string. A record saying `source: consolidation` might have come from any +of consolidation's inputs; the absence of provenance is a fact about the +record, not a puzzle to solve. + +### Correction never deletes + +A newer record names an older one in `provenance.supersedes`. +`apply_supersessions()` marks the old record `superseded_by` **on a copy**, +leaving stored history untouched, and `read_provenance()` discounts its +confidence by `SUPERSEDED_CONFIDENCE_FACTOR`. Both records stay on disk, so +SPARK can hold *"I believed X, then saw Y"* rather than silently having always +believed Y. + +**Only system code may write `supersedes`.** A model that could supersede its +own records could quietly retire inconvenient ones. + +### Retrieval returns topical matches only + +Relevance retrieval never pads with recent-but-irrelevant records. Explicit +`mode="recent"` remains available for callers that want recency. A populated +store with no relevant hit does **not** fall back to raw notes. + +--- + +## Why it looks like this + +*History, not rule.* + +Issue #170. Before this module a record carried at most a coarse `source` +string, so a speculative inner thought distilled by the nightly consolidation +pass was indistinguishable, at retrieval time, from something SPARK had +actually seen or been told. Reflection then cited its own guesses back to +itself as evidence. + +`model_perception` (#177) was added because scene descriptions from a vision +model are neither observation nor narrative: they are grounded in real sensor +evidence, but they are an interpretation of it. Filing them as `observation` +overstated them; filing them as `narrative` threw away the grounding. + +Related: [architecture/memory-and-learning](memory-and-learning.md), which +consumes these records. diff --git a/docs/architecture/resident-brain.md b/docs/architecture/resident-brain.md new file mode 100644 index 00000000..3831e7b2 --- /dev/null +++ b/docs/architecture/resident-brain.md @@ -0,0 +1,175 @@ +# The Resident Brain + +**Owns:** SPARK's persistent Claude Code sessions — `src/pxh/brain.py`, +`src/pxh/brain_daemon.py`, `src/pxh/tmux_claude.py`, `bin/px-brain`, +`bin/px-claude-session`, `bin/px-brain-status`, `bin/tool-brain-reply`. + +--- + +## Invariant + +### SPARK's Claude calls are migrating off `claude -p` onto a resident session + +This is a settled direction, not a tradeoff to re-argue. A one-shot subprocess +throws away context on every call and cannot use SPARK's own tools. + +The migration is incomplete and that is expected. `PX_BRAIN_KINDS` (default +`research,compose,post_qa,reflection`) selects which kinds route to the brain; +everything else still takes the old path. It is read at call time so the +rollout can be widened or rolled back live, and there is one dial rather than +two that can disagree — `bin/px-post` consults the same one for its QA gate. + +`evolve` cannot move until the brain can work inside a git worktree: a resident +session's tool envelope is fixed at launch and cannot be widened per call. + +Remaining `claude -p` call sites are listed in +[operations/llm-routing](../operations/llm-routing.md). + +### Replies come back through the filesystem, never the pane + +`capture-pane` returns *rendered* terminal output — wrapping, spinners, ANSI +escapes, a finite scrollback. An answer scraped from it is at the mercy of the +terminal. The session answers by running a tool instead. + +**Pane for humans, filesystem for machines.** + +Mailbox at `state/brain//`: + +| Path | Meaning | +|---|---| +| `inbox/.json` | request | +| `outbox/.json` | reply, written by `bin/tool-brain-reply` | +| `dead/` | swept on session recreate | +| `current.json` | the in-flight request — what wedge detection keys on | +| `validation.json` | proof a real handshake landed — what readiness means | + +### Readiness is a proven round trip, never the prompt glyph + +The glyph renders identically for a session that is listening and for one +sitting behind a permission dialog it cannot answer. That collapse was a real +bug, and it is why readiness is defined by evidence instead. + +`bin/px-brain` sends one real request through `tool-brain-reply` and requires +one real reply echoing a nonce, recording the outcome in `validation.json`. +`brain.session_state()` derives one of four strings from that marker **at read +time, never stored**: + +| State | Meaning | +|---|---| +| `validated` | a real round trip landed, on the model the marker records | +| `validating` | a handshake is in flight (or aged out) | +| `no_marker` | session is up but has never proven it can answer | +| `session_absent` | tmux has no such session | + +`ask_brain()` proceeds only on `validated`. Noticing that the *configured* +model has since changed is `handshake_reason`'s separate job, and is what +triggers a re-handshake. + +`bin/px-brain-status` prints all four states plus model and marker age in one +command. **Start there before attaching to a pane.** + +`run_handshake` deliberately does *not* gate on `pane_ready()` — the real +reply-with-nonce is itself the authoritative test, so a glyph check would add +only a redundant, misleading gate. `handshake_reason` is different: inside the +bounded window right after a recycle it *does* consult `_is_idle`, because +there the supervisor already knows a real turn (the recycle's own +journal-append-then-`/clear`) is in flight, and the glyph is what says that +turn has finished. Injecting mid-turn splices two prompts into one and produces +a plausible-looking wrong answer. + +### `ask_brain()` returns `None` on every failure and never raises + +`None` means "fall back" — callers drop to the Ollama tiers exactly as they do +when Claude is unreachable. There is deliberately no exception path; this sits +under daemons. + +### Single-flight lock per session + +Two concurrent `send-keys` runs do not queue, they interleave into one garbled +prompt. The failure mode is not "slow", it is "both answers wrong". A caller +that cannot get the `FileLock` within `LOCK_WAIT_S` falls back rather than +queueing. + +The supervisor itself is guarded by an `fcntl` flock +(`state/brain/.supervisor.lock`) so a second copy started by hand refuses to +run rather than racing the systemd-managed one. + +### There is exactly one spelling of `tool-brain-reply`, and it is absolute + +Claude Code matches a `Bash(...)` allowlist rule against the command by +*prefix*. `Bash($PROJECT_ROOT/bin/tool-brain-reply:*)` admits an absolute +invocation and nothing else — a bare or repo-relative spelling misses it and +raises a permission dialog nobody is attached to answer, which is a wedge. +Relative also cannot work for the io session, whose cwd is outside the repo. + +`brain.TOOL_BRAIN_REPLY` is the constant. The nudge and both allowlists use it, +and both system prompts carry a `{{TOOL_BRAIN_REPLY}}` placeholder that +`bin/px-claude-session` substitutes at launch. **Never write a literal +`tool-brain-reply` into a prompt** — pinned by +`test_launcher_renders_one_absolute_reply_spelling`. + +### `tool-brain-reply` validates everything + +It is reachable from the untrusted io session, so it checks: a bare uuid4 id +(it becomes a filename), that the id names a *pending* request (otherwise a +valid uuid is a write primitive aimed at the outbox), and a JSON payload under +`MAX_REPLY_BYTES`. + +### Mailbox directories are `1777` and the lock file `0666` + +Same reasoning as `state/health/`, and load-bearing for the same reason: +SPARK's daemons do not all run as the same user, and a root-created `0755` +directory locks every `pi` daemon out of `atomic_write`'s `mkstemp`. +**Do not tighten either.** See +[operations/state-and-runtime](../operations/state-and-runtime.md). + +### The supervisor's first job is holding an attached client + +tmux 3.3a's `send-keys` fails outright when no client is attached, so without a +read-only attached client per session, injection fails precisely when nobody is +watching. `TERM` must be set in the unit (`tmux attach` refuses without one). + +`KillMode=process` is deliberate: restarting the supervisor must not kill the +sessions it supervises. + +The supervisor also sweeps pending requests to `dead/` on session (re)create, +unwedges (Escape, then kill after `ESCAPE_GRACE_S`), and recycles context on +turn count plus nightly at 02:00 Hobart — **always at an idle moment**, since a +`/clear` between nudge and reply loses the request. Wedge detection keys on +`current.json`, never on stale inbox files: an abandoned inbox entry means a +caller gave up, not that the session is stuck. + +### Editing a system prompt requires killing the session + +`docs/prompts/spark-brain-system.md` and `spark-io-system.md` bake in at +launch. `KillMode=process` means restarting `px-brain.service` will not reload +them — the session must be killed. + +### Reflection through the brain returns the thought; it does not act + +`docs/prompts/spark-brain-system.md` tells the session that a `reflection` turn +is answered by *returning* the thought. The caller dispatches the `action` +field itself, so a session that speaks during reflection makes it happen twice. + +--- + +## Why it looks like this + +*History, not rule.* + +Readiness used to be defined as "the prompt glyph is showing", and this file +once documented that as the design. It was wrong in the one case that mattered: +a session parked behind a permission dialog renders the same glyph as an idle +one, so the supervisor cheerfully injected requests into a session that could +not answer, and the caller waited out its deadline. + +The absolute-path rule for `tool-brain-reply` came from the same class of +failure — a repo-relative spelling missed the allowlist prefix and raised a +dialog in an unattended pane. + +`px-brain.service` was, for about eleven hours, believed to be an +authentication problem. It had simply never been installed. + +Design: [2026-08-01 px-brain design](../superpowers/specs/2026-08-01-px-brain-design.md) +and [2026-08-17 handshake validation](../superpowers/specs/2026-08-17-brain-handshake-validation-design.md). +Both are decision fossils — see [the fossil banner](../superpowers/README.md). diff --git a/docs/git-workflow.md b/docs/git-workflow.md new file mode 100644 index 00000000..d630d214 --- /dev/null +++ b/docs/git-workflow.md @@ -0,0 +1,135 @@ +# Git Workflow + +**Owns:** branching, staging discipline, and how work is identified. + +--- + +## Invariant + +### `master` is trunk and the only code truth + +Branch new work off `master`. `origin` is `git@github.com:adrianwedd/spark.git`. + +**The Pi's live tree tracks `master`, and Cloudflare Pages auto-deploys +`site/` from it.** A merge to `master` is a deploy to a real robot and a +publish to a public website. There is no staging environment between them. + +### GitHub Issues are work identity + +A change's identity is its issue number, not its branch name, not a plan +document, and not a line in a memory file. Reference the issue in the commit +or PR so the reasoning stays findable after the branch is deleted. + +Branch names follow `type/short-slug` — `fix/audio-sink-policy-gate`, +`feat/px-brain-persistent-session`, `docs/constitution-and-canonical-map`. + +Commits use Conventional Commits with a scope: `fix(policy):`, `feat(mind):`, +`docs(prompt):`, `test(brain):`, `chore:`. Subject lines here state **what +became true**, not what was done — `a session the sink cannot read is not a +session without quiet mode`. + +### Never blanket-stage + +**Forbidden, without exception:** + +``` +git add -A +git add . +git add -u +git commit -a +git commit --all +``` + +**Stage exact owned paths**, then inspect what you staged: + +```bash +git add path/to/file.py path/to/test_file.py docs/thing.md +git diff --cached +``` + +This is not style. The working tree on this robot routinely carries unrelated +dirty work — a half-finished experiment, a live-tuned constant, a runtime +artifact a daemon just dropped. Blanket staging sweeps all of it into someone +else's commit, and on a repository whose `state/` holds a child's session data +it can commit things that must never be published. + +**Preserve unrelated dirty work.** If you find changes you did not make, +leave them. If they textually overlap the file you must change, commit them +*first* as their own clearly-labelled commit rather than absorbing them. + +### Never end a task with a dirty tree + +Code, tests, and documentation land in **one** commit, and then it is pushed. +A change that is committed but not pushed does not exist to anyone else; a +change whose docs land in a later commit is a change whose docs will not land. + +### Runtime state is never committed + +See [operations/state-and-runtime](operations/state-and-runtime.md). If +`git status` shows a `state/` file you did not create, it is a daemon's output +— do not stage it, and check whether it should be gitignored. + +### Do not commit or push unless asked + +Committing is an outward-facing action on a repository that deploys on merge. + +--- + +## Working on the robot + +The checkout at `/home/pi/picar-x-hacking` **is** the running robot. Daemons +are reading these files while you edit them. + +- Changing a `bin/tool-*` takes effect on the next invocation, immediately. +- Changing a `src/pxh/` module takes effect when the owning daemon restarts. +- Changing `docs/prompts/spark-*.md` requires killing the tmux session — + prompts bake in at launch, and `KillMode=process` means a service restart + will not reload them. See + [architecture/resident-brain](architecture/resident-brain.md). + +### Worktrees + +`.worktrees/` is gitignored and excluded from pytest collection. Use a worktree +when work needs isolation from the live tree — but remember the live daemons +still run from the main checkout. + +## Multi-model QA + +Independent review of a diff, run in parallel and synthesised: + +```bash +hermes -z "QA prompt" 2>&1 +agy --dangerously-skip-permissions --add-dir /path/to/repo --print-timeout 10m --print "QA prompt" 2>&1 +gemini -p "QA prompt" 2>&1 +echo "QA prompt" | codex exec --full-auto - 2>&1 +``` + +**`agy --print` takes the prompt as its value, and must come last.** An earlier +spelling put `--print` first with the prompt trailing, so `--print` consumed +`--dangerously-skip-permissions` as its value, the prompt was never read, and +agy answered a question about the flag and exited 0. + +That is the dangerous failure mode for a review tool: **a QA run that returns +cleanly having reviewed nothing looks exactly like a pass.** Check that the +output discusses your actual code before believing it. `agy`'s default timeout +is 5m, which is short for a whole-diff review. + +Give `agy` a named file list and a ranked list of what to look for. It does +markedly worse with "review this branch". + +--- + +## Why it looks like this + +*History, not rule.* + +The blanket-staging prohibition is written this strongly because the failure is +silent and asymmetric: `git add -A` succeeds, the commit looks clean in +`git log --oneline`, and the unrelated file only surfaces when someone else +bisects to it weeks later. The cost of `git add ` is a few seconds; the +cost of the alternative is unbounded. + +The one-commit rule for code + tests + docs came from documentation drift that +this very restructure exists to undo: docs promised for "a follow-up commit" +reliably did not arrive, and CLAUDE.md accumulated claims that outlived the +code they described. diff --git a/docs/hardware/audio-and-mic.md b/docs/hardware/audio-and-mic.md new file mode 100644 index 00000000..24dc18f8 --- /dev/null +++ b/docs/hardware/audio-and-mic.md @@ -0,0 +1,100 @@ +# Audio and Microphone + +**Owns:** capture and playback. `src/pxh/mic_stream.py`, `bin/tool-voice`, +`bin/px-wake-listen`, `bin/px-mic-check`, the espeak/aplay pipeline. + +Whether SPARK is *allowed* to speak is +[architecture/policy-and-authority](../architecture/policy-and-authority.md). +This page is about whether the sound is correct. + +--- + +## Invariant + +### Capture is `arecord`. Never PyAudio. + +PortAudio's ALSA backend is broken on the C-Media USB mic fitted to SPARK. +Opened at 44100 Hz it delivers only ~29,900 samples/sec, sitting in a permanent +overrun-recovery loop (~7 DROP+PREPARE+START cycles/sec under `strace`) and +discarding buffered audio each cycle. + +The listener **must** pass `exception_on_overflow=False` or a single overrun +kills the daemon — so roughly a third of every utterance is silently spliced +out. + +**There is no clipping, no run of zeros, and no envelope anomaly. Every offline +metric on the recorded WAV looks clean.** Only listening reveals it. This is +why the rule is stated as a prohibition rather than a preference. + +`ArecordStream` mirrors `pyaudio.Stream.read/start_stream/close`, so call sites +are unchanged. + +### The drain thread is load-bearing, not decoration + +A reader thread drains the pipe continuously into a bounded deque. The listener +stops reading for seconds at a time (STT, then an LLM call), and a 64 KB pipe +holds only ~0.37s at 44.1 kHz. Without the drain thread `arecord` would block +on write and overrun its own ALSA buffer — rebuilding the exact bug being +fixed. + +**Drops are counted and logged (`dropped_chunks`), never silent.** The original +failure was invisible precisely because nothing counted them. + +### `bin/px-mic-check` is the regression test + +Chirp-train loopback through SPARK's own speaker. Healthy: **18/18 chirps, +≤3 ms deviation, 0 drops.** The broken PyAudio path scored 13/18 with the +timeline compressed by seconds. + +Needs the mic free — `sudo systemctl stop px-wake-listen` first. + +### Root must set `PULSE_SERVER`, or audio silently fails + +Speech path: `espeak --stdout` → WAV bytes → `aplay -D pulse` → PulseAudio → +HifiBerry DAC → speaker. + +When a script runs as **root** (`px-perform`, `tool-voice`) it must set +`PULSE_SERVER=unix:/run/user/1000/pulse/native` in the `aplay` subprocess +environment. Root's `XDG_RUNTIME_DIR=/run/user/0` cannot find the pi-user +socket, and **`aplay` exits 0 while playing nothing.** + +### `robot_hat.enable_speaker()` before any audio + +It toggles GPIO 20 for the MAX98357A amplifier. Skip it and `aplay` again exits +0 with silence. + +### PulseAudio holds the DAC exclusively + +`aplay -D robothat` (ALSA bypass) fails "device busy". Route through PulseAudio. + +### Whisper anti-hallucination settings are not tuning knobs + +`temperature=0`, `condition_on_previous_text=False`, `no_speech_threshold=0.6`. +Post-filters reject: non-ASCII dominant, phantom phrases, repetitive text. + +### Do not add `bpe_model` to `load_stt_model()` + +The installed sherpa-onnx does not support the kwarg. + +STT priority chain: SenseVoice (primary, ~5s) → faster-whisper (best AU accent) +→ sherpa-onnx Zipformer → Vosk (wake-word grammar only). Models are gitignored +and must be downloaded separately. + +--- + +## Why it looks like this + +*History, not rule.* + +Two separate silent-failure classes shaped this page, and both share a +signature: **the tool reports success.** + +PyAudio's overrun loop produced WAV files that passed every automated check. +The bug survived because everyone was measuring the recording rather than +listening to it. A chirp-train loopback was the first test that could see it, +which is why `px-mic-check` exists as a *timing* test rather than a +signal-quality one. + +The root/PulseAudio failure is the same shape: `aplay` returns 0. Nothing logs +an error. The speaker is simply silent, and the natural conclusion is that the +hardware is broken. diff --git a/docs/hardware/gpio-and-alive.md b/docs/hardware/gpio-and-alive.md new file mode 100644 index 00000000..7b8f92fc --- /dev/null +++ b/docs/hardware/gpio-and-alive.md @@ -0,0 +1,97 @@ +# GPIO Exclusivity and px-alive + +**Owns:** which process may touch the hardware, and how it hands over. +`bin/px-alive`, `bin/px-env`'s `yield_alive`, `src/pxh/gpio_lease.py`. + +Decision authority — whether an action is *allowed* — is +[architecture/policy-and-authority](../architecture/policy-and-authority.md). +This page is about physical exclusivity. + +--- + +## Invariant + +### Exactly one process holds the Picarx handle + +`Picarx()` claims GPIO5 via `reset_mcu()`, and **`close()` does not release +it** — only full process exit does, and `lgpiod` needs time to reclaim the pin +after the client fd closes. + +`px-alive` is the default holder. It keeps a **persistent** handle; do not +refactor it to create and destroy one per action, because each `reset_mcu()` +leaks GPIO5. + +### Handover is by signal, then by lease + +Two mechanisms, for two different durations: + +**Short tool runs — `yield_alive`** (defined in `bin/px-env`). Sends `SIGUSR1` +to the pid in `$LOG_DIR/px-alive.pid`, polls `/proc/` until it disappears, +then waits a further 2.0s for `lgpiod` to reclaim the pin. systemd restarts +px-alive 10s later. + +**Long-running owners — the lease.** `src/pxh/gpio_lease.py` holds a tokenized +`state/gpio_lease.json` and refreshes it while hardware is in use. +`GpioLeaseGuard` acquires it; `wander.py` exports `PX_GPIO_LEASE_ID`, which is +how `bin/tool-describe-scene` and `bin/tool-announce` **borrow** the lease +instead of aborting when they find one held. + +`state/exploring.json` is **not** a lease. It describes wander intent and state +only. Do not use it to arbitrate hardware. + +> **Known defect — [#205](https://github.com/adrianwedd/spark/issues/205).** +> `yield_alive`'s poll loop is `for i in $(seq 25)` at 0.2s — a 5s budget with +> **no failure branch**. A slow px-alive shutdown (measured 8.75s, camera +> teardown) falls through to the `sleep 2.0` and hands the caller a bogus +> success, which then dies with `'GPIO busy'`. It reads as flaky hardware. +> Not fixed here. + +### Readiness is not liveness + +`px-alive.service` is `Type=notify`. `WatchdogSec=15` **only arms after +`READY=1`**, which the daemon sends from `notify_ready()` at the first state +where it is actually working — holding the Picarx handle, or deliberately not +holding it (on charger, in I2C backoff). + +Hardware acquisition therefore runs under `TimeoutStartSec=60`, because +`Picarx.__init__` can block past 15s contending for I2C with a tool that just +took GPIO. Normal acquisition is ~6s. Pre-`READY` heartbeats also send +`EXTEND_TIMEOUT_USEC`, covering an unbounded park behind a foreign lease. + +**Do not add heartbeats inside initialisation instead.** That would keep the +watchdog fed while wedged, blinding it to the exact thing it exists to catch. + +### `os.getlogin()` must stay patched + +`picarx.py:48` calls `os.getlogin()` in `Picarx.__init__()`. Under systemd +there is no `/dev/tty`, so it raises `OSError: [Errno 6]`. + +`~/.local/lib/python3.11/site-packages/usercustomize.py` wraps `os.getlogin()` +with a fallback to `LOGNAME`/`USER`. **Do not remove it** — it affects all 14+ +GPIO scripts, and root needs its own copy after a re-image. + +### The grayscale ADC lies for the first ~0.75s + +Reads taken immediately after `Picarx()` return a fabricated +`[2571, 3085, 3599]`. Use `wander.wait_for_grayscale()`. A *partially* latched +read counts as latched. `race.py` has not been verified against this. + +--- + +## Why it looks like this + +*History, not rule.* + +The 2.0s tail on `yield_alive` is empirical, not theoretical: `px.close()` +returning is not the same event as the kernel releasing the pin, and 1s was not +enough. + +The readiness/liveness split came out of a restart storm — 86 watchdog kills in +a measured 6 hours. Three causes, and the first was a pre-`READY` `WATCHDOG=1` +that armed the watchdog before the daemon was doing anything watchable. The +other two were SD-card fsyncs and are covered in +[operations/state-and-runtime](../operations/state-and-runtime.md). + +What finally cracked it was a `/proc` D-state sampler, after three wrong +theories. Re-run that sampler after *each* fix rather than after all of them — +the storm had more than one cause and fixing one left it looking unchanged. diff --git a/docs/hardware/power.md b/docs/hardware/power.md new file mode 100644 index 00000000..67144268 --- /dev/null +++ b/docs/hardware/power.md @@ -0,0 +1,70 @@ +# Power and Battery + +**Owns:** charge detection and the emergency shutdown. +`src/pxh/battery_trend.py`, `bin/px-battery-poll`. + +--- + +## Invariant + +### Charging cannot be detected from adjacent polls + +On this pack the charge signal is **smaller than the noise around it**. +Measured 2026-08-06: ~0.004 V gained per 30s poll on the charger, against +consecutive readings swinging by up to **0.17 V**. + +Differencing two polls therefore measures noise. That bug reported +`charging: false` through an entire afternoon plugged in. + +### Recovery is a rolling max, then a least-squares slope + +Two facts make the signal recoverable: + +- **Most of the swing is load, not ADC error** — px-alive's servo sweeps drag + the rail down. Load only ever pulls voltage *down*, never up, so a rolling + maximum over `SMOOTH_WINDOW` recovers resting voltage. Measured: residual + noise 0.042 V → 0.026 V. +- **A least-squares slope over `WINDOW` smoothed samples uses every point**, + rather than differencing two of them. + +| Constant | Value | Meaning | +|---|---|---| +| `SMOOTH_WINDOW` | 3 | rolling max, rejects load dips | +| `WINDOW` | 16 | trend window — 8 minutes at a 30s poll | +| `TREND_V_PER_POLL` | 0.006 | noise reaches ~0.0043 at 3σ; a real charge ~0.008 | +| `CONFIRM` | 3 | consecutive agreeing windows before the state flips | + +### The thresholds are deliberately skewed, and the asymmetry is the safety property + +Bootstrapped over measured residuals (2000 trials per condition): **0.6% false +charge per 30-minute window, 85% detection of a real charge.** + +A false `charging` **suppresses the low-battery emergency shutdown** — which is +how a pack gets to brown the Pi out. A missed one costs a needless shutdown or +a late chime. **When in doubt, say not charging.** + +Detection costs ~10 minutes, so the plug-in chime lags. That is the price of +the window, not a bug. + +**Re-tune only against a fresh measured trace. Never against intuition.** + +### Emergency shutdown at ≤10% + +Speaks a warning, then `sudo shutdown -h now`. The alarm beeps +(`mind._play_alarm_beeps()`) are an inventoried **ungated** audio producer — +see [architecture/policy-and-authority](../architecture/policy-and-authority.md). + +--- + +## Why it looks like this + +*History, not rule.* + +The rolling max is not a smoothing filter chosen for elegance. It exploits a +physical asymmetry specific to this robot: electrical load can only reduce +terminal voltage. On a system where noise were symmetric, a rolling max would +bias the estimate upward and be the wrong tool. + +The skew toward under-reporting charge exists because the two error directions +have wildly different costs, and the expensive one is silent — a suppressed +shutdown does not announce itself, it just ends with a corrupted SD card. diff --git a/docs/hardware/wander-safety.md b/docs/hardware/wander-safety.md new file mode 100644 index 00000000..fcd5466a --- /dev/null +++ b/docs/hardware/wander-safety.md @@ -0,0 +1,115 @@ +# Wander Safety + +**Owns:** the cliff guard and the autonomous exploration engine. +`src/pxh/wander.py`, `bin/px-wander`, `bin/tool-wander`. + +--- + +## Invariant + +### The engine is a module, not a script + +`bin/px-wander` is a thin bash wrapper — privilege self-elevation, calibration +guard, `exploring.json` write, `yield_alive` — around `src/pxh/wander.py`. The +engine is importable so the cliff guard can be regression-tested in-process. + +Do not replace the launcher with a direct Python invocation; it does work the +engine relies on. + +### Calibrate before wandering on a new floor + +Place all grayscale sensors over the surface and run +`bin/px-wander --calibrate-cliff` (`--accumulate` keeps the darkest floor +across spots). + +The ADC power-on latch is **rejected**, including a partially latched read, so +calibration fails closed until live sensor values appear. See +[hardware/gpio-and-alive](gpio-and-alive.md). + +### The cliff guard is layered, and every layer earned its place + +Motor noise tripped every early live run. Do not simplify any one of these +away: + +1. **Median-of-3 sampling** — a single read is noise. +2. **Confirmation by persistence**, not by one stationary read. +3. **A stationary re-read** to confirm an in-motion trip. +4. **Sonar echo-timeout retries** (`SONAR_RETRIES`) before counting a sensor + failure. +5. **Board-gap vs. drop discrimination by *width*, not depth** — a floorboard + gap and a table edge look identical on depth. + +`EDGE_ABORT_COUNT` and `SENSOR_FAIL_ABORT_COUNT` end the run rather than +letting it degrade. + +### GPIO protection must be refreshed, not written once + +Every live wander writes `state/exploring.json` **before** constructing +`Picarx`, and runs a 20s `_ExploringRefresher` thread for the whole run. +px-alive ignores the file once its mtime is older than 60s, so a single +start-of-run write protects only the first minute. + +`wander.py` acquires a `GpioLeaseGuard` and **exports `PX_GPIO_LEASE_ID`** so +`tool-describe-scene` and `tool-announce` can borrow the lease. + +### Probe-turn recovery reverses with the SAME steer angle + +Bicycle model: mirroring the steer angle *doubles* the heading change instead +of undoing it. This is counter-intuitive and has been got wrong before. + +### Vision timeouts are a strict ordering, not three independent numbers + +`wander.DESCRIBE_SCENE_TIMEOUT` (165s) must outlive `tool-describe-scene`'s +entire run: `vision.CLAUDE_TIMEOUT` plus photo capture (including an 8s stream +pause) plus its **bounded** 60s `tool-voice` step. + +That bound on the speech step is what stops wander killing the tool mid-run: +`tool-voice` blocks indefinitely when another process holds the audio device. + +The relationship is pinned by +`test_describe_scene_timeout_has_margin_over_claude`, which reads the tool's +real constant rather than a literal **and** pins the surplus. A bare floor +check would stay green right up to the moment there was no margin left, which +is the failure it exists to catch. + +--- + +## Related: autonomous racing + +`bin/px-race` / `src/pxh/race.py` is the *other* autonomous motion system, and +it is documented in full in +[docs/SCRIPTS.md § bin/px-race](../SCRIPTS.md) — PD gains, the safety-layer +priority order, and per-lap learning. Not duplicated here. + +Two things worth knowing before you touch it: + +- **`pd_edge` uses a negative `Kp` (−20.0) on purpose.** Positive error (drift + right) must produce a negative steer (left correction) for the error + convention it uses. Unit tests use a generic `kp=20.0`; that is fine and not + a contradiction. +- **The race loop makes no LLM, network, or audio calls.** It must stay that + way — every one of those has an unbounded tail, and the loop's safety layers + assume they run every cycle. +- **The grayscale power-on latch has not been verified against `race.py`.** + `wander.wait_for_grayscale()` handles it; whether race does is unconfirmed. + +--- + +## Why it looks like this + +*History, not rule.* + +Each cliff-guard layer was added after a specific live failure, which is why +the list reads as over-engineered and is not. The width-not-depth +discrimination came from SPARK reversing away from floorboard gaps in the +hallway until the run aborted. + +`bin/tool-wander` runs `px-wander` under `sudo -n`, and for a long time that +environment flowed straight down into `tool-describe-scene`'s `claude` call, so +vision ran as root with root's `HOME` and silently returned +`FALLBACK_DESCRIPTION` on every real wander. Fixed (#202) by dropping privilege +with `runuser -u pi` for the CLI and passing `HOME` through. `HOME` alone was +the wrong fix — it littered root-owned files in pi's `~/.claude`. The cold +start that exposed it also forced `CLAUDE_TIMEOUT` 45→60 and +`DESCRIBE_SCENE_TIMEOUT` 150→165 **together**, which is why they must be +changed together. diff --git a/docs/ALL_DOCS_REVIEW.md b/docs/historical/ALL_DOCS_REVIEW-2026-05-16.md similarity index 100% rename from docs/ALL_DOCS_REVIEW.md rename to docs/historical/ALL_DOCS_REVIEW-2026-05-16.md diff --git a/HANDOFF.md b/docs/historical/HANDOFF-2026-05-20.md similarity index 100% rename from HANDOFF.md rename to docs/historical/HANDOFF-2026-05-20.md diff --git a/docs/historical/README.md b/docs/historical/README.md new file mode 100644 index 00000000..5dc81e16 --- /dev/null +++ b/docs/historical/README.md @@ -0,0 +1,23 @@ +# Historical Documents + +> **These documents are evidence, not operational truth.** +> +> They are kept because they record what was believed and decided at a point in +> time, which is useful when reconstructing why something is the way it is. +> They are **not** maintained, and several of them are actively contradicted by +> current code. +> +> For what is true now, start at [CLAUDE.md](../../CLAUDE.md) and +> [docs/architecture/overview.md](../architecture/overview.md). + +--- + +## Contents + +| Document | Date | Status | +|---|---|---| +| [HANDOFF-2026-05-20.md](HANDOFF-2026-05-20.md) | 2026-05-20 | Session notes. The LLM tier reorder and expression cooldown it records did land; its "system status" table is a snapshot of that day only. | +| [ALL_DOCS_REVIEW-2026-05-16.md](ALL_DOCS_REVIEW-2026-05-16.md) | 2026-05-16 | A concatenated dump of every doc as of that date, made for a review pass. Superseded wholesale by the canonical docs. | + +Design specs and implementation plans have their own archive and banner: +[docs/superpowers/](../superpowers/README.md). diff --git a/docs/operations/deployment.md b/docs/operations/deployment.md new file mode 100644 index 00000000..a1579f67 --- /dev/null +++ b/docs/operations/deployment.md @@ -0,0 +1,123 @@ +# Deployment and Services + +**Owns:** what runs on the Pi, as whom, and how it is installed. +`systemd/`, `bin/px-env`, the Cloudflare tunnel. + +--- + +## Invariant + +### Every `bin/` script sources `bin/px-env` + +It sets `PROJECT_ROOT`, `LOG_DIR`, adds `$PROJECT_ROOT/src` and +`/home/pi/picar-x` to `PYTHONPATH`, defines `yield_alive`, and sets the default +audio device. A script that does not source it will fail in ways that look like +import errors. + +### `bin/` scripts run under `/usr/bin/python3`, not the venv + +`picarx` and `robot_hat` live in **system** site-packages. The one exception is +`bin/px-wake-listen`, which needs `.venv`. + +Development and tests use the venv: `source .venv/bin/activate`. + +### `.env` is not loaded by `px-env` + +Secrets are loaded by `bin/px-mind` and some other daemons, **not** by +`bin/px-env`. A tool run standalone may therefore lack `OLLAMA_CLOUD_API_KEY` +and similar, and will fail in a way that looks like a network problem. + +### The services + +| Service | Script | User | Restart | +|---|---|---|---| +| `px-alive` | `bin/px-alive` | root | always, 10s (`StartLimitIntervalSec=0`) | +| `px-wake-listen` | `bin/px-wake-listen` | pi | always, 10s | +| `px-battery-poll` | `bin/px-battery-poll` | root | always, 10s | +| `px-mind` | `bin/px-mind` | pi | always, 10s | +| `px-brain` | `bin/px-brain` | pi | always, 10s (`KillMode=process`) | +| `px-post` | `bin/px-post` | pi | always, 30s | +| `px-api-server` | `bin/px-api-server` | pi | always, 2s | +| `px-frigate-stream` | `bin/px-frigate-stream` | pi | always, 10s | +| `px-evolve` | `bin/px-evolve` | pi | on-failure, 30s | +| `px-blog` | `bin/px-blog` | pi | on-failure, 30s | +| `px-tts-glados` | GLaDOS TTS :7861 | pi | always, 10s | +| `cloudflared` | tunnel → spark-api.wedd.au | pi | always, 10s | + +**The root/pi split is why several state directories are `1777`** — see +[operations/state-and-runtime](state-and-runtime.md). + +Unit files and the pip-cleanup timer are documented in +[systemd/README.md](../../systemd/README.md). Install to +`/etc/systemd/system/`, then `daemon-reload` and `enable --now`. + +### `px-brain` uses `KillMode=process` on purpose + +Restarting the supervisor must not kill the tmux sessions it supervises. A +consequence: restarting the service does **not** reload a changed system +prompt, because prompts bake in at session launch. Kill the session. + +### The API is single-worker + +`src/pxh/api.py` on port 8420. **Not multi-worker safe** — it holds in-process +state (rate-limit stores, job table). Do not add workers. + +- Public rate limit: 120 req/min per IP; `/api/v1/public/chat` is 10 msg/10min +- `X-Forwarded-For` is trusted from `127.0.0.1`/`::1` **only** — never from + Cloudflare +- PIN verify returns a session token (4h TTL); the raw Bearer token is never + exposed to a browser +- Per-IP PIN lockout (`state/pin_lockout.json`): 3 failures → 5 min, 10 → 30 + min, capped at 1000 IPs / 10k rate-limit entries with oldest-first eviction +- Device reboot/shutdown is two-step: `POST /api/v1/device/{action}` returns a + nonce, confirmed within 60s + +### The site auto-deploys from `master` + +Cloudflare Pages serves `site/`. **A merge to `master` publishes the public +site**, so treat `site/` changes as outward-facing. + +- `site/css/colors.css` — single-source 12-mood palette. All JS reads + `getComputedStyle().getPropertyValue('--mood-' + mood)`; **never hardcode + hex**. +- `site/js/config.js` — the single API base URL. **Never hardcode URLs in JS.** +- `site/workers/og-rewrite.js` — rewrites OG meta server-side, because social + crawlers do not execute JS. + +### Address the announce relay by IP + +`ANNOUNCE_RELAY_URL` in `spark_config.py` points at `192.168.0.249:7862`. +**Never `M5.local`** — Nest speakers fetch the audio URL themselves and cannot +resolve mDNS. The relay's own `RELAY_PUBLIC_BASE_URL` must match, or every +audio URL it hands out points at the wrong address. + +Health check from the Pi: `curl http://192.168.0.249:7862/health` + +### Other surfaces + +Thin surfaces that publish or receive, each with one thing worth knowing: + +| Surface | Entry point | The gotcha | +|---|---|---| +| Social posting | `bin/px-post` | watches `thoughts-spark.jsonl` (salience ≥0.7 or a spoken action) behind a Claude QA gate. **Ambiguous QA responses default to pass** — the gate is a safety net, not a quality bar. | +| Blog | `bin/px-blog`, `bin/tool-blog` | writes a `state/blog.json` envelope served at `GET /api/v1/public/blog`; OG meta is rewritten by the same Cloudflare Worker pattern as `/thought/*`. | +| MCP server | `bin/mcp-server` | 5 **read-only** tools over stdio (`spark_status`, `spark_thoughts`, `spark_awareness`, `spark_sonar`, `spark_vitals`), registered in `.mcp.json`. | +| Home Assistant | `ha/custom_components/spark_conversation/` | routes Nest/Hub Max voice through `POST /api/v1/public/chat`. HA 2026.x wants `supported_languages` as a `@property` and `AddConfigEntryEntitiesCallback`. | +| Obi chat | `POST /api/v1/obi-chat` | authenticated, 10s rate gate, both sides logged; text sanitised — see [privacy](../architecture/privacy.md). | +| Location push | cron on M5.local | pushes `state/findmyhub.json` to the Pi every 5 min. Its contents are **excluded from reflection** — see [privacy](../architecture/privacy.md). | + +Per-script detail: [docs/SCRIPTS.md](../SCRIPTS.md). + +--- + +## Why it looks like this + +*History, not rule.* + +`px-brain.service` was believed for about eleven hours to have an +authentication problem. It had never been installed. Check `systemctl status` +before debugging the thing the failure appears to be about. + +The relay moved from a wired `.100` to `.249` when M5's wired adapter was +unplugged, and the mDNS prohibition was learned from Nest speakers silently +failing to fetch audio they had been handed a `.local` URL for. diff --git a/docs/operations/health.md b/docs/operations/health.md new file mode 100644 index 00000000..c28ad377 --- /dev/null +++ b/docs/operations/health.md @@ -0,0 +1,92 @@ +# Daemon Health + +**Owns:** `src/pxh/health.py`, `state/health/`, `bin/px-health-report`. + +--- + +## Invariant + +### Health answers a question `systemctl status` cannot + +`systemctl` knows whether a process is running. Health knows whether it is +**doing its job**. Every daemon calls `record_success()` / `record_failure()`; +`read_health()` aggregates. + +### Status is derived at read time, never stored + +A dead daemon cannot leave a lying `ok` behind. The ladder: + +`ok` → `degraded` (1–2 failures) → `stale` (silent past its per-component +window) → `failing` (≥3 consecutive) / `missing` (no file at all). + +Absent files report `missing` rather than being silently omitted — a daemon +that never started is exactly what you want to see. `KNOWN_COMPONENTS` is +derived from `STALE_AFTER_S`, so a component with a window is a component +that gets reported. + +### Staleness windows are per-component, because cadences differ by orders of magnitude + +`px-mind` ticks every 60s; `px-blog` runs daily. One shared window would either +call the blog broken or never notice the mind had stopped. + +| Component | Window | Why | +|---|---|---| +| `px-mind` | 300s | awareness ticks every 60s | +| `px-mind-reflection` | 3600s | backs off to 8× its 300s base when nobody is around | +| `px-alive` | 300s | idle actions are sporadic; heartbeat is periodic | +| `px-wake-listen` | 900s | reports on wake events plus a periodic heartbeat | +| `px-post` | 3600s | only runs when a postable thought appears | +| `px-blog` | 86400s | daily at its most frequent | +| `px-brain`, `px-brain-io` | 300s | ticks every 10s, throttles writes to once a minute | +| default | 900s | `DEFAULT_STALE_AFTER_S` | + +### Successes throttle. Failures never do. + +`record_success(..., min_interval_s=N)` exists because `px-alive` ticks twice a +second and an fsync per tick would wear the SD card. + +**A failure clears the throttle**, so the recovery is written immediately. +Without that, a flapping component accumulates failures while its successes are +dropped, and reads as `failing` while working. + +### Reporting never raises + +Health must not be able to kill the daemon it reports on. Every reporting path +swallows its own errors. + +### Read `read_health()` directly when px-mind might be down + +`px-mind` publishes the aggregate to `state/health.json` and into +`awareness["health"]`, and `summarize()` feeds reflection context. That +snapshot is convenient, not authoritative. **Any reader that must be correct +when px-mind is down calls `read_health()` itself.** + +### Storage is one file per component, `1777` + +See [operations/state-and-runtime](state-and-runtime.md) for why. Do not +consolidate into one file and do not tighten the mode. + +--- + +## Known limitation + +**Health is blind to chronic partial failure.** Status keys off *consecutive* +failures, so a component failing a steady 18% of the time reads as `ok` +indefinitely — every failure is cleared by the next success. + +When diagnosing, check success/failure **ratios**, not `overall`. + +--- + +## Why it looks like this + +*History, not rule.* + +The store was a single shared JSON file first. It needed a lock; the lock was +created by whichever daemon got there first; `px-alive` runs as root and +created it 0644; every `pi` daemon then failed with `EACCES` and reported +nothing at all — a health system whose failure mode was total silence. + +Splitting to one file per component removed the lock, the read-modify-write +race, and the ownership hazard in a single change, which is why it is preferred +over "fix the lock permissions". diff --git a/docs/operations/llm-routing.md b/docs/operations/llm-routing.md new file mode 100644 index 00000000..93f30ccb --- /dev/null +++ b/docs/operations/llm-routing.md @@ -0,0 +1,111 @@ +# LLM Routing, Budgets and Metering + +**Owns:** which model answers, what it costs, and how that is counted. +`mind.call_llm`, `src/pxh/claude_session.py`, `src/pxh/token_log.py`, +`src/pxh/brain.py`'s meter. + +--- + +## Invariant + +### Reflection has a four-tier chain, and tier 1 is local + +`mind.call_llm` (`src/pxh/mind.py`): + +| Tier | Backend | When | +|---|---|---| +| 1 | Ollama on M5 (LAN) | primary for **all** personas, including SPARK | +| 2 | Claude Haiku | SPARK fallback when M5 is unreachable, or `PX_MIND_BACKEND=claude` | +| 3 | Ollama Cloud | when M5 is unreachable and Claude fails (needs `OLLAMA_CLOUD_API_KEY`) | +| 4 | Ollama on the Pi | opt-in via `PX_MIND_LOCAL_OLLAMA=1`; **off by default, OOM risk** | + +`PX_MIND_BACKEND` selects the shape: `auto` (SPARK→Claude fallback, +others→Ollama only), `claude` (Claude primary), or `ollama`. + +### Use `M5.local`, never the bare hostname `M5` + +`OLLAMA_HOST` defaults to `http://M5.local:11434`. The UDR7 stopped serving the +bare `M5` hostname: `getent hosts M5` returns nothing, while +`getent hosts M5.local` resolves to 192.168.0.249 and answers `/api/tags` in +28 ms. + +**A bare `M5` makes tier 1 fail instantly and silently spends money on tier 2.** +Treat any `192.168.1.x` address in an older document as stale — the network was +renumbered to `192.168.0.x`. + +### Tier 2 asks the resident brain first + +`mind.call_claude` → `call_brain_reflection` (kind `reflection`), and only +shells out to `claude -p` when `ask_brain` returns `None`. Warm context instead +of a cold process per thought, and **metered**. + +Reflection reaches the meter via `ask_brain` **without** going through +`claude_session.py`'s per-type cooldowns. That is deliberate: reflection runs +every 5 minutes and a daily cap would simply stop it. The meter gives +visibility without a cap. + +### `backend=` in the reflection log is the *configured* primary, not the tier that answered + +To know which tier actually served, grep the log for `falling back`. +`call_llm()` also sets `result["backend"]` to the tier that served — use that +programmatically. + +### Claude session types are budgeted + +`src/pxh/claude_session.py`: + +| Session type | Model | Cooldown | Daily quota | +|---|---|---|---| +| `evolve` | Opus | 24h | 1 | +| `self_debug` | Sonnet | 6h | 2 | +| `research` | Haiku | 2h | 3 | +| `compose` | Haiku | 4h | 2 | +| `conversation` | Sonnet | 15min | 4 | +| `blog` | Haiku | 30min | 5 | +| `consolidate` | Haiku | 20h | 1 | + +Global: 30min between sessions (except `self_debug`/`blog`), 8/day cap. At ≤2 +remaining, only `self_debug` and `evolve` are permitted. Bypass with +`PX_CLAUDE_BUDGET_DISABLED=1`. Log: `state/claude_sessions.jsonl`. + +### Spend visibility requires `by_backend` + +`token_log.log_usage()` takes a `backend` argument and splits totals under +`by_backend` in `state/token_usage.json`. + +**The top-level totals mix free Ollama with paid Claude and cannot answer "what +am I spending".** Read `by_backend`. + +### The remaining `claude -p` call sites are known and finite + +`mind.py` (`call_claude_haiku`, now the *fallback* under the brain), +`api.py` (`_call_claude_public`), `bin/claude-voice-bridge`, `bin/px-blog`, +`bin/px-post` (legacy branch), `src/pxh/vision.py` (via +`bin/tool-describe-scene`), `bin/px-cron-say`. + +The `claude -p` fallback under the brain meter is **still unmetered** — that is +the remaining hole in spend accounting. + +Rollout is controlled by `PX_BRAIN_KINDS`; see +[architecture/resident-brain](../architecture/resident-brain.md). + +### Personas need `think: false` on Ollama + +Reasoning chains re-enable refusal in small models. `clean_response()` strips +scaffolding dividers before voice output. + +--- + +## Why it looks like this + +*History, not rule.* + +Tier 1 was Claude for SPARK originally. Moving the LAN model to primary was a +cost decision, and the hostname bug then quietly undid it: a bare `M5` failed +tier 1 in milliseconds and fell through to paid Haiku on every single +reflection, which is a failure mode that looks exactly like working software. + +The meter exists because reflection's tier 2 bypassed the session budget +entirely — 501 unbudgeted Claude calls in 19 days before anyone counted. +`state/token_usage.json`'s top-level totals did not reveal it, because they +counted free Ollama calls in the same number. diff --git a/docs/operations/state-and-runtime.md b/docs/operations/state-and-runtime.md new file mode 100644 index 00000000..089c1f2e --- /dev/null +++ b/docs/operations/state-and-runtime.md @@ -0,0 +1,105 @@ +# State and Runtime + +**Owns:** where data lives and how it is written. +`src/pxh/state.py`, `src/pxh/runtime_paths.py`, and the `state/` directory. + +--- + +## Invariant + +### Runtime state is not source code + +`state/` holds the robot's living state. Almost none of it is tracked: only +`state/session.template.json` and `state/spark-reflect/CLAUDE.md` are in git. + +**Never commit a runtime state file.** A tracked `session.json`, +`awareness.json`, or `thoughts-*.jsonl` would make every deploy a state +rollback and would put a child's session data into a public repository. The +`.gitignore` rules are a safety mechanism, not tidiness. + +`site/data/feed.json` and `site/data/blog.json` are the deliberate exception — +they are the public site's **offline fallback**, so they are tracked on purpose. + +**First use:** `cp state/session.template.json state/session.json` + +### Three storage classes, and putting a file in the wrong one has bitten us + +| Class | Location | For | +|---|---|---| +| Durable | `state/` on the SD card | survives reboot and matters afterwards | +| Runtime | `/run/spark` (tmpfs) | rewritten every loop, meaningless after a power cut | +| Logs | `$LOG_DIR` | append-only, rotated | + +`src/pxh/runtime_paths.py` owns the runtime class. `RUNTIME_DIR_ENV` is +`PX_ALIVE_HEARTBEAT_DIR` — a historical name that governs the whole runtime +directory, not just the heartbeat. Writer (`bin/px-alive`, root) and readers +(`api`, `health`, `mind`, `mcp_server`, all as `pi`) must agree on the +location, so the env var and default live in one module rather than in each of +them. + +**A file rewritten every loop belongs on tmpfs.** Measured on the live Pi: a +169-byte fsync+replace into `state/` has a p50 of 12 ms but a tail reaching +**21.5 s** under load. The identical write to tmpfs measures 0.63 ms with no +tail. + +### `atomic_write()` is the only way to write durable state + +mkstemp + fsync + `os.replace`. The fsync is for SD-card durability; the +replace is what makes a reader either see the old file or the new one, never a +half-written one. + +`mkstemp` needs **directory** write permission, which is why several state +directories are `1777` — see below. + +### Session access is lock-protected, and `FileLock` is not reentrant + +`state.py` guards `state/session.json` with a `FileLock` at +`LOCK_TIMEOUT_S = 10` — fail fast rather than hang forever. + +**`update_session()` calls `ensure_session()` *before* acquiring the lock**, +precisely because `FileLock` is not reentrant. Moving that call inside the lock +deadlocks. + +Readers that must not block have `load_session_readonly()`. + +### Directories shared between root and pi are `1777` + +`state/health/` and `state/brain/` are created sticky and world-writable, like +`/tmp`. `_ensure_health_dir()` re-chmods on every write. + +This is load-bearing. SPARK's daemons do not all run as the same user — +`px-alive` and `px-battery-poll` run as root while everything else runs as +`pi` — and a root-created `0755` directory locks every `pi` writer out of +`atomic_write`'s `mkstemp`. Whichever user wins the creation race, both can +write. + +**Do not "tighten" these to 0755.** + +### One file per component, not one shared file + +`state/health/.json`. A shared file would need a `FileLock`, and a +root-created lock at 0644 locks out every `pi` daemon with `EACCES`. +Per-component files remove the lock, the read-modify-write race, and the +ownership hazard together. See [operations/health](health.md). + +--- + +## Why it looks like this + +*History, not rule.* + +The tmpfs split was forced by a watchdog restart storm on `px-alive`. A `/proc` +sampler caught the daemon in uninterruptible sleep on `fsync` of +`state/tmp.tmp` in 27 of 58 samples, 24 of them parked in +`jbd2_log_wait_commit`, with 23 consecutive samples on a single temp file. That +accounted for 66 of 86 watchdog kills in a measured 6-hour window. + +`WatchdogSec=15` sits *under* the 21.5 s write tail, so systemd was SIGABRT-ing +a perfectly healthy daemon — and because the process was blocked in +uninterruptible I/O, it took a SIGKILL to actually die. + +The heartbeat moved to tmpfs first and the storm continued, because the live +sonar write had been left behind and was enough to sustain it on its own. That +is why `runtime_paths.py` is generic in the filename rather than one pair of +helpers per file: **anything that lands in this class later gets the same +treatment automatically.** diff --git a/docs/superpowers/README.md b/docs/superpowers/README.md new file mode 100644 index 00000000..780a8065 --- /dev/null +++ b/docs/superpowers/README.md @@ -0,0 +1,49 @@ +# Specs and Plans — Decision Fossils + +> **These documents are evidence, not operational truth.** +> +> Everything under `specs/` and `plans/` records what was *decided* and *why*, +> on the date in its filename. None of it is maintained afterwards. A spec +> describes the system as it was intended at the moment of writing; the code +> moved on, and the spec did not. +> +> **Do not implement from a document here, and do not cite one as current +> behaviour.** Read it to understand *why* something is the way it is, then +> verify the *what* against the code and against +> [the canonical docs](../architecture/overview.md). + +--- + +## How to use these + +| You want to know | Read | +|---|---| +| What the system does today | [docs/architecture/overview.md](../architecture/overview.md) | +| Why it does that | the spec here, plus the "Why it looks like this" section of the canonical doc | +| What was tried and rejected | the spec's alternatives section | + +Specs whose subject is still live are cross-linked from the canonical doc that +owns the topic. Those links point *backwards* — from current truth to its +rationale — never the other way round. + +## Contents + +- **`specs/`** — designs produced by the brainstorming workflow, written + before implementation. `YYYY-MM-DD--design.md`. +- **`plans/`** — step-by-step implementation plans, including their staging + and commit commands. Those commands were correct for the tree that existed + then; **do not run them.** + +## Known-superseded highlights + +These are frequently mistaken for current truth: + +| Fossil | Superseded by | +|---|---| +| `specs/2026-08-01-px-brain-design.md` | shipped and changed — [resident-brain](../architecture/resident-brain.md) | +| `specs/2026-08-17-brain-handshake-validation-design.md` | shipped — readiness is now a proven round trip | +| `plans/*` staging commands | [git-workflow](../git-workflow.md) — stage exact owned paths | +| any test count in any plan | [testing](../testing.md) | + +Other historical material — superseded session notes and doc reviews — is in +[docs/historical/](../historical/). diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 00000000..ea570ed5 --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,151 @@ +# Testing + +**Owns:** how the suite is isolated, what "green" means, and what it cannot +prove. `tests/conftest.py`, `pyproject.toml`. + +--- + +## Invariant + +### Tests are hermetic by default + +This repository is checked out **on the robot it controls**. A test that reads +or writes live state is not a flaky test — it is a test that changes the +robot's behaviour and then reports on itself. + +A test must not, unless explicitly marked `live`: + +- read or write anything under the live `state/` +- read the live `/run/spark` runtime directory +- make a billed LLM call +- reach the network +- touch GPIO, the microphone, or the speaker + +### Running + +```bash +python -m pytest # full suite +python -m pytest -m "not live" # skip hardware tests +python -m pytest tests/test_state.py +python -m pytest -k test_name +sudo .venv/bin/python -m pytest tests/test_tools_live.py -v -s # live hardware +``` + +`testpaths = ["tests"]` and `norecursedirs` exclude `.worktrees` — pytest does +**not** honour `.gitignore`, and without these it collected every worktree's +same-named `tests/` tree and died with 229 import-mismatch errors. Bare +`pytest` from the repo root must be correct: the local run is this project's +real gate. + +### Two isolation mechanisms, and they cover different things + +**Autouse fixtures** (unconditional, every test) isolate **in-process** writes: + +| Fixture | Redirects | Hazard if absent | +|---|---|---| +| `_isolate_health_writes` | `health.health_dir()` | mock health records overwrite the live dashboard | +| `_isolate_brain_mailbox` | `brain.brain_root()` | a real request lands in the running session's inbox, spends budget, and makes SPARK act | +| `_isolate_alive_heartbeat` | `PX_ALIVE_HEARTBEAT_DIR` | tests read the *live* robot's heartbeat and pass for the wrong reason | + +Each redirects **only its own root**, not `PX_STATE_DIR` globally, because many +tests deliberately set `PX_STATE_DIR` themselves. + +**The `isolated_project` fixture** is opt-in and isolates **subprocesses**. It +supplies an `env` dict with `PROJECT_ROOT`, `LOG_DIR`, `PX_SESSION_PATH`, +`PX_STATE_DIR`, `PX_BYPASS_SUDO=1`, `PX_VOICE_DEVICE=null`, and a pinned +night-silence window. + +> **Session isolation is not yet on `master`.** In-process reads of +> `state/session.json` still reach the live file, which is why a live +> `spark_quiet_mode: true` reddens several `test_mind_utils` tests on the +> robot. An autouse `_isolate_session` fixture is landing via +> [#212](https://github.com/adrianwedd/spark/issues/210). Until it merges, +> isolate `PX_SESSION_PATH` before blaming a branch for those failures. + +### The night-silence window must be pinned, or the suite is time-dependent + +`isolated_project` sets `PX_NIGHT_SILENCE_START_H=99` (never true), because +`bin/tool-voice` evaluates policy for itself. Without it, **every subprocess +test of a speaking tool passes by day and returns `suppressed` after 19:00 +Hobart.** + +The env-var seam exists precisely because the enforcement points are +subprocesses — a test cannot monkeypatch inside `bin/tool-voice`. Tests that +mean to exercise night silence override **both** values. + +### Targeted green is not repository green + +Running `-k` on the tests you touched proves your change; it does not prove the +repository. Run the full suite before claiming done. + +And a green suite does not prove the **live** paths. These require explicit +live evidence and cannot be inferred from tests: + +- GPIO acquisition and handover +- the resident tmux sessions and their trust boundary +- audio actually reaching the speaker +- anything behind `sudo` + +### Some tests are structural tripwires, not coverage + +They fail on purpose when the code changes shape: + +- `test_every_audio_producer_is_inventoried` — a new file reaching audio must + be classified +- `test_reflection_awareness_json_is_allowlisted` — a new awareness key stays + out of the prompt +- `test_launcher_renders_one_absolute_reply_spelling` — one spelling of + `tool-brain-reply` +- `test_describe_scene_timeout_has_margin_over_claude` — pins a *relationship* + and its surplus, not a literal + +Do not "fix" these by updating the expected value. Fix the code, or make the +classification deliberately. + +### `tests/test_policy_invariants.py` is not evolvable + +It and `src/pxh/policy.py` are in `claude_session.BLACKLIST_FILES`. Evolvable +policy coverage lives in `tests/test_policy.py`. **Keep that split** — see +[architecture/policy-and-authority](architecture/policy-and-authority.md). + +--- + +## Known limitations + +- **A full run produces failures that are not your change.** Before blaming a + branch, check them against this list: + + | Failure | Cause | + |---|---| + | `tests/test_tools_live.py` (11) | live hardware — needs `sudo` and the mic/GPIO free | + | `test_mind_utils` (6) | samples the live session; [#210](https://github.com/adrianwedd/spark/issues/210) | + | `TestBudgetSummary` (2) | fixed offsets vs. Hobart calendar day — fails before ~08:20 local; [#213](https://github.com/adrianwedd/spark/issues/213) | + | `TestRaceEndpoint` (1–2) | fixed waits on a background thread under load; [#211](https://github.com/adrianwedd/spark/issues/211) | + +- **Some failures differ between runs** and pass in isolation: 15s subprocess + timeouts under load, and a process-wide `time.sleep` patch in `test_race` + catching leaked `test_api` threads. Re-run in isolation before believing any + `test_race` / `test_api` / `test_evolve_coverage` failure. +- **Test runs write into the real `logs/`.** Millisecond-clustered entries, or + ones naming `/tmp/pytest-of-pi/`, are artifacts rather than incidents. +- **Test runs can trip `px-post`'s in-memory 3-failure Bluesky auth disable.** + `sudo systemctl restart px-post` afterwards. + +--- + +## Why it looks like this + +*History, not rule.* + +Every autouse fixture here was added after a test run changed the live robot. +The health fixture was added when a suite run overwrote live health with mock +values and the dashboard reported whatever the tests last asserted. The brain +mailbox fixture was added because an unisolated test dropped a real request +into the running session's inbox and it was answered. + +The heartbeat fixture is the subtlest of the three: `#192` moved the heartbeat +to tmpfs and `resolve_heartbeat_read_path()` prefers `/run/spark` +unconditionally — correct in production, but on a host where `px-alive` is +running it meant isolated tests read the live beat. Eight tests that passed on +a CI box failed on the robot, and would have passed there *for the wrong +reason* had the fixture been inverted. diff --git a/systemd/README.md b/systemd/README.md index d1df4fed..adbb3950 100644 --- a/systemd/README.md +++ b/systemd/README.md @@ -1,6 +1,8 @@ # systemd units -Service units for the eleven SPARK daemons (see CLAUDE.md "Systemd Services"). +Service units for the SPARK daemons. The full table — script, user, and +restart policy for each — is in +[docs/operations/deployment.md](../docs/operations/deployment.md). Install to `/etc/systemd/system/` on the Pi, then `daemon-reload` + `enable --now`. ## Maintenance timer: pip /tmp cleanup diff --git a/tests/test_docs.py b/tests/test_docs.py new file mode 100644 index 00000000..7e9c70d0 --- /dev/null +++ b/tests/test_docs.py @@ -0,0 +1,127 @@ +"""Documentation structure tests. + +Three checks, deliberately no more. Documentation rot in this repo has always +been *silent* — CLAUDE.md claimed "six kinds" of provenance for as long as +there were seven, and nothing failed. These convert the classes of rot that +can be checked mechanically into red tests, and leave the rest to review. + +Not a docs framework. Do not grow this into one: a test that asserts on prose +becomes a test that has to be edited every time prose improves. +""" +from __future__ import annotations + +import re +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[1] + +# Files whose links are checked. The canonical docs plus the entry points that +# route readers into them — the places where a broken link strands someone. +LINK_CHECKED = ( + "CLAUDE.md", + "AGENTS.md", + "systemd/README.md", + "docs/testing.md", + "docs/git-workflow.md", + "docs/historical/README.md", + "docs/superpowers/README.md", +) +LINK_CHECKED_DIRS = ("docs/architecture", "docs/hardware", "docs/operations") + +# [text](target) — non-greedy text, target up to the first closing paren. +_MD_LINK = re.compile(r"\[[^\]]*\]\(([^)\s]+)\)") + +_SKIP_PREFIXES = ("http://", "https://", "mailto:", "#") + + +def _link_checked_files() -> list[Path]: + files = [REPO_ROOT / name for name in LINK_CHECKED] + for directory in LINK_CHECKED_DIRS: + files.extend(sorted((REPO_ROOT / directory).glob("*.md"))) + return files + + +def _relative_links(path: Path) -> list[str]: + text = path.read_text(encoding="utf-8") + out = [] + for target in _MD_LINK.findall(text): + if target.startswith(_SKIP_PREFIXES): + continue + out.append(target) + return out + + +@pytest.mark.parametrize( + "doc", _link_checked_files(), ids=lambda p: str(p.relative_to(REPO_ROOT)) +) +def test_relative_links_resolve(doc: Path): + """A link into the canonical docs must point at a file that exists. + + This is the check that would have caught the docs tree drifting apart as + subsystems were renamed. Anchors are stripped: whether a heading exists is + not mechanically checkable in a way worth the false positives. + """ + assert doc.exists(), f"link-checked file is missing: {doc}" + broken = [] + for target in _relative_links(doc): + resolved = (doc.parent / target.split("#", 1)[0]).resolve() + if not resolved.exists(): + broken.append(target) + assert not broken, ( + f"{doc.relative_to(REPO_ROOT)} links to missing paths: {sorted(broken)}" + ) + + +def test_constitution_forbids_blanket_staging(): + """The staging prohibition must survive edits to CLAUDE.md. + + It is the one rule whose violation is silent and unbounded: `git add -A` + succeeds, the commit reads clean, and the unrelated file only surfaces + when someone bisects to it. If a rewrite drops it, that must be loud. + """ + text = (REPO_ROOT / "CLAUDE.md").read_text(encoding="utf-8") + for forbidden in ("git add -A", "git add .", "git commit -a"): + assert forbidden in text, ( + f"CLAUDE.md no longer names {forbidden!r} as forbidden staging" + ) + assert "exact owned paths" in text, ( + "CLAUDE.md no longer states the positive rule (stage exact owned paths)" + ) + + +@pytest.mark.parametrize( + "banner_doc", ("docs/superpowers/README.md", "docs/historical/README.md") +) +def test_historical_docs_carry_a_banner(banner_doc: str): + """Specs, plans and archived notes must announce that they are fossils. + + Without the banner a reader finds a confident, dated design document and + reasonably assumes it describes the running system. Several of them + describe systems that were never built that way. + """ + path = REPO_ROOT / banner_doc + assert path.exists(), f"missing historical index: {banner_doc}" + text = path.read_text(encoding="utf-8") + assert "evidence, not operational truth" in text, ( + f"{banner_doc} is missing the fossil banner" + ) + + +def test_every_canonical_doc_separates_invariant_from_history(): + """Each canonical doc must say which parts are rules and which are history. + + Collapsing the two is how CLAUDE.md grew to 495 lines: incident narrative + and binding rule in the same voice, so neither could be trimmed safely. + """ + missing = [] + for directory in LINK_CHECKED_DIRS: + for doc in sorted((REPO_ROOT / directory).glob("*.md")): + text = doc.read_text(encoding="utf-8") + if "## Invariant" not in text or "## Why it looks like this" not in text: + missing.append(str(doc.relative_to(REPO_ROOT))) + assert not missing, ( + "canonical docs missing an '## Invariant' or " + f"'## Why it looks like this' section: {missing}" + )