Skip to content

Gridded (ISIMIP DDM30) routing as a data source, the ddrs-run skill, and three silent-failure fixes - #40

Merged
taddyb merged 7 commits into
masterfrom
worktree-gridded-routing
Sep 11, 2026
Merged

taddyb merged 7 commits into
masterfrom
worktree-gridded-routing

Conversation

@taddyb

@taddyb taddyb commented Sep 10, 2026

Copy link
Copy Markdown
Owner

Adds gridded (ISIMIP DDM30) routing to ddrs as a data source, mirroring DeepGroundwater/ddr#194, plus the ddrs-run skill and three code fixes that writing it turned up.

ddrs has no engine/ module and this does not add one. DDR#194 is a data-preparation feature (its own summary: "the routing engine, mmc.py, merit.py and the MERIT/Lynker paths are untouched"), so ddrs consumes its products and ports none of the raster aggregation or Q′ regridding.

1. Gridded routing

The key observation is that ddrs already had the right abstraction. Reach subdivision (params.subdivision) models exactly the structure a DDM30 sub-reach network has, so the whole routing / training / eval / baseline path is untouched and only the builder input is new:

Gridded sub-reach zarr (DDR) ddrs subdivided store
parent_cell per row, contiguous, upstream-first order, parent_order, parent_offset
attributes looked up by parent cell AttributesStore::open(…, parent_order)
Q′ per cell ÷ k_per_node pieces_per_row_divisor
gauge outlet = cell's last sub-reach outlet(p) = parent_offset[p+1] − 1

New data_sources.gridded_network names DDR's sub-reach adjacency zarr; ddrs plan relabels it into that layout and cuts the gauge subgraphs (src/adjacency/gridded.rs, cache::resolve_or_build_gridded). The gauge CSV's cell column is a serde alias of COMID. The key is mutually exclusive with geospatial_fabric and the explicit adjacency pair, and params.subdivision.enabled is rejected alongside it because the store arrives already split.

Results on the committed examples/juniata_gridded/ bundle (byte-identical to DDR's, CPU, 30 epochs):

NSE KGE
summed-Q′ baseline, ddrs / DDR 0.594 / 0.594 0.672 / 0.671
routed, seed 42 0.751 0.730
routed, seeds {42, 7, 123} 0.744–0.758 0.728–0.732
routed, DDR Python 0.795 0.737

The baseline is deterministic and matches DDR to rounding through independent readers, which makes it a live cross-implementation check. The routed gap has two known causes, both recorded in the spec: different RNG window streams, and ddrs running the KAN head per cell where DDR runs it per sub-reach with an accumulated log10_uparea. ddrs also applies no da_ratio output correction.

CONUS is wired up too (config/sources/conus-gridded.yaml, config/experiments/gridded_conus.yaml, scripts/snap_gridded_gauges.py reproducing DDR's 620-gauge population). plan builds 5,198 cells into 91,867 sub-reaches in 0.6 s; baseline median NSE 0.354. Training beyond a mechanics smoke was not run.

Two bugs this exposed

  • upstream_comids was per-piece on subdivided stores, so the summed-Q′ baseline counted each divide once per sub-reach. This also affected MERIT params.subdivision baselines.
  • DDR's gridded Q′ stores are Qr(time, divide_id), the transpose of the contract, and zarrs answers an out-of-range subset with fill values rather than an error, so the fixed-axis reader silently produced an all-NaN baseline. The reader now sniffs the axis order from dimension names, falls back to shape, and refuses a shape matching neither. Fixture: tests/fixtures/qr_daily_time_major.ic.

2. ddrs-run skill

A third repo skill covering launching, watching, resuming and auditing runs, and every ddrs subcommand, split from ddrs-dev (build/code/test) and ddrs-eval-plots (visualize). Written against the binary and source rather than from memory, and tested with a subagent doing a realistic task with and without it.

3. Three code fixes

That testing found five documentation defects; three were real code bugs, now fixed:

  • ddrs show printed no metrics in text mode. They lived only in --json, so ddrs show <id> | grep nse came back empty on a healthy run and read like a failure — and ddrs-eval-plots recommended exactly that grep. Text mode now prints the metrics block and, for a train-and-test run, reduces the baseline (whose entries are per-gauge arrays, not medians) to a median per metric beside the routed value and delta, flagging when routed loses:
baseline (summed Q', median over gauges)
  NSE                    0.5943   routed 0.7511   Δ +0.1568
  • geodataset: was validated by nothing. Being provenance-only is what made a wrong value dangerous: the run succeeds and its audit trail names the wrong network, and ddrs sources use swaps only the data_sources block so a stale label survives a dataset switch. Absent is now inferred from the adjacency source; contradicting is a load error; explicit adjacency paths stay unconstrained because a pre-built store can be either network.
  • A typo'd config key silently took its default, the documented single most common cause of "my config change did nothing". All seven remaining config structs now set deny_unknown_fields.

Gates

Gate Result
cargo test 630 passed / 0 failed, 101 binaries
compare_ddr_sandbox ABSOLUTE MATCH
juniata_acceptance NSE 0.7903 / KGE 0.8810
gridded_acceptance NSE 0.7511 / KGE 0.7299

Both acceptance figures are identical to their pre-fix values, which is the evidence that the routing path is untouched. Invariants 1–7 are unaffected.

Review notes

  • deny_unknown_fields is the one change with real blast radius. Every shipped config and all 630 tests pass, but a personal ddrs.yaml or an old run snapshot carrying a stray or removed key will now fail to load instead of ignoring it. Intended, and loud rather than silent, but it is a behaviour change.
  • Only DDR's sub-reach network is supported. The unsplit cell adjacency (no parent_cell) is rejected with a message naming the script that produces a sub-reach store.
  • Design, assumptions and deviations: docs/superpowers/specs/2026-09-08-ddrs-gridded-routing-design.md.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EqnrtmU91yzuVpGnADnXLP

taddyb and others added 7 commits September 8, 2026 22:47
…(mirrors ddr #194)

`data_sources.gridded_network` names DDR's sub-reach adjacency zarr
(order, parent_cell, indices_0/1, length_m, slope). `ddrs plan` relabels it
into the existing subdivided store layout (parent = 0.5° cell, pieces = the
cell's contiguous sub-reaches, gauge read at the last piece) and cuts gauge
subgraphs from it — adjacency::gridded::GriddedNetwork +
cache::resolve_or_build_gridded. Attributes (COMID = cell) and Q'
(divide_id = cell, split across pieces by pieces_per_row_divisor) need no
change; the gauge CSV's `cell` column is a serde alias of COMID. Exclusive
with geospatial_fabric / explicit adjacency; params.subdivision.enabled is
rejected with it. No engine port: DDR's raster/regridding products are
consumed as-is.

Two bugs the gridded path exposed, both fixed:
- GageSubgraph::upstream_comids was per-piece on subdivided stores, so the
  summed-Q' baseline counted each divide m times (also MERIT subdivision).
- DDR's gridded Q' stores are Qr(time, divide_id); zarrs returns fill values
  for an out-of-range subset, so the fixed-axis reader silently read NaN.
  StreamflowStore now sniffs the axis order (dimension_names, shape fallback,
  refuses neither); fixture tests/fixtures/qr_daily_time_major.ic.

examples/juniata_gridded/: DDR's bundle byte-identical + statistics sidecar
+ ddrs.yaml + README. Baseline NSE 0.594 / KGE 0.672 (DDR 0.594 / 0.671);
routed 0.751 / 0.730 at seed 42 (seeds {42,7,123}: 0.744–0.758 / 0.728–0.732;
DDR 0.795 / 0.737 — RNG windows + ddrs's per-cell KAN head). CONUS:
config/sources/conus-gridded.yaml, config/experiments/gridded_conus.yaml,
scripts/snap_gridded_gauges.py (620 gauges, DDR's population); plan builds
5,198 cells → 91,867 sub-reaches in 0.6 s, baseline median NSE 0.354.

Tests: gridded_bundle (never skips), gridded_acceptance (release floors),
unit tests for the reader, config guards, CSV alias, upstream_comids dedup,
time-major fixture. Gates: cargo test 607/0, compare_ddr_sandbox ABSOLUTE
MATCH, juniata_acceptance unchanged (0.7903 / 0.8810 / 0.6947).

Docs: spec (with outcome), CLAUDE.md, ddrs-dev skill (change→gate row,
config guards, traps T11 time-major Qr / T12 CUDA 13.3 vs cudarc),
docs/usage/inputs-reading.md, nh-qprime-store-contract.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqnrtmU91yzuVpGnADnXLP
…aims

New skill `.claude/skills/ddrs-run/`: the runbook for operating ddrs, split
from ddrs-dev (build/code/test) and ddrs-eval-plots (visualize). SKILL.md
carries the judgment calls — a three-step preflight, training recipes,
dataset switching, run tracking, checkpoint resume, and what does not work.
references/commands.md is the flag-by-flag reference for all eight
subcommands, the workspace layout, and exit codes.

Every claim was verified against the binary or the source on 2026-09-09,
which turned up four things the docs had wrong or missing:

- Text-mode `ddrs show` never prints metrics; `--json` is required. The
  ddrs-eval-plots skill recommended `ddrs show <id> | grep -E "nse|kge"`,
  which silently returns nothing on a good run. Corrected there too.
- `<run_dir>/baseline/manifest.json` metrics are per-gauge ARRAYS, not
  reduced medians, so comparing them to the run manifest's
  `median_nse_finite` needs an explicit median. Snippet included and tested
  (gridded Juniata: routed 0.751/0.730 vs baseline 0.594/0.672).
- `src/config.rs` sets `deny_unknown_fields` on TWO structs
  (`DisaggregationSection` and `Subdivision`), not one as ddrs-dev's config
  reference claimed. Corrected there.
- `ddrs sources` resolves groups at `<config dir>/config/sources/`, so
  `--config config/experiments/x.yaml` reports "no groups" and its runs
  never get a data-source segment in the run id.

The stale-binary trap is documented with the misleading error it actually
produces, reproduced here: a pre-gridded `~/.cargo/bin/ddrs` rejects a valid
`gridded_network:` config with "adjacency sources are missing".

ddrs-dev: Route table entry for the new skill; the maintenance note now says
three skills rather than two.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqnrtmU91yzuVpGnADnXLP
…n run

A subagent given only the skill produced a correct runbook but had to leave it
to read source for five things. All now verified and documented:

- The committed starting configs (config/merit_training.yaml,
  config/experiments/gridded_conus.yaml, the two examples/) were never named,
  so the only route to a gridded config looked like hand-building one.
- `ddrs plan`'s bootstrap offers exactly two options and its "clean template"
  is hardcoded to config/merit_training.yaml (plan_bootstrap.rs), so
  bootstrapping while expecting gridded silently yields MERIT.
- `geodataset:` is validated by nothing. It is a provenance label written into
  zarr attrs and the run's config snapshot, so a stale value after
  `sources use` produces a successful run with a mislabelled audit trail.
- A resume mints a NEW run id: run_id is built from the invocation's own start
  timestamp (run.rs), so the interrupted run's manifest, log and checkpoints
  freeze at the crash and one training curve spans two run dirs.
- `run` never re-prints the baseline table that `plan` shows.

Also notes that `cargo install` rewrites the machine-global ~/.cargo/bin/ddrs
shared across worktrees, so with several branches in play an absolute
target/release/ddrs is the safer invocation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqnrtmU91yzuVpGnADnXLP
…un skill

All three were defects the docs had merely described. Fixed in code, and the
docs that described them updated to match.

1. `ddrs show` printed no metrics in text mode. The numbers were only in
   --json, so `ddrs show <id> | grep nse` came back empty on a healthy
   train-and-test run and read like a failure (ddrs-eval-plots recommended
   exactly that grep). Text mode now prints the metrics block, and for a
   train-and-test run reduces <run_dir>/baseline/manifest.json, whose
   metrics are per-gauge ARRAYS rather than medians, to a median per metric
   shown beside the routed value and the delta, flagging the case where
   routed does not beat baseline. That comparison previously required
   hand-parsing two JSON files and knowing to take the median.

2. `geodataset:` was validated by nothing. It is provenance only (zarr attrs
   plus the run's config.yaml snapshot), which is what made a wrong value
   dangerous: the run succeeds and its audit trail names the wrong network.
   `ddrs sources use` swaps only the data_sources block, so switching
   MERIT to gridded or back left a stale label behind. An absent value is
   now INFERRED from the adjacency source, and a contradicting one is a load
   error. Explicit conus_adjacency/gages_adjacency stay unconstrained: a
   pre-built store can be either network, so the label is undecidable.

3. A typo'd config key silently took its default, the documented single most
   common cause of "my config change did nothing". All seven remaining
   config structs now set deny_unknown_fields (ConfigRaw, DataSources,
   Experiment, KanHeadConfigSection, LossConfig, ParamsRaw,
   TestingOverridesRaw; DisaggregationSection and Subdivision already had
   it), so a misspelling is a load error naming the key.

Tests: typo rejection in each of the five config sections, the mlp: alias
still parsing, geodataset inference/contradiction/undecidable, and
median_finite plus render_scalar units. The shared config test helper no
longer stamps `geodataset: merit`, which would contradict any gridded block.

Gates: cargo test 630 passed / 0 failed (101 binaries), compare_ddr_sandbox
ABSOLUTE MATCH, juniata_acceptance 0.7903/0.8810 and gridded_acceptance
0.7511/0.7299, both identical to pre-fix, so the routing path is untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqnrtmU91yzuVpGnADnXLP
The gridded (DDM30) tests existed but nothing ran them automatically:
gridded_acceptance self-skips under debug_assertions, so the debug `test`
job never touched it, and the pre-push hook only ran ddr_sandbox_match.

- .githooks/pre-push now runs ddr_sandbox_match + gridded_bundle in one
  cargo invocation, so the build is shared. Both never skip (the sandbox
  fixture and the Juniata gridded bundle are committed), so a broken
  checkout fails locally rather than in CI. Measured 0.43 s warm, which
  keeps the hook cheap enough to leave enabled.
- The release `acceptance` job gains a "Gridded (DDM30) end-to-end
  acceptance" step alongside the Juniata one. It reuses that job's release
  build and the test itself is ~10 s, so the added CI cost is small next to
  the build it shares.

CLAUDE.md and the ddrs-dev testing reference updated to name both jobs and
both hook tests, including why the debug job cannot cover either acceptance
test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqnrtmU91yzuVpGnADnXLP
taddyb added a commit that referenced this pull request Sep 11, 2026
89 runs and ~20 experiment bundles exist under .ddrs/, and the reasoning
behind almost all of them lived only in conversation history, which gets
compacted away. CLAUDE.md already required committing that knowledge; nothing
enforced it and nothing made it cheap.

docs/journal/YYYY-MM.md is append-only and has two parts: a ledger with one
row per completed run or experiment (smoke tests and shards included, so no
run is lost), and entries carrying four judgement fields -- Question, What we
did, Result, Conclusion.

The split is the design. scripts/journal.py scrapes everything a manifest
knows (metrics, git sha, dirty flag, epochs, gauge count, wall time) and
writes _TODO_ for everything it cannot know. It never invents a hypothesis,
and --mode backfill writes ledger rows only: the reasoning behind a run from
two months ago is not recoverable, and one confident guess would stop the
record being evidence. 224 historical records were backfilled that way.

Hooks in .claude/settings.json: PostToolUse on ddrs run / ddrs experiment
writes the record, Stop blocks only on stubs the current session opened (a
session with no context would be pressured into inventing one), SessionStart
reports runs finished outside a session and writes nothing. The script is
stdlib-only with no git subprocess, is regex-gated so unrelated Bash calls
return in ~30 ms, exits 0 on any internal error so it can never fail the
command it observes, and DDRS_JOURNAL_OFF=1 disables it.

No Rust, no build, no CI gate: src/, Cargo.toml and the invariants are
untouched. scripts/test_journal.py covers idempotency, smoke classification,
that judgement is never auto-written, and that the Stop nag is session-scoped
and cannot loop.

The one hand-written entry describes the gridded DDM30 port and links to a
spec that lands with PR #40; that link resolves when #40 merges.

Design, rejected alternatives, blast radius and known gaps:
docs/superpowers/specs/2026-09-10-research-journal-design.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqnrtmU91yzuVpGnADnXLP
@taddyb
taddyb merged commit 571e2f6 into master Sep 11, 2026
4 checks passed
@taddyb
taddyb deleted the worktree-gridded-routing branch September 11, 2026 02:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant