Gridded (ISIMIP DDM30) routing as a data source, the ddrs-run skill, and three silent-failure fixes

.claude/skills/ddrs-dev/SKILL.md

@@ -27,6 +27,7 @@ config reference, test-authoring patterns, and the current research status.
 | Diagnose a failed, hung, or wrong-result run | `references/traps.md` §Symptom → trap |
 | Check whether a question is already settled | `references/research-status.md` |
 | Build or change the training gauge CSV / population | `references/gauge-population.md` |
+| Launch / watch / resume / audit a training run, or use any `ddrs` subcommand | skill `ddrs-run` |
 | Plot or interpret eval output | skill `ddrs-eval-plots` |
 
 ## Contents
@@ -114,6 +115,7 @@ Full commands and rationale in `references/testing.md`.
 | `config/**/*.yaml` only | **D** | `ddrs plan --config … --workspace …` exits 0, no drift |
 | `.github/workflows/ci.yml`, `.githooks/**` | — | push to a branch and let the PR run vet it; there is no local gate for workflow changes |
 | `examples/juniata/**` (bundle, config, README) | — | `cargo test --test juniata_bundle` (never skips — bundle is committed), `cargo test --release --test juniata_acceptance` (end-to-end metric floors), and `ddrs --config examples/juniata/ddrs.yaml plan` exits 0 from the repo root |
+| `src/adjacency/gridded.rs`, `resolve_or_build_gridded`, `examples/juniata_gridded/**`, the `Qr` axis-order sniff in `src/data/store/icechunk.rs` | **C** | `cargo test --test gridded_bundle` (never skips), `cargo test --test hourly_streamflow` (time-major fixture), `cargo test --release --test gridded_acceptance`, then Tier C |
 | Plotting / analysis scripts only | — | no gate |
 | `epochs`, `learning_rate`, `batch_size`, loss weights within documented ranges | — | no gate |
 
@@ -209,9 +211,49 @@ Two deviations from DDR's bundle: `data/statistics/*.json` is **committed**
 `ddrs.yaml` ignore. Regenerate the bundle in the ddr repo
 (`extract_bundle.py`), then re-copy `data/` plus the generated statistics JSON.
 
+## Gridded (ISIMIP DDM30) routing — `examples/juniata_gridded/` (2026-09-09)
+
+ddrs consumes DDR PR #194's gridded products as a **data source**; it ports
+none of the raster/regridding engine. `data_sources.gridded_network` points at
+DDR's sub-reach adjacency zarr (`order`, `parent_cell`, `indices_0/1`,
+`length_m`, `slope`) and `ddrs plan` relabels it into the existing subdivided
+store layout (parent = cell, pieces = the cell's contiguous sub-reaches, gauge
+read at the last piece) via `adjacency::gridded::GriddedNetwork` +
+`cache::resolve_or_build_gridded`. Attributes and Q′ are keyed on the cell id,
+so `parent_order` lookups and `pieces_per_row_divisor` (Q′/k) need no change.
+The gauge CSV's `cell` column is a serde alias of `COMID`. Guards: exclusive
+with `geospatial_fabric` and the explicit zarr pair; `params.subdivision.enabled`
+is rejected with it. Spec:
+`docs/superpowers/specs/2026-09-08-ddrs-gridded-routing-design.md`.
+
+```bash
+target/release/ddrs --config examples/juniata_gridded/ddrs.yaml plan
+target/release/ddrs --config examples/juniata_gridded/ddrs.yaml run --workflow train-and-test --backend cpu
+```
+
+Verified 2026-09-09 (4 cells / 27 sub-reaches, 30 epochs, CPU, ~15 s): baseline
+NSE 0.594 / KGE 0.672 (DDR 0.594 / 0.671); routed NSE 0.751 / KGE 0.730 at seed
+42, seeds {42, 7, 123} span 0.744–0.758 / 0.728–0.732 (DDR 0.795 / 0.737 — RNG
+windows plus ddrs's per-cell KAN head vs DDR's per-sub-reach `log10_uparea`).
+CONUS: `config/sources/conus-gridded.yaml` + `config/experiments/gridded_conus.yaml`
+(620 gauges from `scripts/snap_gridded_gauges.py`, DDR's population); `plan`
+builds 5,198 cells → 91,867 sub-reaches in 0.6 s, baseline median NSE 0.354 /
+KGE 0.505 over 1995-10-01..1997-09-30 (DDR's 0.390 includes its `da_ratio`
+output correction, which ddrs does not apply).
+
+Two things this work found, both fixed: (1) `GageSubgraph::upstream_comids`
+returned one entry per piece on subdivided stores, so the summed-Q′ baseline
+counted each divide `m` times — now deduplicated by id (also affected MERIT
+`params.subdivision` baselines); (2) DDR's gridded Q′ stores are
+`Qr(time, divide_id)`, and zarrs returns fill values for an out-of-range subset
+instead of erroring, so the old fixed-axis reader silently read NaN — the
+reader now sniffs the axis order (traps.md T11).
+
 ## Maintenance
 
-This skill and `ddrs-eval-plots` are the only two skills in this repo. When a run,
+Three skills live in this repo: this one (build / code / configure / test / debug),
+`ddrs-run` (launch, watch, resume, and audit runs; the full `ddrs` command
+reference), and `ddrs-eval-plots` (visualize and interpret run output). When a run,
 eval, or export completes, update the relevant section here in the same session that
 produced the knowledge — do not leave it only in a findings doc. If a rule here is
 superseded, correct it in place with the new nuance rather than deleting it.

.claude/skills/ddrs-dev/references/build-and-env.md

@@ -25,6 +25,13 @@ Verified against `Cargo.toml` and source on 2026-07-30.
 
 If a system HDF5 leaks into the static build: `unset HDF5_DIR; unset NETCDF_DIR`.
 
+- **CUDA newer than `cudarc`'s table** (13.3.1 on this host since 2026-09 vs
+  cudarc 0.19.7's max 13.2): the build script panics with `Unsupported cuda
+  toolkit version`. Set `CUDARC_CUDA_VERSION=13020` — a gitignored
+  `.cargo/config.toml` with `[env] CUDARC_CUDA_VERSION = "13020"` is the
+  per-checkout fix (traps.md T12). Cached `target/` dirs mask it until the
+  build script reruns.
+
 ## Fork pins
 
 All **13** `burn-*` crates must resolve from `github.com/taddyb/burn` branch

.claude/skills/ddrs-dev/references/config.md

@@ -4,9 +4,13 @@ Struct: `src/config.rs::Config`. Loaded via
 `Config::from_yaml_file_with_mode(path, ConfigMode::Training|Testing)`.
 Six top-level sections. Verified against source 2026-07-30.
 
-**No `deny_unknown_fields` except `DisaggregationSection`** (added 2026-08-03) —
-everywhere else a typo'd key silently takes its default instead of erroring. This
-is the single most common cause of "my config change did nothing".
+**Every section sets `deny_unknown_fields`** as of 2026-09-09 (`ConfigRaw`,
+`DataSources`, `Experiment`, `KanHeadConfigSection`, `LossConfig`, `ParamsRaw`,
+`TestingOverridesRaw`, plus `DisaggregationSection` and `Subdivision`, which had it
+already), so a typo'd key is a load error naming the key. Historically only the last
+two had it and everywhere else a typo silently took its default, which was the
+single most common cause of "my config change did nothing". On a binary older than
+2026-09-09, that silence is still the first thing to suspect.
 
 ## Contents
 
@@ -37,10 +41,14 @@ to §`params:` if you are looking for `tau` (it is not a routing sub-step count)
 (feature-concatenated on COMID, NaN-filled, `deserialize_one_or_many_paths`).
 An empty list is a hard error.
 
-Adjacency rule: provide **either** both `conus_adjacency` + `gages_adjacency`,
-**or** `geospatial_fabric` (managed build into `.ddrs/adjacency/<key>/`).
-Exactly one of the pair ⇒ error; neither source ⇒ `"adjacency sources are missing"`.
-For multi-layer gpkg set `geospatial_fabric_layer` (participates in the cache key).
+Adjacency rule: provide **exactly one** of (a) both `conus_adjacency` +
+`gages_adjacency`, (b) `geospatial_fabric` (MERIT managed build into
+`.ddrs/adjacency/<key>/`), (c) `gridded_network` (DDR's DDM30 sub-reach
+adjacency zarr, managed build via `resolve_or_build_gridded`; added 2026-09-09).
+Exactly one of the pair ⇒ error; neither source ⇒ `"adjacency sources are missing"`;
+`gridded_network` with (a) or (b) ⇒ error. For multi-layer gpkg set
+`geospatial_fabric_layer` (participates in the cache key). `params.subdivision.enabled`
+is rejected alongside `gridded_network` (the store is already split by DDR).
 
 `aorc_precip` is required whenever `kan_head.disaggregation:` is present — see below.
 
@@ -177,6 +185,10 @@ Four validators run at `Config::from_yaml_file`, plus one at dataset open.
 | `validate_data_sources` | one of the adjacency pair | `` "`gages_adjacency` is missing" `` |
 | | neither adjacency nor fabric | `"adjacency sources are missing"` |
 | | `geospatial_fabric_layer` on a non-gpkg | `"geospatial_fabric_layer"` + `".gpkg"` |
+| | `gridded_network` + `geospatial_fabric` | `"gridded_network"` + `"geospatial_fabric"` |
+| | `gridded_network` + explicit adjacency pair | `"gridded_network"` + `"conus_adjacency"` |
+| `validate_subdivision` | `subdivision.enabled: true` + `gridded_network` | `"params.subdivision"` + `"gridded_network"` |
+| `validate_geodataset` | `geodataset:` contradicting the adjacency source (`ddm30` with `geospatial_fabric`, `merit` with `gridded_network`) | `"geodataset"` + the source key. Absent ⇒ inferred; explicit adjacency paths ⇒ any label allowed |
 | `validate_leakance` | `use_leakance` + `use_cuda_graphs` | both key names |
 | `validate_ddr_match` | `use_cuda_graphs: true` without the deprecated `ddr_match: true` | `"use_cuda_graphs: true` requires the DEPRECATED `ddr_match: true"` |
 | `validate_disagg_pretrained` | `freeze: true` without `pretrained_checkpoint` | `"freeze: true requires pretrained_checkpoint"` |

.claude/skills/ddrs-dev/references/testing.md

@@ -48,6 +48,15 @@ debug_assertions** (a debug train takes minutes), so it only runs with
 readers when you want end-to-end confirmation. Verified 2026-09-02:
 NSE 0.7903 / KGE 0.8810 / baseline 0.6947 in 18.3 s.
 
+```bash
+cargo test --release --test gridded_acceptance -- --nocapture   # ~30 s after build
+```
+
+The gridded (DDM30) twin on `examples/juniata_gridded/`: floors NSE ≥ 0.70 /
+KGE ≥ 0.69 (seed-42 reference 0.751 / 0.730; seeds {42, 7, 123} span
+0.744–0.758 / 0.728–0.732), baseline NSE ∈ [0.57, 0.62] (deterministic 0.594,
+matches DDR). The baseline band is the one that catches a wrong `Qr` axis order.
+
 ### Tier B — KAN head
 `src/nn/`, `Cargo.toml` rskan tag
 
@@ -77,12 +86,12 @@ Exit 0, no drift warnings.
 
 `.github/workflows/ci.yml`, on every PR and master push (no path filters:
 required checks must never be skipped): job `test` = debug
-`cargo test --features fixtures --no-fail-fast` (13 min warm, 12–17 min
-cold); job `acceptance` = release `compare_ddr_sandbox` +
-`juniata_acceptance` (17 min warm, ~25 min cold). Warm is only modestly
-faster because `rust-cache` keeps dependency artifacts only: the ddrs
-crate and every test binary rebuild on each run (measured 2026-09-04,
-runs 33817851139 cold and 33830503587 warm). The acceptance job
+`cargo test --features fixtures --no-fail-fast` (~12–17 min cold; warm is
+shorter); job `acceptance` = release `compare_ddr_sandbox` +
+`juniata_acceptance` + `gridded_acceptance` (~25 min cold; warm is shorter;
+both acceptance tests self-skip under `debug_assertions`, so the debug `test`
+job never runs them — the release job is their only CI coverage). The
+acceptance job
 builds with `CARGO_PROFILE_RELEASE_LTO=false` (env override in the workflow
 only): the thin-LTO link of the test binary was 30 of 40 minutes on the
 2-core runner while the 30 training epochs took ~13 s, so the training is
@@ -97,16 +106,19 @@ local tier gates.
 
 Both jobs install a CUDA toolkit only because compilation requires it
 (build-and-env.md). Local hook: `git config core.hooksPath .githooks`
-enables `.githooks/pre-push` (runs `ddr_sandbox_match`; bypass with
-`git push --no-verify`).
+enables `.githooks/pre-push`, which runs `ddr_sandbox_match` +
+`gridded_bundle` in one cargo invocation (~0.4 s warm; both never skip, so a
+broken checkout fails locally rather than in CI). Bypass with
+`git push --no-verify`.
 
 ## What covers what
 
 | Area | Tests |
 |---|---|
 | Routing | `mmc`, `routing_utils`, `geometry` |
 | DDR parity (invariant 1) | `ddr_sandbox_match` (in-process, plain `cargo test`) + `compare_ddr_sandbox` example (diagnostics, exits 1 on mismatch) |
-| End-to-end acceptance | `juniata_acceptance` (release-only; metric floors + beats-baseline on the committed bundle) |
+| End-to-end acceptance | `juniata_acceptance`, `gridded_acceptance` (release-only; metric floors + beats-baseline on the committed bundles) |
+| Gridded (DDM30) ingestion | `gridded_bundle` (sub-reach store → subdivided layout, gauge at the cell's last piece, cache hit, dataset opens), `cargo test --lib adjacency::gridded` (synthetic-store validation), `hourly_streamflow::time_major_store_reads_identically_to_divide_major`, `cargo test --lib zarr::tests::upstream_comids_names_each_subdivided_parent_once` |
 | Sparse / autograd | `sparse_gradcheck`, `sp8_gradcheck` |
 | KAN head | the 4 `kan_head_*` fixture tests (need `--features fixtures`) |
 | Leakance | `leakance_gradcheck`, `leakance_off_parity`, `zeta_accum` |

.claude/skills/ddrs-dev/references/traps.md

@@ -18,6 +18,8 @@ is what a future session needs, not the narrative.
 | Mid-eval `object not found` from icechunk | T8 |
 | `.ddrs/` appeared somewhere unexpected | T9 |
 | Resumed run trains zero batches | T10 |
+| Baseline predicts ~0 (FHV −100 %) on a new Q′ store, no error | T11 |
+| Fresh checkout/worktree fails in `cudarc`'s build script: `Unsupported cuda toolkit version` | T12 |
 
 ---
 
@@ -195,6 +197,31 @@ Resume additionally requires `experiment.epochs > E` or the resumed run trains z
 batches. Stored weights are f16, so a resumed trajectory drifts slowly — expected,
 not a bug.
 
+## T11 — Time-major `Qr(time, divide_id)` read as fill values (fixed 2026-09-09)
+
+DDR's gridded (DDM30) `build_gridded_qprime.py` writes `Qr(time, divide_id)`;
+the MERIT stores and the contract are `Qr(divide_id, time)`. zarrs answers an
+out-of-range `retrieve_array_subset` with **fill values, not an error**, so the
+old fixed-axis `read_slab` returned NaN for every cell and the summed-Q′
+baseline scored NSE −0.655 with FHV −100 % — no message anywhere. `open` now
+decides the axis order from the array's `dimension_names`, falling back to its
+shape against the coordinate lengths, and refuses a shape matching neither
+(`detect_time_major`, `src/data/store/icechunk.rs`). Fixture:
+`tests/fixtures/qr_daily_time_major.ic`. A baseline that is *deterministic and
+matches DDR* is the check that catches this class of bug; that is why both
+acceptance tests band the baseline tightly.
+
+## T12 — CUDA toolkit newer than `cudarc` knows (2026-09-09)
+
+`cudarc 0.19.7` (via `cuda-version-from-build-system`) parses `nvcc --version`
+and panics on a version outside its table (max 13.2). This host moved to CUDA
+13.3.1 in 2026-09; the main tree's `target/` still carried the 13.2 build-script
+output, so only a fresh worktree/checkout hit it. Fix: `CUDARC_CUDA_VERSION=13020`
+(gitignored `.cargo/config.toml` `[env]` block, or export it). 13.3 is
+driver/ABI-compatible with 13.2 for the dynamic-loading path ddrs uses. The
+main tree will hit the same panic the next time its cudarc build script reruns
+(any `CUDA_*` env change or a `cargo clean`).
+
 ## Exit codes
 
 `src/cli/types.rs`: 0 Success · 1 Generic · 2 ConfigInvalid · 3 DataSourceMissing ·

.claude/skills/ddrs-eval-plots/SKILL.md

@@ -200,7 +200,11 @@ the first dual win on the dHBV2-UH store. Previous best was 0.7152 / 0.7106
 +0.037 NSE and −0.007 KGE). Full table and the 2026-07-30 KGE qualification:
 `ddrs-dev` → `references/research-status.md`.
 
-`ddrs show <id> | grep -E "nse|kge|loss"` is the cheap pre-plot triage.
+`ddrs show <id>` is the cheap pre-plot triage: since 2026-09-09 it prints the
+`metrics` block and, for a `train-and-test` run, the summed-Q' baseline median with
+the routed value and delta beside it, so "did it beat the baseline" is answered
+without opening a notebook. On a binary older than that, text mode printed no
+metrics at all and you need `--json`. See skill `ddrs-run`.
 
 ### Have the parameters converged?