fix: config cascade applies on run_app path

run_app never called config::setup(), so the global cascade was empty
and every from_cascade() subsystem (governor, worker pool, batch engine,
scaling) silently took defaults regardless of the app's config -- the
recurring "set a cascade value, nothing happens". Now run_app populates
the cascade (guarded by try_get().is_none()), a --config file is ingested
via the new ConfigOptions.config_file (the dormant to_config_options is
fixed), and present-but-malformed sections WARN instead of silently
defaulting. ServiceRuntime logs which platform sections were found vs
defaulted.

BEHAVIOUR: apps' worker_pool / self_regulation / batch_processing /
scaling sections are now HONOURED (were silently defaulted) -- review
those sections on the bump.

Also adds ScalingSignalsCell::set_custom + a custom.<name> CEL namespace
so apps push DOMAIN scaling signals (cloud-API backlog, ClickHouse
backlog) for Tier-3 pressures; unknown-at-load custom refs warn + keep
(runtime fallback), unknown top-level idents still hard-reject.

Tests: config_cascade_honoured, run_app_populates_config,
from_cascade_deser_error, + custom-signal end-to-end.

Publish: true

Author: catinspace-au

docs/MIGRATIONS.md

@@ -10,6 +10,101 @@ six core DFE apps migrate in lockstep.
 
 ---
 
+## 2.8.11 -- config cascade ACTUALLY applies on the run_app path
+
+A `fix:`. Before this release, [`run_app`](../src/cli/app.rs) built the
+`ServiceRuntime` (governor, worker pool, batch engine, scaling) entirely
+from `*::from_cascade()` WITHOUT ever calling `config::setup()`, so the
+global config singleton was empty and every platform subsystem silently
+took its hard-coded defaults regardless of the app's config file. A
+`--config <file>` was also never ingested (the cascade discovers config
+in DIRECTORIES; a file path pushed into the directory list is never
+found). Both are now fixed.
+
+### Platform sections are now HONOURED (BEHAVIOUR CHANGE)
+
+Apps' platform sections in their config file -- `worker_pool`,
+`self_regulation`, `batch_processing`, `scaling`, `expression`, and any
+other `from_cascade()` section -- are now APPLIED. They were silently
+DEFAULTED before.
+
+- **Action required on the bump:** REVIEW these sections in every app's
+  config. Any stale or unintended value that was harmlessly ignored will
+  now take effect. e.g. a leftover `worker_pool: { min_threads: 64 }`
+  that previously did nothing will now actually size the pool.
+- No API removal, no signature change -- existing apps recompile
+  unchanged. The change is purely that config you set now does what it
+  says.
+
+### `run_app` populates the cascade (guarded)
+
+The `Run` arm of `run_app` now calls `config::setup()` with the app's
+`env_prefix`, `app_name`, and the `--config` file BEFORE `load_config()`
+and the `ServiceRuntime` build. Guarded by `config::try_get().is_none()`
+so apps that already call `setup()` / `setup_async()` themselves (e.g.
+for the Postgres config layer) are NOT double-initialised.
+
+### New `ConfigOptions.config_file` (additive)
+
+`ConfigOptions` gains `config_file: Option<PathBuf>` (default `None`).
+When set, the named YAML file merges ABOVE the discovered
+`defaults`/`settings`/`settings.{env}` files but BELOW the PostgreSQL
+layer (async path) and BELOW environment variables -- an explicit
+override that still yields to ENV. `CommonArgs::to_config_options` now
+sets this from `--config` instead of (wrongly) pushing it into
+`config_paths`.
+
+### Observable startup + deser WARN
+
+- `ServiceRuntime::build` now emits one startup `tracing::info!` line
+  summarising which platform sections were found in the cascade vs
+  defaulted (`cascade_initialised`, `self_regulation`, `worker_pool`,
+  `batch_processing`, `scaling`, `expression`). The previously-silent
+  "defaulted everything" failure is now visible.
+- New `Config::unmarshal_key_or_warn` /
+  `unmarshal_key_registered_or_warn`: a config section that is PRESENT
+  but malformed (typo / type mismatch) now logs a `tracing::warn!` and
+  falls back to the default, instead of silently swallowing the error.
+  Wired into the four `ServiceRuntime`-build readers: governor,
+  worker-pool, batch-engine, scaling. An ABSENT key is unchanged --
+  still silent default (the default-ON governor relies on this).
+  (`ScalingEngineConfig::from_cascade` shares the `scaling` key with
+  `ScalingPressureConfig`, which already warns for that section, so it
+  was left as-is to avoid a double WARN.)
+
+### Custom / domain scaling signals (additive)
+
+The scaling engine's CEL pressures could previously reference ONLY the
+8 FIXED transport signals. Apps can now push DOMAIN signals so a
+`scaling.pressures` expression can scale on them -- essential for
+non-rustlib-inbound apps (e.g. the fetcher, whose smart default is
+otherwise CPU-only).
+
+- **New `ScalingSignalsCell::set_custom(&self, name: &str, value: f64)`**
+  -- insert/overwrite a named domain signal (e.g. cloud-API pending
+  fetch backlog / API throttle, ClickHouse insert backlog). Pushed from
+  the app at runtime; no setter per signal.
+- **New `TransportSignals.custom: BTreeMap<String, f64>`** (default
+  empty), populated by `ScalingSignalsCell::snapshot()`. Existing
+  `TransportSignals { .. }` struct literals that enumerate all fields
+  without `..Default::default()` must add the `custom` field (or switch
+  to `..Default::default()`); all setters and the typed fields are
+  unchanged.
+- **CEL surface:** domain signals are exposed under a DEDICATED
+  `custom.<name>` map, SEPARATE from the strict fixed-signal `metrics`
+  map. Scale on them like `custom.clickhouse_backlog / params.ch_target`.
+- **Validation contract (no API change, behaviour note):** custom names
+  are unknown at config-load, so the load-time dry-run cannot
+  pre-populate them. Syntax errors and unknown TOP-LEVEL identifiers are
+  still HARD-rejected at load. A reference to a `custom.<name>` (or any
+  map key absent at load) is downgraded to a load `warn!` and KEPT; the
+  runtime guard falls back to last-good / smart-default if it errors at
+  tick time. Startup never fails for a `custom.*` reference.
+- No API removal, no signature change -- apps recompile unchanged unless
+  they hit the `TransportSignals` struct-literal note above.
+
+---
+
 ## 2.8.10 -- horizontal scaling pressure + metrics overhaul
 
 A `fix:` (the existing scaling was not fit for purpose). Adds the

docs/deployment/KEDA.md

@@ -149,7 +149,9 @@ trigger, and the job of `ScalingEngine`.
   `{ns}_scaling_pressure{name="default"}` = `max(CPU, inbound)` gated by
   circuit-open, on a periodic tick.
 - **Tier 3 -- your correlated composite (config):** define CEL
-  expression(s) over the local context.
+  expression(s) over the local context, including app-pushed DOMAIN
+  signals (`custom.<name>`, see Wiring) -- e.g. a fetcher's cloud-API
+  pending-fetch backlog, a loader's ClickHouse insert backlog.
 
 ### The smart default
 
@@ -181,11 +183,21 @@ scaling:
 
 CEL context: top-level `cpu_utilisation_ratio`, `circuit_open`,
 `transport_inbound_pressure_ratio`, `transport_outbound_pressure_ratio`,
-`memory_ratio`; `params.<key>`; `metrics.<signal>`
-(`kafka_assigned_lag`, `inflight`, `shed_rate`, ...). CEL has no `max()`
--- use a ternary. Expressions are validated at LOAD (syntax + an
-unknown-identifier dry-run); a broken expression falls back to the smart
-default with a loud, operator-facing error rather than failing startup.
+`memory_ratio`; `params.<key>`; the FIXED transport `metrics.<signal>`
+(`kafka_assigned_lag`, `inflight`, `shed_rate`, ...); and app-pushed
+DOMAIN signals under a SEPARATE `custom.<name>` map (rustlib 2.8.11).
+CEL has no `max()` -- use a ternary.
+
+Validation at LOAD: the expression is COMPILED (syntax errors and
+unknown TOP-LEVEL identifiers are hard-rejected -> fall back to the
+smart default with a loud, operator-facing error). A reference to a
+`custom.<name>` (or any map key not present at load) is NOT a hard
+error -- those domain signals are pushed at RUNTIME, so a load-time
+dry-run cannot pre-populate them. Such a reference is kept with a load
+`warn!`; if it still errors at tick time (signal never pushed), that
+single pressure falls back to its last-good value, then to the smart
+default. The runtime guard is the safety net -- startup never fails for
+a `custom.*` reference.
 
 ### Multi-output
 
@@ -224,9 +236,36 @@ runtime.scaling_signals.set_kafka_assigned_lag(lag as f64);
 runtime.scaling_signals.set_circuit_open(breaker.is_open());
 ```
 
+For the 8 FIXED transport signals there is a typed setter (above). For a
+DOMAIN signal -- anything rustlib cannot know (a cloud-API backlog, an
+upstream throttle, a downstream insert backlog) -- push it by name with
+`set_custom` and reference it in a pressure as `custom.<name>`:
+
+```rust
+// fetcher: cloud-API pending-fetch backlog + provider throttle
+runtime.scaling_signals.set_custom("pending_fetch", queue.len() as f64);
+runtime.scaling_signals.set_custom("api_throttle", throttle_ratio);
+```
+
+```yaml
+scaling:
+  params:
+    fetch_target: 500        # PER-POD backlog one pod tolerates
+  pressures:
+    - name: fetch
+      # CEL has no max() -- ternary picks the worse of backlog vs throttle
+      expression: >
+        custom.pending_fetch / params.fetch_target > custom.api_throttle
+        ? custom.pending_fetch / params.fetch_target
+        : custom.api_throttle
+```
+
 If your inbound is NOT a rustlib transport (e.g. cloud-API polling), the
 compound inbound is 0 and the default reduces to CPU-only -- add a
-DOMAIN term (Tier 3) for your real backlog signal.
+DOMAIN term (Tier 3) via `set_custom` for your real backlog signal. A
+loader with a ClickHouse sink is the same story: push the insert backlog
+with `set_custom("clickhouse_backlog", n)` and scale on
+`custom.clickhouse_backlog / params.ch_target`.
 
 ### Emit your scaling signals as metrics
 

src/cli/app.rs

@@ -216,6 +216,30 @@ pub async fn run_app<A: DfeApp>(app: A) -> Result<(), CliError> {
                 "starting service"
             );
 
+            // Populate the global config cascade BEFORE load_config + the
+            // ServiceRuntime build, so every `from_cascade()` subsystem
+            // (governor, worker pool, batch engine, scaling) reads the app's
+            // real config instead of silently defaulting. Guarded by the
+            // try_get() check so apps that already call config::setup()
+            // themselves (e.g. those needing setup_async for Postgres) are not
+            // double-initialised -- setup() returns Err(AlreadyInitialised)
+            // otherwise.
+            #[cfg(feature = "config")]
+            if crate::config::try_get().is_none() {
+                let opts = crate::config::ConfigOptions {
+                    env_prefix: app.env_prefix().to_string(),
+                    app_name: Some(app.name().to_string()),
+                    config_file: args.config.as_deref().map(std::path::PathBuf::from),
+                    ..Default::default()
+                };
+                if let Err(e) = crate::config::setup(opts) {
+                    tracing::warn!(
+                        error = %e,
+                        "config cascade setup failed; from_cascade subsystems will use defaults"
+                    );
+                }
+            }
+
             let config_path = args.config.as_deref();
             let config = app.load_config(config_path)?;
 

src/cli/args.rs

@@ -109,17 +109,20 @@ impl CommonArgs {
     }
 
     /// Convert to `ConfigOptions` for use with `config::setup()`.
+    ///
+    /// The `--config <path>` flag names a FILE, so it populates
+    /// [`ConfigOptions::config_file`](crate::config::ConfigOptions::config_file)
+    /// -- NOT `config_paths`, which is a list of DIRECTORIES to search for the
+    /// standard base names. (Before 2.8.11 this wrongly pushed the file path
+    /// into `config_paths`, where directory discovery never found it.)
     #[cfg(feature = "config")]
     #[must_use]
     pub fn to_config_options(&self, env_prefix: &str) -> crate::config::ConfigOptions {
-        let mut opts = crate::config::ConfigOptions {
+        crate::config::ConfigOptions {
             env_prefix: env_prefix.to_string(),
+            config_file: self.config.as_deref().map(std::path::PathBuf::from),
             ..Default::default()
-        };
-        if let Some(ref path) = self.config {
-            opts.config_paths.push(path.into());
         }
-        opts
     }
 }
 
@@ -166,6 +169,43 @@ mod tests {
         assert_eq!(args.effective_log_level(), "error");
     }
 
+    #[cfg(feature = "config")]
+    #[test]
+    fn test_to_config_options_sets_config_file_not_paths() {
+        let args = CommonArgs {
+            config: Some("/etc/svc/config.yaml".to_string()),
+            log_level: "info".to_string(),
+            log_format: "auto".to_string(),
+            metrics_addr: "0.0.0.0:9090".to_string(),
+            verbose: false,
+            quiet: false,
+        };
+        let opts = args.to_config_options("MY_SVC");
+        assert_eq!(opts.env_prefix, "MY_SVC");
+        // The file path lands in config_file, NOT config_paths (the 2.8.11 fix).
+        assert_eq!(
+            opts.config_file,
+            Some(std::path::PathBuf::from("/etc/svc/config.yaml"))
+        );
+        assert!(opts.config_paths.is_empty());
+    }
+
+    #[cfg(feature = "config")]
+    #[test]
+    fn test_to_config_options_no_config_file_when_absent() {
+        let args = CommonArgs {
+            config: None,
+            log_level: "info".to_string(),
+            log_format: "auto".to_string(),
+            metrics_addr: "0.0.0.0:9090".to_string(),
+            verbose: false,
+            quiet: false,
+        };
+        let opts = args.to_config_options("MY_SVC");
+        assert!(opts.config_file.is_none());
+        assert!(opts.config_paths.is_empty());
+    }
+
     #[test]
     fn test_effective_log_level_custom() {
         let args = CommonArgs {

src/cli/runtime.rs

@@ -281,6 +281,11 @@ impl ServiceRuntime {
             .check_on_startup();
         }
 
+        // Turns the previously-silent "from_cascade defaulted everything"
+        // failure into one observable startup line.
+        #[cfg(feature = "config")]
+        log_cascade_section_summary();
+
         // Log runtime context
         tracing::info!(
             environment = %ctx.environment,
@@ -357,6 +362,25 @@ impl ServiceRuntime {
     }
 }
 
+/// Emit one startup line summarising which platform config sections were found
+/// in the cascade vs defaulted. Cheap (key-presence checks, no deserialisation).
+/// This is the observable counterpart to the silent pre-2.8.11 failure where
+/// `from_cascade` defaulted everything because the cascade was never populated.
+#[cfg(feature = "config")]
+fn log_cascade_section_summary() {
+    let cfg = crate::config::try_get();
+    let present = |key: &str| cfg.is_some_and(|c| c.contains(key));
+    tracing::info!(
+        cascade_initialised = cfg.is_some(),
+        self_regulation = present("self_regulation"),
+        worker_pool = present("worker_pool"),
+        batch_processing = present("batch_processing"),
+        scaling = present("scaling"),
+        expression = present("expression"),
+        "Config cascade sections (true = found in config, false = using defaults)"
+    );
+}
+
 /// Periodic scaling-pressure tick: sample CPU (rate of the cumulative counter
 /// over the wall window / cores), read the pushed transport signals, and let the
 /// engine evaluate + publish its gauges. Off the data hot-path (interval-driven).