feat: telemetry-backed adaptive Livy polling to avoid Fabric 429 throttling

[Raki (mdrakiburrahman)]

Closes #261

Why this change is needed

Fabric's REST front end enforces a quota per identity and returns HTTP 429 with a Retry-After. The adapter was a heavy consumer of it, so runs collapsed into sustained throttling and models ran extremely slowly.

Two causes:

1. Poll cost was linear in threads, and blind. Statements were polled on a fixed 0.3s → 1.5s interval per thread. A 40-minute model was polled ~1600 times; a 2-second model got the same schedule.
2. Backoff was thread-local. Each 429 slept only the thread that saw it. The other N−1 kept hammering, so the bucket never refilled.

Reproduced live: at 6 threads the old schedule spent 118 calls/min on polling alone. Reported in #261, with analysis in this comment.

How

• Throttle governor (always on) shared by all Fabric REST calls — Livy, MLV, shortcuts. Sliding-window limiter plus a shared Retry-After gate, so a 429 seen by any thread parks every thread. New api_calls_per_minute (default 150); 0 disables the limiter but keeps shared backoff.
• Elapsed-proportional poll schedule (always on, no config). Interval grows with elapsed time and with a per-SQL-shape runtime learned during the run, so long models are polled rarely and short ones still return promptly.
• Opt-in adaptive_polling (default false) drives scheduling from Spark task counters read via a monitor REPL, making poll cost constant in threads. Statements are tagged with a dbt-aligned jobGroup, which also improves Spark UI visibility.

Considerations for the reviewer:

• Telemetry is advisory only. Only an authoritative GET .../statements/{id} may resolve, complete or fail a statement. "All known jobs terminal" never means done — one SQL statement runs several Spark jobs and the group is empty before the first and between jobs.
• Learned runtimes are session-scoped and in-memory. Nothing is written to disk, nothing is keyed to an identity, and the store is cleared on cleanup.
• Retries never resubmit side-effecting SQL, and no wait is unbounded — everything respects statement_timeout.
• Alternatives rejected: a fixed longer interval (penalises short models), and Livy's progress field (Fabric's HC statement GET does not return one).

Correctness bugs found and fixed along the way — an ambiguous submit could execute DML twice, cancel() was a no-op, singleton poll backoff was unbounded, NaN/Infinity Retry-After caused a spin/hang, and capacity 429s at critical priority retried with no pause.

Test

npx nx run dbt-fabricspark:test --output-style=stream green locally, plus full CI (unit → live Fabric functional → local-e2e). 757 unit tests; every regression guard mutation-tested.

Heuristics payload — telemetry-driven backoff

One 300s INSERT, one prior sample this session, jitter pinned to 0. interval = base + elapsed × 0.12, converging on the learned ETA, capped at MAX_INTERVAL:

 elapsed  interval  reason            telemetry (tasks done/total)
     0.0      0.25  initial-probe     —
     0.2      2.12  learned-eta       0/400
     2.4      3.14  learned-eta       3/400
     5.5      4.64  learned-eta       7/400
    10.2      6.87  learned-eta      13/400
    17.0     10.17  learned-eta      22/400
    27.2     15.05  learned-eta      36/400
    42.2     22.28  learned-eta      56/400
    64.5     30.00  learned-eta      86/400   <- MAX_INTERVAL

Polls to detect completion, versus the old fixed schedule:

statement   legacy   cold (no estimate)   warm (1 sample)   overshoot
       2s        5                    4                 4      0.24s
      20s       19                   16                 8      0.18s
     120s       85                   31                13      0.16s
     300s      205                   39                19      0.16s
    2400s     1605                  109                89      0.16s

Overshoot stays sub-second because lengthening also requires elapsed < predicted × 0.85 and is capped by LENGTHEN_MULTIPLE and MAX_INTERVAL. A 100,000× over-estimate on a 2s statement still resolves in 2 polls.

Live Fabric — 6 threads, identical workload

Mode	Wall clock	API calls	calls/min
Before	217.7s	427	118
After (default)	233.3s	143	37
After, adaptive_polling: true	228.9s	129	34

3.0x fewer calls by default at the same wall clock. The run absorbed 19 real 429s and a CapacityLimitExceeded; each logged its pause and every model completed. Functional suite passes in both no_schema and with_schema modes.

[Raki (mdrakiburrahman)]

Benchmark vs Privy:

Privy

#265

---

HC Livy with this PR

It did not help at all: