feat(metrics): preserve canonical engine load sections

[lucifer1004]

Motivation

Engine GetLoads responses carry canonical load sections (memory, queues, aggregate) that were dropped at the gRPC boundary, and engine load gauges were only polled when a load-aware routing policy was active — coupling engine observability to routing configuration.

What this changes

• SchedulerLoadSnapshot gains memory/queues snapshots and WorkerLoadResponse gains version + an optional aggregate section, with mappings for SGLang, vLLM (aggregate None), and TokenSpeed engines.
• Every successful routing-owned load poll is always re-exported as smg_engine_* Prometheus gauges; --engine-metrics is narrowed to only force polling when no load-aware policy is active. No additional engine RPC is introduced.
• Worker eviction now sentinels smg_engine_* series whenever a load was ever published, not only under the flag.
• Usage::with_cached_tokens(0) records an explicit zero (cold miss) instead of omitting cache accounting.

Tests

New conversion test (conversion_preserves_version_sections_and_aggregate), explicit-zero builder test, and the metrics-endpoint integration test; openai-protocol 93/0, smg metrics/policies/monitor lib suites 285/0, engine_metrics_test 1/1.

[coderabbitai]

Warning

Review limit reached

Next included review available in 55 minutes.

View limit details

Limit details: You’ve used all 2 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a5666ff5-ca6d-49f9-af37-8e1c4441843f

📥 Commits

Reviewing files that changed from the base of the PR and between 4715c68 and bdf5604.

📒 Files selected for processing (13)

• crates/grpc_client/src/sglang_scheduler.rs
• crates/grpc_client/src/tokenspeed_scheduler.rs
• crates/grpc_client/src/vllm_engine.rs
• crates/protocols/src/common.rs
• crates/protocols/src/worker.rs
• model_gateway/src/config/types.rs
• model_gateway/src/main.rs
• model_gateway/src/observability/metrics.rs
• model_gateway/src/policies/least_load.rs
• model_gateway/src/policies/power_of_two.rs
• model_gateway/src/routers/grpc/zmq_client.rs
• model_gateway/src/worker/monitor.rs
• model_gateway/tests/engine_metrics_test.rs

Warning

Your free Security trial is over. An organization admin can activate billing to continue.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[lucifer1004]

Note for reviewers: this PR and #2227 (TokenSpeed sampling seeds) both touch crates/grpc_client/src/tokenspeed_scheduler.rs (and one line of zmq_client.rs). Disjoint hunks; the second to land may need a trivial rebase.

[slin1237]

can you please resolve conflict

[lucifer1004]

Rebased onto current main (4715c68); conflicts resolved. Upstream absorbed the seed changes via #2227; this branch now carries only the canonical engine-load metrics theme, with the engine_metrics doc text reconciled against the new disable_load_monitoring/overload-protection fields from #2286.