DEP (light): Operator control to disable /engine worker control routes

[nnshah1]

Summary

The /engine/* routes on the worker system-status server (DYN_SYSTEM_PORT) expose advanced worker control operations that not all deployments need. Add a runtime-level policy that lets operators harden workers by removing these routes — either unilaterally (all of them) or selectively (a chosen subset) — enforced uniformly across backends at a single dispatch point, with default behavior unchanged.

Motivation

The /engine/* routes provide advanced controls: profiling, memory occupation/release, elastic-EP scaling, weight updates, and model-taint updates. Many deployments never use them. Today there is no unified operator control to pare down or remove the exposed set, and which routes are exposed depends on the backend. Operators who want to run workers with a minimal control surface should be able to disable these routes, either entirely or selectively per deployment.

Proposal

Add an EngineRoutePolicy to EngineRouteRegistry, resolved once from environment at construction, and enforce it in engine_route_handler before the registry lookup. Because every /engine/* route across all backends funnels through this single handler (system_status_server.rs /engine/{*path} -> engine_route_handler -> EngineRouteRegistry), one enforcement point covers every backend uniformly — no per-backend logic to keep in sync.

enum EngineRoutePolicy {
    AllowAll,                    // default — backward compatible
    DisableAll,                  // DYN_DISABLE_ENGINE_ROUTES=1
    Allowlist(HashSet<String>),  // DYN_ENGINE_ROUTES_ALLOW="control/start_profile,update/model_taints"
    Denylist(HashSet<String>),   // DYN_ENGINE_ROUTES_DENY="control/update_weights_from_disk,..."
}

• Policy-denied routes return 403 (distinct from 404 "not registered").
• Matching is on the full route string — some routes (e.g. update/model_taints) have no control/ prefix, so a prefix-based policy would miss them.
• Default AllowAll keeps the runtime backward compatible; a restrictive policy is opt-in via env (and can be set in operator/Helm manifests for a hardened shipped posture).
• No inference-path impact: the gate lives on the system-status server; the inference data path never traverses engine_route_handler. Policy is resolved once at startup; per control-call cost is a single O(1) check.

Routes this governs (main today):

Route (/engine/…)	sglang	vLLM	trtllm	Class
control/start_profile, control/stop_profile	✅	✅	—	profiling
control/release_memory_occupation, control/resume_memory_occupation	✅	—	✅	memory
control/sleep, control/wake_up	—	✅	—	memory
control/scale_elastic_ep, control/is_scaling_elastic_ep	✅	✅	—	scaling
control/update_weights_from_disk / _tensor / _distributed / _ipc	✅	✅	—	weight update
control/update_weight_version, get_weight_version, init/destroy_weights_update_group	✅	✅	—	weight mgmt
pause_generation, resume_generation, flush_cache, abort_request, liveness_probe	—	✅	—	control
load_lora, unload_lora	—	✅	—	model mgmt
update/model_taints	✅	✅	✅	model mgmt

Guidance — prefer a scoped denylist over the master switch when workers still need operational routes. update/model_taints is a legitimate operational route on all backends (routing taints), so DYN_DISABLE_ENGINE_ROUTES=1 disables it too. Deployments that only want to drop, say, the weight-update routes should denylist those and leave profiling/memory/taint routes available.