Harden OpenShell eval pipeline: judge-safe model routing, gateway alias, best-effort cleanup

[shricharan-ks]

Why this is needed

The OpenShell OpenClaw profile (eval-engine=aeh_openshell_openclaw) evaluates the Forge "Chief of Staff" agent end-to-end: it stages a submission workspace, boots the agent in an OpenShell sandbox on a KubeVirt VM, drives scene-based cases (morning-briefing, analysis-panel), scores them with deterministic + LLM judges, and publishes results to MLflow. Running it in a per-user namespace (verification was done in forge-nommen) exposed five gaps that made runs either fail outright or silently lose all LLM-judge signal. This PR closes those gaps; none of them are covered by #108/#109.

Problems earlier → how this fixes them

#	Problem before	Root cause	Fix in this PR
1	Every LLM judge errored; mean_reward unusable. The judge client (score.py) only speaks the Anthropic dialect (POST /v1/messages). For an openai/ model, LiteLLM converts that to the upstream's /responses API, which the upstream vLLM gateway rejects with 403 Authentication parameters missing.	Model routing, not code: the judge override pointed at an openai/ route.	New judge-glm-5-3 LiteLLM route (hosted_vllm/) that keeps judge calls on /chat/completions, plus aeh-judge-model-override=judge-glm-5-3 default.
2	Slow, off-production eval model. Default llm-model was rits/zai-org/glm-5-3 (reasoning model), while the deployed agent runs GLM-5-3-Flash. Evals were slower and didn't measure the model users actually get.	Pipeline defaults drifted from the deployed agent config.	Defaults switched to rits/zai-org/GLM-5-3-Flash (pipeline + example run + aeh-model-override).
3	Gateway endpoint broke outside its home namespace. The default openshell-gateway-endpoint embedded another namespace's DNS and didn't validate against the gateway certificate, so the same Pipeline failed when created anywhere else.	The gateway runs on the openshell-saw-agent VMI in the workspace namespace; there was no stable local name for it.	New namespace-local alias Service openshell (port 17670) selecting the VMI; default endpoint https://openshell:17670, valid wherever the alias is installed.
4	The finally task could fail or hang the whole run and leak PVCs. Cleanup used bitnami/kubectl from Docker Hub (rate-limited pulls) with no request timeout — a hung API call held cleanup until the finally budget expired and a failed delete left the PVC behind; a failed finally marks the PipelineRun failed.	Cleanup acted like a gate when it is a janitor.	In-cluster OpenShift CLI image (image-registry.openshift-image-registry.svc:5000/openshell/cli), --request-timeout=20s, 3 retries, onError: continue.
5	oc apply -n <ns> -f … failed for anyone but the original namespace. Manifests pinned namespace: ab-eval-flow / gz-forge-eval, so applying into a per-user namespace returned "the namespace from the provided object does not match".	Hardcoded metadata.	Namespace fields removed from all deployable manifests; the example PipelineRun uses short service names (http://litellm:4000, https://openshell:17670) that resolve in whatever namespace the run is created in.

Additionally, failures were slow to diagnose because nothing could tell which dependency (litellm, mlflow, minio, k8s API, github, pypi, agent gateway, postgres) was unreachable — hence the standalone probe task below.

Changes, one by one

1. feat(forge-saw): add stable namespace-local openshell gateway alias Service — new config/forge-saw/openshell-eval-alias.yaml. A selector Service for the openshell-saw-agent VMI on port 17670. Justification: gives every namespace a stable, certificate-valid https://openshell:17670 endpoint without cross-namespace DNS or cert pinning; installed with the stack, referenced by the Pipeline default.
2. feat(litellm): add GLM-5-3-Flash routes for agent and judge traffic — config/litellm/configmap.yaml adds rits/zai-org/GLM-5-3-Flash (openai/ lane, agent traffic) and judge-glm-5-3 (hosted_vllm lane, judge traffic) over the same INFERENCE_ENDPOINT_URL/INFERENCE_API_KEY env. Justification: the dialect-safe judge route is the only way the Anthropic-dialect judge client gets served by this OpenAI-dialect upstream (problem 1); the Flash route matches the deployed agent (problem 2).
3. feat(pipeline): add eval-stack-probe task — new pipeline/tasks/eval-stack-probe.yaml: HTTP probes (mlflow, litellm, minio, k8s API, github, pypi, local wheels) + TCP probes (agent gateway, integration gateway, postgres). Deliberately not labelled part-of so it can run without the pipeline's NetworkPolicy allowances. Justification: one TaskRun replaces reading thousands of lines of evaluate logs to find a dead dependency; it is a triage tool, not a pipeline stage.
4. fix(cleanup): make PVC cleanup best-effort and use in-cluster CLI image — pipeline/tasks/post/cleanup_pvc.yaml. Justification: problem 4 — cleanup must never fail the run or leak PVCs; the in-cluster registry removes the Docker Hub dependency.
5. feat(pipeline): default to GLM-5-3-Flash with judge route and gateway alias — pipeline/pipelines/ci-pipeline-openshell.yaml + pipeline/runs/openshell-openclaw-pipelinerun.yaml. Justification: wires problems 1–3 into the Pipeline defaults so a plain oc create -f of the example run works in any namespace; keeps explicit timeout budgets (pipeline 3h / tasks 2h30m / finally 15m) and the part-of label required by the canonical NetworkPolicies.
6. test: cover Flash defaults, judge route, alias Service, and cleanup hardening — tests/test_openshell_pipeline_profile.py (10 tests). Justification: locks in every default this PR changes — judge route must stay hosted_vllm, example run must stay namespace-agnostic, cleanup must stay best-effort — so a future edit can't silently regress them. Also asserts the evaluate step takes the model key from $(params.llm-api-key) (no namespace-secret dependency, per Fix OpenShell evaluation Secret dependency #108).
7. chore: drop hardcoded namespaces from deployable manifests — 8 files, one line each. Justification: problem 5; makes every manifest apply cleanly with -n <any-namespace>.

Dependencies

• Stacked on Port Forge OpenShell reliability fixes onto current main #109 (OPEN — must merge first). This branch is cut from its current head (3232787); until Port Forge OpenShell reliability fixes onto current main #109 merges, this PR's diff also shows its 4 commits. After Port Forge OpenShell reliability fixes onto current main #109 merges, GitHub will recompute this PR to show only the 7 commits above.
• Builds on Fix OpenShell evaluation Secret dependency #108 (MERGED) — the evaluate step receives the model API key via $(params.llm-api-key) instead of reading a namespace secret; this PR's tests pin that behavior.
• Fix OpenClaw brief-reader execution and add turn diagnostics GuyZivRH/agent-eval-harness#9 (MERGED to harness main) — OpenClaw brief-reader/gateway fixes the pipeline's harness revision default (main) requires.
• Merge OpenClaw brief-reader fixes onto main for Forge evals GuyZivRH/agent-eval-harness#10 (OPEN) — further brief-reader fixes; not required by this PR.
• Runtime prerequisites (not code): in the run namespace — litellm with the two routes above, MLflow, secrets openshell-mtls / forge-ai-gateway-ca / openshell-credentials / openshell-oidc-credentials, the openshell alias Service, and the agent VMI running ghcr.io/rh-forge/openclaw-saw-agent@sha256:b47b92a6b3fd03327c1f2093a5c28aba0fdf3cb620e9154335688900191fe2b9.

Verification

• End-to-end: PipelineRun scharan-verify-hardening-2 (namespace forge-nommen, 2026-10-09): all 5 tasks Succeeded; recommendation pass, mean_reward 0.7000 (threshold 0.5); 12 judges × 2 cases with 0 errors (previously all LLM judges errored); scorecard 6/6 gates; MLflow experiment recorded with traces + judge feedback; the hardened finally task deleted the run's PVC.
• Baseline comparison: same-day runs on the pre-hardening stack (sana-morning-briefing-pr105-single-*, -collector-diag-*) scored 0.00–0.35 and failed the aeh-low-score gate.
• Tests: tests/test_openshell_pipeline_profile.py — 10/10 pass.
• Cluster parity: deployed objects in forge-nommen verified identical to this branch (all 44 Pipeline param defaults, every task step script, litellm configmap, alias Service selector).

Ops note (out of scope, for the record)

During verification the run-namespace LiteLLM lost its upstream key (the k8s secret holding it had been deleted; only the old pods still had it in memory). #108 already removed the eval task's dependency on that secret; the key for the namespace's LiteLLM deployment itself was restored from the vault-injected value on the running forge-ai-gateway pod. Keeping that namespace secret in sync with Vault is an ops concern outside this repo.

🤖 Generated with Claude Code