Skip to content

feat: ship eval-harness v0.5.0 and Janus benchmark - #44

Merged
hoainho merged 3 commits into
mainfrom
feature/v0.5.0-janus-benchmark
Sep 29, 2026
Merged

hoainho merged 3 commits into
mainfrom
feature/v0.5.0-janus-benchmark

Conversation

@hoainho

@hoainho hoainho commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Ships the typed v0.5.0 evaluation workflows, portable manifests, grading, promotion readiness, reporting, and composite action integration.
  • Replaces implicit arbitrary shell execution with a constrained jq/printf/wc -l runner. This is a documented breaking change: migrate other shell checks to typed graders or opt in only for trusted case content with unsafe_shell: true / EVAL_ALLOW_UNSAFE_SHELL=1; this is not an OS sandbox.
  • Updates install, security, migration, policy, and release documentation.

Janus benchmark

  • Adds docs/JANUS_BENCHMARK.md for Janus harness-cli v0.1.10.
  • Offline deterministic comparison found Janus misses the positive file-output executor case and accepts checks: [] as PASS with zero checks; recommendation: no native executor adapter until Janus fixes both behaviors.
  • No live model requests, LLM quality, latency distribution, reliability distribution, or cost were measured.

Verification

  • npm test: all 57 evaluation test suites pass.
  • Shell-safety regressions cover chained jq environment access and workdir executable hijacking under relative PATH.
  • npm pack --dry-run: 149 files; required runner, pricing, example, and Janus report included.
  • Composite action YAML and all six run blocks parse; Mermaid validation passes for four docs.
  • Janus: cargo test -p harness-cli --features eval (66 passed) and cargo build -p harness-cli --features eval passed.

Release is GitHub tag/release only after merge; npm publication is not configured or attempted.

@hoainho
hoainho merged commit 38635c4 into main Sep 29, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant