v0.18.0 — Reference Target Suite & Accuracy Benchmarking

[SaifullahSayyed]

🚀 What's New in v0.18.0

This release introduces the Reference Target Suite (Task G6) and Ground-Truth Detection Accuracy Metrics (Task G7), enabling robust empirical validation of security scans.

🎯 Reference Target Suite

• 12 Sandboxed Targets covering SQL injection, shell command execution, filesystem exfiltration, MCP tool misuse, memory poisoning, and privilege escalation (both Vulnerable and Hardened versions).
• Exposes GET /health and GET /ground_truth endpoints for automated validation.
• New crucible target CLI app command group:
  • crucible target list — View available targets.
  • crucible target start --name <name> — Run target locally.
  • crucible target validate — Perform bulk target health validation checks.

📊 Accuracy Benchmarking & Metrics

• Scans reference targets programmatically to compute core classification metrics: Precision, Recall, F1-Score, and Accuracy.
• Calculates 95% bootstrap confidence intervals based on joint distribution resampling.
• New crucible benchmark CLI command group:
  • crucible benchmark accuracy — Run accuracy evaluations against reference targets.
• Generates structured JSON reports and Markdown summaries (docs/accuracy_report.md).

🛠️ GitHub Action Update

• Compliant marketplace configuration in action.yml (shortened description for publication limits).

---

This discussion was created from the release v0.18.0 — Reference Target Suite & Accuracy Benchmarking.