Injection and tool-use benchmark on local models #179

Activity

1. itsskofficial

   added

   enhancementNew feature or requestNew feature or request

   help wantedExtra attention is neededExtra attention is needed

   area: engineAgent loop, models, approvals, storage, gatewayAgent loop, models, approvals, storage, gateway

   priority: P2Later: useful, lower urgencyLater: useful, lower urgency

   size: MMedium: one to two weeksMedium: one to two weeks

   on Oct 9, 2026
2. Tanmay-Ts commented on Nov 4, 2026

   

   Tanmay-Ts

   on Nov 4, 2026

   Contributor

   More actions

   Hi @itsskofficial, I'd like to take this one.

   Plan:

   • A bench/injection/ harness, outside testpaths so the default pytest run and CI skip it. It drives the real agent loop with fixture tools: a web page, an email and a file that carry injected instructions, plus sink tools (send an email, write a file, run code) that only record their calls. Nothing leaves the machine.
   • Cases are user tasks times injection payloads, as in AgentDojo, including a fake closing marker and a payload in another language.
   • Scoring is deterministic, with no model judge: task success (the reply has the expected fact), attack attempted (the model called a sink with the attacker's target) and attack executed (that call actually ran). With SAFE-2 in place, executed should stay at 0 and attempted shows what the model tried. SAFE-12 is compared by running with chat.mark_outside_content on and off.
   • sentient bench injection --model ollama_chat/qwen3:8b prints the rates, and the qwen3:8b results with and without spotlighting go in docs/VERIFICATION.md. One FakeProvider test in tests/ keeps the harness from breaking.

   One question: should SAFE-2 get an off switch for the benchmark, or is attempted vs executed enough? I'd rather not add a way to turn the gate off.