Skip to content

agent: bound the agent that works forever and never reports #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
whilo wants to merge 1 commit into
base: main
Choose a base branch
from
+177 −7
Open

agent: bound the agent that works forever and never reports #47

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 79 additions & 7 deletions src/dvergr/chat/agent.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
[dvergr.tools :as tools]
[dvergr.sandbox.workspace :as workspace]
[datahike.api :as dh]
[clojure.string :as str]
[taoensso.telemere :as tel]))

;; ============================================================================
Expand Down Expand Up @@ -70,6 +71,62 @@
(hash (select-keys % [:tool-use/name :tool-use/input])))
recent-tool-calls)))))

(def ^:const silent-tool-run-threshold
"Consecutive tool-only turns — no prose to the room — before we intervene.

Twelve, not three. A long tool run is ordinary work: reading a dozen pages
before answering is exactly what these agents are for, and cutting that
short would be worse than the bug. What is not ordinary is never speaking."
12)

(defn- said-something?
"Did this assistant turn actually address the room? Whitespace does not
count — a model that emits `\n` has not reported anything."
[m]
(not (str/blank? (str (or (:message/content m) (:content m))))))

(defn- called-a-tool? [m]
(seq (or (:message/tool-uses m) (:tool-uses m))))

(defn detect-silent-tool-run
"Detect an agent that keeps working and never reports.

`detect-doom-loop` fingerprints `[:tool-use/name :tool-use/input]`, so it
sees an agent REPEATING itself. It cannot see one that varies its calls and
simply never concludes — which is the shape that actually bit us: turns
0->15 on dev.simm.is, every one a `clojure_eval`, `content-len 0` on all of
them, then the cycle restarted and did it again. Nothing objected, because
nothing was repeated and the dollar budget had not run out.

Counts CONSECUTIVE tool-only assistant turns from the newest end. Speaking
resets the count: an agent that reports progress is working, however many
tools it uses. Returns `{:turns n}` past the threshold, else nil.

A turn with neither tools nor prose is not counted — that is a different
failure, and folding it in here would fire this bound on the wrong evidence."
[messages]
;; Walk back until the agent SPOKE, then count the tool turns in between.
;; Stopping on "not a tool turn" instead would let a model reset the bound by
;; emitting a newline: that turn says nothing, so it must not count as a
;; report, and it has no tools, so it must not count as work either.
(let [streak (->> (reverse messages)
(filter #(= :assistant (or (:message/role %) (:role %))))
(take-while #(not (said-something? %)))
(filter called-a-tool?)
count)]
(when (>= streak silent-tool-run-threshold)
{:turns streak})))

(defn silent-run-nudge
"Corrective for an agent that has worked `n` turns without saying anything."
[n]
(str "You have made " n " tool calls in a row without saying anything to the "
"room. Nobody can see tool calls — from where the humans sit, nothing "
"has happened since they asked.\n\n"
"Reply now, in prose, with what you have found so far and what you are "
"still trying to do. If you are stuck, say what is blocking you. You can "
"keep working after you have reported."))

(def fragment-nudge
"Corrective shown to a model that emitted code as its message instead of
calling a tool. Names the mistake and the fix, without scolding — and it is
Expand Down Expand Up @@ -334,12 +391,13 @@
(:error result)
(pr-str result))
:turn-number turn-number}))
;; Loop bound: if the agent is repeating the SAME tool call with the
;; same args and making no progress, stop the auto-continue cycle
;; instead of spinning until budget runs out. History stays coherent
;; (tool_uses + their results are both present); a nudge tells the
;; model why it stopped, and the next human turn resumes it. This
;; finally wires detect-doom-loop, which was dead code.
;; Two bounds, in order of severity.
;;
;; STOP if the agent is repeating the SAME tool call with the same
;; args and making no progress, rather than spinning until budget
;; runs out. History stays coherent (tool_uses + their results are
;; both present); a nudge tells the model why it stopped, and the
;; next human turn resumes it.
(if-let [looped (detect-doom-loop (chat-ctx/get-messages chat-ctx))]
(do
(tel/log! {:level :warn :id :agent/doom-loop
Expand All @@ -355,7 +413,21 @@
"blocking you.")
:important? true)
:complete)
:continue))
;; NUDGE if it has worked a long time and said nothing. Checked
;; second because it is the weaker signal: this agent is not
;; necessarily stuck, it may be mid-way through something real, so
;; it keeps going — it just has to tell the room first. Said once
;; per streak, since replying is what resets the streak.
(if-let [silent (detect-silent-tool-run (chat-ctx/get-messages chat-ctx))]
(do
(tel/log! {:level :warn :id :agent/silent-tool-run
:data {:turns (:turns silent) :turn turn-number}}
"Agent has run many tool calls without replying — nudging")
(chat-ctx/add-system-note! chat-ctx
(silent-run-nudge (:turns silent))
:important? true)
:continue)
:continue)))

;; No tool calls. Usually that means the agent is done — but a model can
;; also fumble its code into the PROSE channel (stop_reason=stop, no
Expand Down
98 changes: 98 additions & 0 deletions test/dvergr/chat/no_reply_loop_test.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
(ns dvergr.chat.no-reply-loop-test
"The loop shape `detect-doom-loop` cannot see.

MEASURED on dev.simm.is 2026-08-06: an agent ran turns 0→15, every one a
`clojure_eval` tool call, `content-len 0` on all of them — it never said a
word to the room — then the cycle restarted at turn 0 and did it again.
56,802 datahike warnings came out of the read loop it was driving.

`detect-doom-loop` did not fire once, and correctly so: it fingerprints
`[:tool-use/name :tool-use/input]`, so it only catches an agent repeating
the SAME call. This agent varied its code every turn (input tokens climbed
40,275 → 42,265). It was not repeating itself; it was never finishing.

The only other bound is the dollar budget, which at ~40k input tokens a turn
buys a great many turns before it bites — and it bites silently, which is
why the room just saw nothing happen for two hours."
(:require [clojure.test :refer [deftest testing is]]
[dvergr.chat.agent :as agent]))

(defn- tool-turn
"An assistant turn that called a tool and said nothing to the room.
`n` varies the arguments, which is what makes it invisible to the
identical-call fingerprint."
[n]
{:message/role :assistant
:message/content ""
:message/tool-uses [{:tool-use/name "clojure_eval"
:tool-use/input {:code (str "(wiki/read-page \"Page " n "\")")}}]})

(defn- repeated-turn
"An assistant turn calling the SAME tool with the SAME args every time."
[]
{:message/role :assistant
:message/content ""
:message/tool-uses [{:tool-use/name "clojure_eval"
:tool-use/input {:code "(wiki/pages)"}}]})

(defn- spoke-turn
"An assistant turn that actually replied to the room."
[text]
{:message/role :assistant
:message/content text
:message/tool-uses []})

;; =============================================================================
;; The existing guard: what it does and does not see
;; =============================================================================

(deftest detect-doom-loop-catches-only-identical-calls
(testing "identical calls are caught — the guard works as designed"
(is (some? (agent/detect-doom-loop (repeat 4 (repeated-turn))))
"three identical calls trip the fingerprint"))

(testing "VARYING calls are invisible to it — the production shape"
;; This is the assertion that documents the gap. Not a bug in
;; detect-doom-loop; a bound that was never written.
(is (nil? (agent/detect-doom-loop (map tool-turn (range 16))))
"sixteen different tool calls, no reply, and nothing objects")))

;; =============================================================================
;; The bound that was missing
;; =============================================================================

(deftest silent-tool-run-is-detected
(testing "N consecutive tool-only turns with no prose is the signal"
(is (nil? (agent/detect-silent-tool-run (map tool-turn (range 3))))
"a short tool run is ordinary work, not a loop")
(is (some? (agent/detect-silent-tool-run (map tool-turn (range 20))))
"a long one, with nothing ever said to the room, is not"))

(testing "the count is CONSECUTIVE — speaking resets it"
;; An agent that reports progress is working, however many tools it uses.
;; Ordering is newest-last, matching how chat-ctx accumulates messages.
(let [history (concat (map tool-turn (range 20))
[(spoke-turn "Here is what I found so far.")]
(map tool-turn (range 3)))]
(is (nil? (agent/detect-silent-tool-run history))
"a reply clears the streak")))

(testing "whitespace-only content does not count as speaking"
(let [history (concat (map tool-turn (range 10))
[(spoke-turn " \n ")]
(map tool-turn (range 10)))]
(is (some? (agent/detect-silent-tool-run history))
"an empty-looking reply is not a reply")))

(testing "a turn with neither tools nor prose does not extend the streak"
;; Defensive: an empty assistant turn is a different failure, and counting
;; it here would make the bound fire on the wrong evidence.
(is (nil? (agent/detect-silent-tool-run
(repeat 20 {:message/role :assistant
:message/content ""
:message/tool-uses []})))))

(testing "it reports how many turns went by, for the nudge to quote"
(let [r (agent/detect-silent-tool-run (map tool-turn (range 20)))]
(is (integer? (:turns r)))
(is (>= (:turns r) agent/silent-tool-run-threshold)))))