Skip to content

feat(netproxy,diagnose): record network prompts nobody answered in time - #260

Open
Retr0MrWave wants to merge 2 commits into
mainfrom
feat/abandoned-prompt-audit
Open

feat(netproxy,diagnose): record network prompts nobody answered in time#260
Retr0MrWave wants to merge 2 commits into
mainfrom
feat/abandoned-prompt-audit

Conversation

@Retr0MrWave

Copy link
Copy Markdown
Contributor

Issue: Closes #257 · Refs #241

What

  • New net.prompt_abandoned audit event for a network prompt that was raised but never resolved.
  • Prompt-sourced net.decision events now carry waited_ms — how long the dialog was open before it was answered.
  • omac diagnose reports both, and no longer claims "Nothing was blocked by the network policy" for a run where either happened.

Why

While a prompt is open the connection is held, and the requesting tool is subject to its own timeout. The opencode Skainet plugin aborts model discovery after 3s; answering a GUI dialog measurably takes longer (5s measured here, with the dialog already in focus). So the tool gives up, the fetch fails silently, and the allow you then grant applies to a request nobody is waiting for.

Both halves of that were invisible:

  • If the run ends with the dialog still open, no net.decision is ever written. omac diagnose reported 0/1 connection(s) blocked · Nothing was blocked by the network policy for a run that had silently failed.
  • If the dialog is answered, just too late, a perfectly ordinary allow is recorded and nothing looks wrong at all.

This is not an edge case: the compiled-in default profile (internal/sandboxprofile/resolve.go:23) has no allow_domain, so on a fresh install every host prompts on first contact.

How

  • The dialog deliberately runs on a detached context (internal/netprompt/prompt.go:312 builds its timeout from context.Background()), so an open prompt outlives the request that raised it and dies with the process. Detection therefore belongs at teardown: Filter.DrainAbandonedPrompts records whatever is still in flight when the proxy closes. It runs last in Server.Close, after the listener and connections are gone — draining earlier clears the coalescing map while requests can still arrive, which would raise a second dialog for a host already being asked about.
  • Nothing in the proxy can observe the requester leaving (clientSourceCtx is also context.Background()-derived), so for the answered-too-late case the prompt's open duration is recorded on the decision instead, and diagnose flags anything ≥3s (promptLatencyThresholdMS — the shortest client budget observed in the wild).
  • A prompt already recorded as abandoned does not also emit a decision: promptWait carries a drained flag so one request cannot appear as both abandoned and allowed.
  • Abandoned entries are aggregated per host:port with a repeat count and capped at 8, mirroring maxBlockedShown — otherwise --run all emits one line per occurrence across the whole persistent log.

Verification

go build ./..., go vet ./internal/... clean. go test ./internal/netproxy/ -race clean. go test ./... passes except TestIntegrationWorktreeKnownLimitations and TestIntegrationWorkflowInterpretersRunnable, which fail identically on clean main on this host (local toolchain paths absent from the default profile).

Drain path, end to end, no human interaction (the client times out on its own):

omac sandbox run --profile <prompt-enabled> --audit-log /tmp/a.jsonl \
  -- curl -s --max-time 1 https://example.com          # exit 28

diag:  net ABANDONED example.com:443 (prompt raised 967ms ago, no verdict —
       the requesting tool stopped waiting)
audit: net.prompt_abandoned example.com 443 waited_ms=967
omac diagnose:
       last run · filtered mode · 0/0 connection(s) blocked · 1 prompt(s) abandoned
       • 1 network prompt(s) … were raised but never answered in time

Late-answer path (the case the first commit missed — a waited_ms=5200 prompt-sourced allow):

• 1 prompt(s) were answered after 3.0s or more — a short-timeout tool may have failed anyway
  chat.model.tngtech.com:443 was held for 5.2s before the prompt was answered.
  Fix: add these hosts to network.allow_domain so no prompt is raised.

…and no "Nothing was blocked" in either case. Runs with neither condition are unchanged (regression test asserts the note still fires).

The second commit is review fixes; the late-answer detection, the Close ordering, the double-report suppression and the detail cap each have a test.

Follow-up

🤖 Generated with Claude Code

Ilia Zhuravok and others added 2 commits August 27, 2026 19:38
While a prompt is open the connection is held, and the requesting tool is
subject to its own timeout — the opencode Skainet plugin aborts model
discovery after 3s, while answering a dialog measurably takes longer. The
tool gives up, no verdict is ever reached, and no net.decision is written.
`omac diagnose` then reported "0/1 connection(s) blocked · Nothing was
blocked by the network policy" for a run that had silently failed.

The dialog is deliberately detached from the request context
(internal/netprompt builds its timeout from context.Background()), so an
open prompt outlives the request that raised it and dies with the process.
Detection therefore belongs at teardown: Filter.DrainAbandonedPrompts
records everything still in flight when the proxy closes, as a new
net.prompt_abandoned audit event carrying how long the prompt was open.

diagnose gains a matching axis: the abandoned count in the status line, a
problem hint naming the host, the wait, and the remedy (pre-allow the host
— answering faster is not reliable), and suppression of the now-misleading
"nothing was blocked" note. Runs with no abandoned prompts are unchanged.

Verified on Linux/bwrap, no human interaction needed (the client times out
on its own):
  omac sandbox run --profile <prompt-enabled> --audit-log /tmp/a.jsonl \
    -- curl -s --max-time 1 https://example.com     # exit 28
  diag log:   net ABANDONED example.com:443 (prompt raised 964ms ago, no
              verdict — the requesting tool stopped waiting)
  audit:      net.prompt_abandoned example.com 443 waited_ms=964
  omac diagnose:
              last run · filtered mode · 0/0 connection(s) blocked ·
              1 prompt(s) abandoned
              • 1 network prompt(s) were raised but never answered in time
                example.com:443 gave up after 964ms …
                Fix: add the host to network.allow_domain …

Deliberately not included: making a bare `omac sandbox run` write the
central audit trail (issue checklist item 4). It would couple sandboxrun to
launcher-config loading, which it does not do today — worth its own change.

go build ./... && go test ./... pass except TestIntegrationWorktreeKnownLimitations
and TestIntegrationWorkflowInterpretersRunnable, which fail identically on
clean main on this host (local toolchain paths absent from the default profile).

Closes #257
Refs #241

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Ilia Zhuravok <ilia.zhuravok@tngtech.com>
…e up

Review findings on this branch. The first is the substantive one: the
original commit only detected prompts still open at shutdown, so the
headline case from #257 — a short-timeout client gives up, the user answers
seconds later, the run continues — still produced a plain allow and no
abandoned record, and diagnose still printed "Nothing was blocked".

Nothing in the proxy observes the requester leaving (both clientSourceCtx
and the dialog ctx derive from context.Background()), so instead of
inferring a disconnect, the prompt's open duration is now recorded on the
decision itself: net.decision carries waited_ms for prompt-sourced
verdicts, and diagnose reports any answered after >=3s (the shortest client
budget observed in the wild — the opencode Skainet plugin's model
discovery) as a likely silent failure. Both halves now suppress the
misleading "nothing was blocked" note.

Also fixed:

- Server.Close drained before the listener and connections were torn down,
  clearing the coalescing map while requests could still arrive: a
  follow-up request for the same host would raise a second dialog and go
  unrecorded. The drain is now last.
- A drained prompt answered later emitted a net.decision too, so one
  request appeared as both abandoned and allowed. promptWait carries a
  drained flag and the resolution path skips the emit (logging why).
- abandonedPromptHints emitted one uncapped detail line per entry, which
  under --run all spans the whole persistent log. Now aggregated per
  host:port with a repeat count and capped at 8, mirroring maxBlockedShown.
- docs: net.prompt_abandoned added to the audit event-type list, plus the
  waited_ms field on prompt decisions.

Verified:
  late-answer path (was invisible, now a problem finding):
    net.decision waited_ms=5200 source=prompt ->
    "1 prompt(s) were answered after 3.0s or more — a short-timeout tool
     may have failed anyway
       chat.model.tngtech.com:443 was held for 5.2s …"
    and no "Nothing was blocked by the network policy"
  drain path still works (curl --max-time 1, no human interaction):
    net.prompt_abandoned example.com waited_ms=967
  go test ./internal/netproxy/ -race passes.

Refs #257
Refs #241

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Ilia Zhuravok <ilia.zhuravok@tngtech.com>
@mwtng
mwtng force-pushed the feat/abandoned-prompt-audit branch from fd0a7b9 to 102da62 Compare August 27, 2026 17:42
@mwtng

mwtng commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

I just rebased and reran the verification. I did not do a review, yet.

But probably one should have a look at the changes to the documentation and if they are really necessary in such detail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Network prompt latency silently defeats short client timeouts, and an abandoned prompt leaves no record

2 participants