Skip to content

fix: credit interventions to their actual actor - #135

Merged
lukemaj merged 1 commit into
mainfrom
fix/133-intervention-actors
Oct 19, 2026
Merged

lukemaj merged 1 commit into
mainfrom
fix/133-intervention-actors

Conversation

@lukemaj

@lukemaj lukemaj commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

What: Credit intervention senders correctly, preserve grouped verdicts, and reject completion notices followed by user instructions.
Why: Prefix matching credited user prose to agents, while timestamp and canonical-id precedence could hide a stored yes.
So what: Round-3 review passed and Luke approved merging; this authorized rebase is pushed for the parent, with no merge or new reviewers started.

Closes #133. This advances the Objective through explicit ownership and visible evidence gaps across native harnesses.

Head: 3d8ae6971ba8ae7c50935a5e4213ed7a194a1208. Base: ed90f9577c0b48cfeb539688ba1370f14e4c4515 (origin/main). This body supersedes earlier candidate proof and attribution totals.

Rebase after #134 and exact candidate proof

  • Host: bestia. Old reviewed head: 49c3766e365fce0af524d56e5d61252dc3d212fd; new head: 3d8ae6971ba8ae7c50935a5e4213ed7a194a1208; new main: ed90f9577c0b48cfeb539688ba1370f14e4c4515.
  • Only conflict: CHANGELOG.md. The two branches had different entries under 1.4.1. I took main's changelog, preserving fix: treat pending gh pr checks as pending (#132) #134's 1.4.1 and the existing 1.4.0 entry, then ran versionctl prepare patch --reason 'Credit intervention senders, preserve grouped verdicts and retain unknown actor reasons' (preview and apply). It added Credit interventions to the right actor: steers, task notices and unknown root prompts #133 under 1.4.2 and updated VERSION and all three plugin mirrors.
  • No conflicts in privacy.py, db.py or analysis.py. Those files, command_status.py, ingest.py and fix: treat pending gh pr checks as pending (#132) #134's tests are byte-for-byte equal to new main. All ten reviewed Credit interventions to the right actor: steers, task notices and unknown root prompts #133 implementation/docs/test files are byte-for-byte equal to 49c3766. No reviewed behavior was edited.
  • python3 -m unittest discover -s tests: 929 passed, 101.937 seconds. python3 -m compileall -q agent_observer tests scripts, git diff --check and versionctl release-check: passed on the clean new head.
  • A fresh SQLite mode=ro backup of the live ledger was fully replayed on its copy using all native sources for intervention-bearing window sessions. 327/327 spectrum-review verdicts resolve, with the entire stored verdicts table unchanged. Recurrence is 64 yes / 259 no / 0 unclear, 323 canonical judgments and four superseded rows.
  • Before copied-ledger initialization, 23 recognized pending-check results were stored as failed. db.init_db corrected all 23 and wrote pr_checks_pending_version=2. After full replay, all 23 recognized results remain pending and analysis._failed returns false for each; marker 2 remains set.
  • Fresh window totals remain 319 user / 4,484 agent / 1,584 unknown = 6,387. Replay sources: 209 claude, 1,074 codex, 395 grok, 1 opencode; elapsed 244 seconds.
  • Push used an explicit lease against 49c3766; remote head verified. The existing Observer cost comment was republished after push. Both fresh ledger copies were deleted and absence verified; no live-ledger sync, merge, installation or new reviewer dispatch.

The requested range-diff has no differences beyond version bookkeeping and changelog resolution. Excluding those five bookkeeping paths gives:

git range-diff eda2c8d..49c3766 origin/main..HEAD -- . \
  ':!VERSION' ':!CHANGELOG.md' ':!.claude-plugin/plugin.json' \
  ':!.codex-plugin/plugin.json' ':!.grok-plugin/plugin.json'
1:  49c3766 = 1:  3d8ae69 fix: credit intervention actors and group T3 steers (#133)

Full unfiltered range-diff:

git range-diff --no-color eda2c8d..49c3766 origin/main..HEAD
1:  49c3766 ! 1:  3d8ae69 fix: credit intervention actors and group T3 steers (#133)
    @@ .claude-plugin/plugin.json
     @@
      {
        "name": "agent-observer",
    --  "version": "1.4.0",
    -+  "version": "1.4.1",
    +-  "version": "1.4.1",
    ++  "version": "1.4.2",
        "description": "Measure agent work across Codex, Claude Code, OpenCode and Grok Build from native records, and compare behavior by model and AgentsMD version.",
        "author": {
          "name": "toolboxmd",
    @@ .codex-plugin/plugin.json
     @@
      {
        "name": "agent-observer",
    --  "version": "1.4.0",
    -+  "version": "1.4.1",
    +-  "version": "1.4.1",
    ++  "version": "1.4.2",
        "description": "Measure agent work across Codex, Claude Code, OpenCode and Grok Build from native records, and compare behavior by model and AgentsMD version.",
        "author": {
          "name": "toolboxmd",
    @@ .grok-plugin/plugin.json
     @@
      {
        "name": "agent-observer",
    --  "version": "1.4.0",
    -+  "version": "1.4.1",
    +-  "version": "1.4.1",
    ++  "version": "1.4.2",
        "description": "Measure agent work across Codex, Claude Code, OpenCode and Grok Build from native records, and compare behavior by model and AgentsMD version.",
        "author": {
          "name": "toolboxmd",
    @@ CHANGELOG.md
     @@
      # Changelog
      
    -+## [1.4.1] - 2026-10-19
    ++## [1.4.2] - 2026-10-19
     +
     +### Changed
     +
     +- Credit intervention senders, preserve grouped verdicts and retain unknown actor reasons
     +
    - ## [1.4.0] - 2026-10-19
    + ## [1.4.1] - 2026-10-19
      
    - ### Added
    + ### Changed
     
      ## README.md ##
     @@ README.md: person. They are never interventions and are counted as `policy_denials`.
    @@ README.md: person. They are never interventions and are counted as `policy_denia
     
      ## VERSION ##
     @@
    --1.4.0
    -+1.4.1
    +-1.4.1
    ++1.4.2
     
      ## agent_observer/adapters/claude.py ##
     @@ agent_observer/adapters/claude.py: import json

Final behavior

  • Claude SDK steers count as one interrupt with the sender recorded in the complete T3 interruption envelope. Same-source native ordering links the preceding interrupt marker and user-rejected tool results through related_ids; raw inputs remain available.
  • Selection excludes duplicate ids. Resolve/show accept any related id and return its canonical steer. Existing verdict ids are never rewritten or deleted.
  • For each behavior and judge, a yes on either an alias or the canonical intervention wins over no/unclear, regardless of timestamps. Among otherwise equal-priority judgments, latest timestamp then lexical original id wins; there is no canonical-id preference. Recurrence and groups count once. Superseded entries preserve original verdict, problem, group and timestamp for audit.
  • verdict list retains original rows and exposes canonical_intervention, canonical_verdict, canonical_problem, and canonical_group. Text output identifies a different canonical verdict.
  • Completion parsing consumes the whole input. Only trailing whitespace and a closed <runtime_info> block are accepted. A pasted notice followed by an instruction is ordinary input. Closed blocks cannot absorb prose after a closing tag through regex backtracking.
  • Steers require the complete interruption boilerplate, rather than its first sentence. Explicit native actor evidence wins over a steer sender. A complete terminal notice remains agent input when Codex carries forward the preceding human conversational turn id.
  • Grok records the same closed transport metadata in _upsert_submission. Its native first-input wrapper and complete runtime/linking/optional-tool-use suffix are parsed only as complete anchored shapes; prose after those blocks is retained. Claude/Codex source versions and Grok import version refresh older records once on a separately authorized sync. No transcript or new free-text sender data is stored.
  • Unknown prompts retain stored reasons. Missing Claude SDK identity is native_sdk_prompt_without_matching_origin; Grok uses native_prompt_without_matching_origin. Reporting remains read-only. Treat pending gh pr checks (exit 8) as pending, not failed #132 analysis/failure classification is untouched.

Review repairs and regression proof

Round 1 exposed orphaned verdict references and missing Grok evidence ingestion. Round 2 exposed incomplete notice boundaries and a judgment conflict policy that could suppress yes. The fixes live in the parser, canonical judgment owner and their regressions, rather than additional workflow rules.

Sanitized native-record regressions cover human/agent steers, tool denials, completion notices, Grok direct/wrapped notices and steers, aliases resolving/showing their canonical steer, recurrence/group/list agreement, superseded audit fields and replay idempotency. Added negative cases include a notice plus newline instruction, prose after runtime metadata, attempted extra closing tags, partial interruption boilerplate and a known human pasting an agent-steer note. Positive cases retain complete native suffixes and Codex's inherited turn id.

Against archived exact reviewed head a16938bc3b1dc504d4f6731c54863d5e433375f2, the pasted-notice regressions fail with agent != user; superseded audit assertions fail on the missing problem field. A separate priority probe reports [no, no] for alias-yes/canonical-no at equal and newer-no timestamps; the repaired implementation reports [yes, yes]. With audit assertions omitted only in the disposable baseline copy, both timestamp cases fail at recurrence's verdict-count assertion. Production regression tests retain all audit assertions and also cover a newer yes.

  • python3 -m unittest discover -s tests: 929 passed, 101.937 seconds on bestia after the rebase (including the seven fix: treat pending gh pr checks as pending (#132) #134 tests). Existing ResourceWarnings are non-failing.
  • python3 -m unittest tests.test_intervention_actors: 11 passed after the final parser compatibility additions. Earlier broader focused run: 94 passed.
  • python3 -m compileall -q agent_observer tests scripts, git diff --check, complete diff self-review: passed.
  • versionctl release-check: passed for v1.4.2 on the exact head above. The already-published upstream v1.4.1 tag was fetched before validation; no tag was created.

Full copied-ledger replay

Window: 2026-10-12T14:20+02:00 inclusive to 2026-10-19T14:20+02:00 exclusive, Warsaw time.

SQLite opened the live ledger with mode=ro and backed it up. The historical untouched baseline was read with the CLI archived from main eda2c8df41d75246ab11d0f7932cd0400f2f932c; its totals remain recorded here. A fresh post-rebase live-ledger copy independently revalidated the candidate totals and verdicts below. A separate copy received full native imports for every source backing intervention-bearing window sessions across all actors: 209 Claude files, 1,074 Codex files, 395 Grok directories and one OpenCode database. All sources were available. Existing mirrored T3 evidence was held constant; stored root reasons were backfilled. No live-ledger sync was run.

bin/agent-observer --db <copy> interventions --created-by all \
  --since 2026-10-12T14:20+02:00 --until 2026-10-19T14:20+02:00 --json
bin/agent-observer --db <copy> interventions --created-by user \
  --since 2026-10-12T14:20+02:00 --until 2026-10-19T14:20+02:00 --json
bin/agent-observer --db <copy> verdict list \
  --behavior work_failed_user --judge spectrum-review --json
bin/agent-observer --db <copy> recurrence --behavior work_failed_user \
  --judge spectrum-review --created-by all \
  --since 2026-10-12T14:20+02:00 --until 2026-10-19T14:20+02:00 --json
Metric Main baseline Full replay
User interventions 329 319
Agent interventions 4,096 4,484
Unknown interventions 1,972 1,584
All interventions 6,397 6,387
User prompts 314 307
User interrupts 6 12
User rejections 9 0
All prompts 6,037 6,014
All interrupts 343 365
All rejections 17 8

Verdicts: 327/327 spectrum-review ids resolve, zero orphaned; the entire stored verdicts table is field-for-field unchanged. CLI listing returns 327 rows attached to canonical interventions. CLI show resolves the previously orphaned yes id sub:claude:85509847-d096-4b83-bd7f-770a94c0b437 with status ok. Private text is excluded from this record.

Recurrence: 323 canonical interventions judged, 64 yes, 259 no, zero unclear, with four superseded judgments retained explicitly. The alias yes counts once.

Grok investigation previously inspected all 131 window prompts mentioning Delegated task: 112 direct notices with attached runtime metadata, 18 notices inside a first-input envelope, and one non-terminal notification about a question. Native sibling context includes linking and tool-use metadata. All 130 terminal notices remain agent inputs. Compared with the previous reviewed replay, 11 Claude inputs with non-notice trailing prose now stay unknown rather than being claimed as agent; the user total is unchanged. Complete suffix parsing preserves those notice shapes without accepting a pasted notice followed by prose; partial interruption notes remain unknown when no author evidence exists.

External unknown-actor cause and limits

The historical root identity gap is tracked in Chromeria #344, forward-repaired in #346. Of the original 570 unknown Claude root prompts, 548 provider references name assistant UUIDs and 22 have no matching inspected record. Observer preserves missing-identity reasons instead of guessing from the opener, text or child threads. Historical matching, deployment and new runtime acceptance are not claimed.

Version, delivery and cleanup

PR #134 merged after round-3 review as main ed90f9577c0b48cfeb539688ba1370f14e4c4515, releasing 1.4.1. This branch was rebased onto that main and versionctl prepared its single pending patch transition as 1.4.1 → 1.4.2. CHANGELOG retains both 1.4.1 and 1.4.0, and all three plugin mirrors are 1.4.2. No version was released or tagged by this work.

The Observer cost comment is republished from a separate temporary ledger for task toolboxmd/agent-observer#135. Shared work, missing usage and unpriced usage remain visible limits; this is not a complete spend estimate.

All task-owned copied ledgers, private replay artifacts and disposable source archives from the initial implementation and both review repairs were deleted after proof was recorded. Their deletion was verified. The live ledger was not modified.

Round-3 independent review passed on 49c3766e365fce0af524d56e5d61252dc3d212fd; Luke approved merging. The rebase changes only version bookkeeping and the changelog conflict, as the range-diff below shows. No new exact-head review is claimed and no reviewers were started. No merge, release, tag, plugin installation, deployment or live-ledger sync. Website/SEO impact: none; this changes local evidence/reporting, with README documenting the contract. The task worktree and branch remain for review; the eventual merger owns their retirement.

Elon record

Requirements and who asked: Luke requires #133 and both repairs, then a behavior-preserving rebase after #134, version 1.4.2, full suite, fresh-copy verdict/marker-2 proof, cost republication, copy deletion and no merge or new reviewers.
Deleted: No verdict migration/deletion, raw-event deletion, guessed actors, fixture exclusions, extra service, child-thread redesign or #132 classification changes.
Bottleneck: Complete input boundaries and canonical judgment precedence determine whether reports preserve human authorship and durable yes evidence.
Checked myself: Round-3 review record, unchanged implementation hashes, full and scoped range-diff, 929-test suite, versionctl release-check, fresh copied-ledger native replay, 327 resolutions, unchanged 64/259 recurrence, marker-2 correction, pushed head, cost comment and copy deletion.

@lukemaj

lukemaj commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Agent work on this PR

Estimated cost at least $1.00 · 52 responses · 27 sessions · 3.1 h wall time

Model Responses Tokens Estimated cost
grok-4.7 (grok, medium) 20 >=31,877,999 (lower bound) unknown
gpt-6.1-sol (codex, medium) 32 6,048,351 $1.00

Flags: 1 human correction · 1 large tool output · 6 repeated commands · 2 repeated failures · 60 repeated reads · 9 repeated skill loads · 6 tests edited after a failure · 1 session without usage records · 20 unpriced responses

Details: snapshot, prices, coverage, counters
  • Task toolboxmd/agent-observer#135: outcome unknown (recorded acceptance only; a finished process never implies it).
  • Proof: fix: credit interventions to their actual actor #135
  • Snapshot efb7c9fd3d2e534015ccffb1d6510703475d07b4d52110a24b4f2c14afa08fd1, records up to 2026-10-19 16:28 UTC.
  • 27 sessions on claude, codex, grok; AgentsMD 17.3.1.
  • The total includes work shared with 11 other tasks, counted whole in each, so sums across PRs overlap.
  • Not counted: 906 responses (at least $88.89) in sessions shared with other PRs that worked in no single PR's checkout.
  • 42 responses in these sessions worked on other PRs and are counted there.
  • Totals reconcile with the measured sessions: yes. Evidence complete: no.
  • Prices: list-price estimate from T3 local rate table (path withheld) as of 2026-10-18, schedule 414551dda4c47258113dcfa46a5dfa5aa512406e3dfcadcca3381421595c5588. Unknown prices stay unknown, never zero.
    • T3 LiteLLM rate table when present; bundled schedule covers the rest.
    • Model price sources: T3 local rate table (path withheld).
    • Unpriced: unknown model (20).
  • Native session usage or worker ownership is unavailable.
  • Harness-reported cost: none reported.
  • Usage totals are not billing. Subscription spending is separate and is never posted as spend.
  • Crashed runs are counted separately: 0.

Token counters by model (native counter semantics; never added across semantics):

  • grok-4.7 (grok, medium), grok counters: input includes cached, output includes reasoning: input >=31,215,754 (lower bound); cache-read >=29,139,328 (lower bound); cache-write >=0 (lower bound); output >=662,245 (lower bound); reasoning >=456,595 (lower bound); harness total >=31,877,999 (lower bound).
  • gpt-6.1-sol (codex, medium), codex counters: input includes cached, output includes reasoning: input 6,020,334; cache-read 5,956,608; cache-write 0; output 28,017; reasoning 11,063; harness total 6,048,351.

Generated tokens (reasoning and other output are disjoint; other output includes code, tool calls and replies):

Model Reasoning Other output Reasoning share Split known for
grok-4.7 (grok, medium) unknown unknown unknown 0 of 20
gpt-6.1-sol (codex, medium) 11,063 16,954 39.5% 32 of 32

Usage by work phase (this task's own responses):

Phase Model Responses Input Cache read Cache write Output Reasoning Total Priced part
unclassified gpt-6.1-sol (codex, medium) 32 6,020,334 5,956,608 0 28,017 11,063 6,048,351 $1.003283

Recorded activity by phase (observations, not separate token bills):

Phase Tool calls MCP results Reads File changes Failed tool results
unclassified 187 10 45 31 16

Selected rates (USD per million tokens). These rates value the report at the selected schedule date; they do not establish historical prices or subscription spending.

Model Input tier Input Cache read Cache write Other output Reasoning
gpt-6.1-sol all 2 0.1 2.5 10 10
grok-4.7 all unknown unknown unknown unknown unknown

Other output and reasoning are priced without double counting inclusive native output. Missing rates remain unknown.

Local measurement from native records; usage totals are not billing. Updated in place by agent-observer publish.

@lukemaj

lukemaj commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Independent review of exact head abcee57 (round 1)

  • Muse Spark 1.3 (xhigh, Muse Code on cavallo): PASS. Fail-before/pass-after probe for steer, notice and unknown reason; raw events preserved. Non-blocking: grouped ids lose their stored verdicts (judged harmless); narrow grouping windows; version collision with fix: treat pending gh pr checks as pending (#132) #134.
  • Grok 4.7 (medium): FAIL.
    1. Blocking: resolve (interventions.py:437) returns None for grouped ids, so 10 of the 327 spectrum-review verdicts no longer resolve after a replay: the 9 denial ids and the grouped interrupt marker sub:claude:85509847-d096-4b83-bd7f-770a94c0b437, which carries a yes verdict (built before the agreed design). Fix: keep duplicates out of select, and make resolve return the canonical steer for any id in its related_ids.
    2. Non-blocking: the Grok adapter (adapters/grok.py:2179) never records intervention evidence; 5 window Grok prompts are exact agent steers and stay unknown; 46 Grok prompts mention "Delegated task" without matching the anchored envelope.
    3. Non-blocking: the PR body's all-actor totals undercount the full replay (agent 4,397 / unknown 1,671 vs 4,301 / 1,767; user 319 matches).

Coordinator adjudication: 1 accepted as required (verdicts are durable judge evidence; Muse's "harmless" missed the yes verdict). 2 accepted as required for completion notices, because acceptance criterion 2 covers notices regardless of harness: record the same evidence for Grok prompts and replay once; investigate the 46 non-matching Grok prompts and either match genuine notice shapes or state why they are not notices. Agent-sent Grok steers follow from the same fix. 3 accepted: update the PR body with full-replay numbers. Fix round requested; both reviewers will review the new head.

@lukemaj
lukemaj force-pushed the fix/133-intervention-actors branch from abcee57 to 9f307d0 Compare October 19, 2026 14:21
@lukemaj
lukemaj force-pushed the fix/133-intervention-actors branch from 9f307d0 to a16938b Compare October 19, 2026 14:23
@lukemaj

lukemaj commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Independent review of exact head a16938b (round 2)

  • Muse Spark 1.3 (xhigh, cavallo): PASS. Round-1 items resolved: grouped ids resolve to the canonical steer, verdict list keeps the original id with canonical_intervention, recurrence counts one judgment per canonical intervention; Grok evidence recorded with a one-time replay; replay of the cavallo ledger reclassified no genuine human prompt. Findings: 1. Low: a pasted completion notice followed by a newline and the user's own instruction still matches _COMPLETION (intervention_evidence.py, prefix match with lookahead), crediting the user's text to the agent. 2. Note: when an alias yes and the canonical no have equal timestamps the no wins (tie prefers canonical), so a yes can be superseded by a no. 3. Note: Grok child-session notices stay unknown while Codex child notices become agent.
  • Grok 4.7 (medium): did not complete; two attempts stopped on API 402 "Grok Build usage balance exhausted" (the account has quota; other Grok sessions run). A fresh Grok session will review the next head.

Coordinator adjudication: 1 accepted as required (a user's own words must never be credited to an agent; anchor the notice match to the end of input modulo whitespace and <runtime_info>). 2 accepted as required (the brief's rule: a yes is never hidden by a no; when an alias and its canonical disagree, a yes wins and the superseded entry keeps its problem and group). 3 accepted as non-blocking (child threads are a non-goal). Fix round requested.

@lukemaj
lukemaj force-pushed the fix/133-intervention-actors branch 2 times, most recently from dfd7671 to 49c3766 Compare October 19, 2026 15:27
@lukemaj

lukemaj commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Dispatch rule for every remaining round on this PR (Luke, 2026-10-19 17:55 Warsaw): launch no new threads on cavallo. Run Muse Spark 1.3 xhigh (Muse Code) and every other reviewer or worker on bestia, or on rocky. Muse Code is available on bestia (login present since 16:59). Already-running cavallo reviewers may finish; nothing new starts there.

@lukemaj

lukemaj commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Independent review of exact head 49c3766 (round 3)

  • Muse Spark 1.3 (xhigh): PASS. Both round-2 fixes verified (24 notice/steer boundary cases, 7 yes/no precedence cases, all failing on a16938b); replay of the cavallo ledger reclassified no genuine human prompt; 11 cue test failures on macOS are pre-existing on main (test_cue fails on macOS (11 tests) while CI passes #138).
  • Opus 5.5 (medium) (Luke's replacement for Grok, whose Build sessions fail on a provider 402): PASS. All 327 spectrum-review verdicts resolve; the yes on 85509847 still counts even against a newer no; full replays of main and head reproduce the PR body numbers (329/4,096/1,972 to 319/4,484/1,584); every one of the 385 unknown-to-agent and 3 user-to-agent window rows checked individually; 922 tests OK. Low: the plural-notice pattern accepts free text in place of the id list (filed as a follow-up Issue).

Coordinator adjudication: no blocking findings. The low finding needs a user to type the exact closing sentence; tracked separately rather than another full round. Ready for merge approval, after #134: whichever merges second must rebase and take the next patch version (#134 is at 1.4.1, this PR also at 1.4.1).

@lukemaj
lukemaj force-pushed the fix/133-intervention-actors branch from 49c3766 to 3d8ae69 Compare October 19, 2026 16:29
@lukemaj

lukemaj commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Independent review of the rebased head 3d8ae69 (onto #134, version 1.4.2)

Coordinator: ready to merge under Luke's approval.

@lukemaj
lukemaj merged commit 8ffae39 into main Oct 19, 2026
2 checks passed
@lukemaj
lukemaj deleted the fix/133-intervention-actors branch October 19, 2026 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Credit interventions to the right actor: steers, task notices and unknown root prompts

1 participant