Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "agent-observer",
"version": "1.4.1",
"version": "1.4.2",
"description": "Measure agent work across Codex, Claude Code, OpenCode and Grok Build from native records, and compare behavior by model and AgentsMD version.",
"author": {
"name": "toolboxmd",
Expand Down
2 changes: 1 addition & 1 deletion .codex-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "agent-observer",
"version": "1.4.1",
"version": "1.4.2",
"description": "Measure agent work across Codex, Claude Code, OpenCode and Grok Build from native records, and compare behavior by model and AgentsMD version.",
"author": {
"name": "toolboxmd",
Expand Down
2 changes: 1 addition & 1 deletion .grok-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "agent-observer",
"version": "1.4.1",
"version": "1.4.2",
"description": "Measure agent work across Codex, Claude Code, OpenCode and Grok Build from native records, and compare behavior by model and AgentsMD version.",
"author": {
"name": "toolboxmd",
Expand Down
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# Changelog

## [1.4.2] - 2026-10-19

### Changed

- Credit intervention senders, preserve grouped verdicts and retain unknown actor reasons

## [1.4.1] - 2026-10-19

### Changed
Expand Down
39 changes: 39 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -411,6 +411,45 @@ person. They are never interventions and are counted as `policy_denials`.
Claude writes a rejected tool call twice, as the denial and as an
interrupt in the same second; that is one intervention.

T3 steer envelopes identify their sender. A steer is one `interrupt`, keyed
by its follow-up prompt; `related_ids` links the native interrupt marker and
tool denials it replaces in this view. The raw messages and events remain.
Delegated-task terminal-state notices count as `agent`, including when the
harness labels their input as user-authored. Unknown actors carry an
`actor_reason`; native import stores the reason without the message body.
`native_sdk_prompt_without_matching_origin` means a Claude SDK prompt has
no matching Chromeria origin. A root thread alone cannot establish its author,
and a provider reference to an assistant UUID cannot identify the user prompt.
The next authorized sync replays older Claude and Codex sources once to
index this evidence; reporting itself remains read-only.

Grouped ids remain valid: `intervention show` resolves them to their canonical
steer and its follow-up text. `verdict list` retains every stored verdict under
its original id and adds `canonical_intervention`. Recurrence and verdict groups
count one judgment per canonical intervention and judge. A `yes` always wins
over `no` or `unclear`, regardless of timestamp or which id holds it. Within
that priority, the latest judgment wins, then the lexically greatest original
id, with no canonical-id preference. `verdict list` exposes the winning
`canonical_verdict`, `canonical_problem` and `canonical_group` beside every
original row. Recurrence reports displaced records, including their problem,
group and timestamp, in `superseded_verdicts` and names their count in text
output. Stored verdicts are
never rewritten or deleted by this grouping, and an alias verdict also marks
the canonical intervention as judged for `--unjudged`.

Grok imports the same transport metadata, including first-input
`<user_request>` envelopes and notices immediately followed by `<runtime_info>`.
Its import-version change replays older session metadata once.

Completion recognition consumes the entire input: only trailing whitespace
and a complete `<runtime_info>` block may follow the notice. Pasting a notice
and adding an instruction remains ordinary user input. Grok separates its
complete native runtime/linking/tool-use context before parsing; prose after those
blocks is never discarded. Steers require the complete T3 interruption note,
and explicit native actor evidence takes precedence over steer senders. Complete
terminal notices remain agent inputs even if Codex retains a preceding human
turn id.

```sh
agent-observer interventions --since 2026-09-14 --unjudged correction --json # default --created-by user --created-by agent
agent-observer intervention show sub:claude:<uuid> ev:codex:<id>/lifecycle/turn_aborted:<turn> --json
Expand Down
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.4.1
1.4.2
9 changes: 7 additions & 2 deletions agent_observer/adapters/claude.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
import os
import sqlite3

from .. import db, privacy
from .. import db, privacy, intervention_evidence
from ..identity import SessionIdentity, skill_from_path
from ..ingest import (JsonlSource, MissingNativeId, fingerprint, insert_event,
iso_ts, text_hash)
Expand Down Expand Up @@ -105,7 +105,7 @@ def _valid_counter(value) -> bool:

_FINALITY_TABLE = "claude_response_finality"
_NATIVE_KINDS_TABLE = "claude_native_kinds"
_IMPORT_VERSION = 2
_IMPORT_VERSION = 4


def _ensure_native_kinds(con) -> None:
Expand Down Expand Up @@ -333,6 +333,7 @@ def event(self, family, native_id, ordinal, ts, **kw):

def import_claude_file(con: sqlite3.Connection, path: str,
full: bool = False) -> dict:
intervention_evidence.ensure_table(con)
stats = {"lines": 0, "responses_inserted": 0, "responses_duplicate": 0,
"responses_updated": 0,
"user_messages_inserted": 0, "events_inserted": 0,
Expand Down Expand Up @@ -716,6 +717,10 @@ def _submission(r: _Reader, native_id, ordinal, ts, kind: str, text: str,
r.con.execute(f"INSERT INTO {_NATIVE_KINDS_TABLE}(native_id,kind,origin_allowed) VALUES(?,?,?)"
" ON CONFLICT(native_id) DO UPDATE SET kind=excluded.kind,origin_allowed=excluded.origin_allowed",
(f"{HARNESS}:{native_id}", native_kind or kind, int(origin_allowed)))
intervention_evidence.record(
r.con, f"{HARNESS}:{native_id}", text,
sdk=r.meta.get('entrypoint') == 'sdk-ts',
allowed=origin_allowed)


def _seed_queue_identity(r: _Reader) -> None:
Expand Down
9 changes: 7 additions & 2 deletions agent_observer/adapters/codex.py
Original file line number Diff line number Diff line change
Expand Up @@ -752,13 +752,15 @@ def import_codex_file(con: sqlite3.Connection, path: str,
Malformed lines are recorded in import_errors and skipped. Previously
valid data is never destroyed.
"""
from .. import intervention_evidence
intervention_evidence.ensure_table(con)
stats = {
"lines": 0, "responses_inserted": 0, "responses_duplicate": 0,
"user_messages_inserted": 0, "events_inserted": 0,
"events_duplicate": 0, "compactions": 0, "malformed": 0,
}
src = JsonlSource(con, HARNESS, path, full=full)
if (src.row["codex_context_version"] != 3
if (src.row["codex_context_version"] != 5
or src.row["codex_context_offset"] != src.row["read_offset"]):
# Replay older indexes once, including search extensions previously
# discarded from otherwise unchanged sources (#94).
Expand Down Expand Up @@ -958,7 +960,7 @@ def import_codex_file(con: sqlite3.Connection, path: str,
thread_source=reader.thread_source))
backfill = 1 if _backfill_complete(
con, src.source_id, getattr(src, "last_ordinal", None)) else 0
con.execute("UPDATE sources SET codex_context_version=3,codex_context_offset=?,"
con.execute("UPDATE sources SET codex_context_version=5,codex_context_offset=?,"
"codex_model=?,codex_effort=?,occurrence_backfill=? WHERE id=?",
(src.end_offset, reader.model, reader.effort, backfill,
src.source_id))
Expand Down Expand Up @@ -1825,6 +1827,9 @@ def _ingest_response_item(r: _Reader, obj: dict, native_ordinal, position: int)
values + [native_key])
r.stats["user_messages_updated"] = \
r.stats.get("user_messages_updated", 0) + 1
if first and (same_source or claims):
from .. import intervention_evidence
intervention_evidence.record(r.con, native_key, text)
_note_occurrence(r, "user", native_key, obj)
return
if ptype == "message" and p.get("role") == "assistant":
Expand Down
27 changes: 26 additions & 1 deletion agent_observer/adapters/grok.py
Original file line number Diff line number Diff line change
Expand Up @@ -550,6 +550,7 @@ def _has_any_counter(counters: dict | None) -> bool:


_GROK_META_TABLE = "grok_session_meta"
_IMPORT_VERSION = 2 # Replay older transport-envelope parsing once.


def _ensure_grok_meta_table(con: sqlite3.Connection) -> None:
Expand Down Expand Up @@ -590,7 +591,7 @@ def _summary_fingerprint(summary: dict) -> str:
canonical = json.dumps(summary, sort_keys=True, default=str)
except (TypeError, ValueError):
canonical = repr(sorted(summary.keys()))
return hashlib.sha256(canonical.encode("utf-8", "replace")).hexdigest()
return hashlib.sha256(f"{_IMPORT_VERSION}:{canonical}".encode("utf-8", "replace")).hexdigest()


def _stored_meta(con: sqlite3.Connection, session_key: str):
Expand Down Expand Up @@ -1426,6 +1427,8 @@ def import_grok_session(con: sqlite3.Connection, session_dir: str,
already imported, using their resolved source spelling. Metadata still
comes from session_dir; sharing one file never redirects the other.
"""
from .. import intervention_evidence
intervention_evidence.ensure_table(con)
stats = {"lines": 0, "responses_inserted": 0, "responses_duplicate": 0,
"user_messages_inserted": 0, "events_inserted": 0,
"events_duplicate": 0, "compactions": 0, "malformed": 0}
Expand Down Expand Up @@ -2240,6 +2243,24 @@ def _upsert_submission(con, r: _Reader, src, chat: dict, key: str,
stale values. Turn bindings only fill unknowns.
"""
full_text = "".join(entry["texts"])
# Strip only the complete native context suffix, never prose after it.
# These sibling blocks are Grok's machine-added runtime context;
# they are not part of the offered user input.
context = re.fullmatch(
r"(.*?)\s*<runtime_info>(?:(?!</?runtime_info>).)*</runtime_info>\s*"
r"<pull_request_linking>(?:(?!</?pull_request_linking>).)*</pull_request_linking>\s*"
r"(?:<t3_code_tool_use>(?:(?!</?t3_code_tool_use>).)*</t3_code_tool_use>\s*)?",
full_text, re.DOTALL)
offered_text = context[1] if context else full_text
# Grok's native first-input envelope contains the actual input in
# user_request, after Chromeria instructions. Do not search arbitrary
# prose for a notice: only unwrap this complete, anchored native shape.
wrapped = re.fullmatch(
r"<t3_code_instructions>.*?</t3_code_instructions>\s*"
r"<user_request>(.*?)</user_request>\s*"
r"(?:<runtime_info>(?:(?!</?runtime_info>).)*</runtime_info>\s*)?",
offered_text, re.DOTALL)
transport_text = wrapped[1].strip() if wrapped else offered_text
is_main = r.is_main_session
kind, is_genuine = _classify_prompt(key, chat, is_main)
excerpt = privacy.submission_excerpt(
Expand All @@ -2261,6 +2282,8 @@ def _upsert_submission(con, r: _Reader, src, chat: dict, key: str,
digest, excerpt, is_genuine))
if cur.rowcount:
r.stats["user_messages_inserted"] += 1
from .. import intervention_evidence
intervention_evidence.record(con, native_id, transport_text, sdk=True, allowed=is_main)
return
if digest != existing["text_hash"] \
or (existing["text_excerpt"] or "") != excerpt \
Expand All @@ -2275,6 +2298,8 @@ def _upsert_submission(con, r: _Reader, src, chat: dict, key: str,
if turn_id and not existing["turn_id"]:
con.execute("UPDATE user_messages SET turn_id=? WHERE native_id=?",
(turn_id, native_id))
from .. import intervention_evidence
intervention_evidence.record(con, native_id, transport_text, sdk=True, allowed=is_main)


def _reclassify_existing(con, r: _Reader, session_dir: str,
Expand Down
2 changes: 2 additions & 0 deletions agent_observer/adapters/t3.py
Original file line number Diff line number Diff line change
Expand Up @@ -360,6 +360,8 @@ def sync(con: sqlite3.Connection, root: str | None = None,
_import_messages(con, native, totals, lineage)
_import_prompt_origins(con, native, totals)
totals["user_messages_reclassified"] = _backfill_prompt_origins(con)
from .. import intervention_evidence
intervention_evidence.backfill_root_reasons(con)
totals["responses_retouched"] = _retouch_response_turns(con)
bound = _bind_trees(con, threads, links, lineage)
totals["tasks"] = bound["tasks"]
Expand Down
9 changes: 7 additions & 2 deletions agent_observer/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -1149,7 +1149,8 @@ def _judge(con, ns) -> int:
+ ", ".join(f"{k} {v}" for k, v in payload["policy_denials"].items()))
for m in payload["interventions"]:
print(f"{m['id']} {m['kind']} {m['created_by']} {m['at']} {m['project'] or ''}"
f" opening={json.dumps(m['opening'])}")
f" opening={json.dumps(m['opening'])}"
+ (f" actor_reason={m['actor_reason']}" if m.get('actor_reason') else ''))
return 0
if ns.cmd == "intervention":
ids = list(ns.ids)
Expand Down Expand Up @@ -1180,7 +1181,11 @@ def _judge(con, ns) -> int:
else:
for v in listed:
print(f"{v['intervention']} {v['verdict']} {v['group'] or '-'}"
f" {v['problem'] or ''}")
f" {v['problem'] or ''}"
+ (f" canonical={v['canonical_intervention']}"
if v['canonical_intervention'] not in (None, v['intervention']) else '')
+ (f" canonical_verdict={v['canonical_verdict']}"
if v['canonical_verdict'] not in (None, v['verdict']) else ''))
return 0
if ns.cmd == "verdict":
groups = _verdicts.groups(con, ns.behavior, judge=ns.judge,
Expand Down
67 changes: 67 additions & 0 deletions agent_observer/intervention_evidence.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
"""Closed transport metadata from native inputs, never their private bodies."""

import re


_STEER = re.compile(
r"\ANote from T3 Code: the (user|agent in T3 thread [^\s()]+) "
r"interrupted your previous turn, if it was still running, to deliver the message below\. "
r"Tool calls stopped for this message were interrupted, not refused, even if their results "
r"say the user declined them or does not want them\. "
r"Follow the message below before deciding whether to retry a stopped action\. "
r"Stopped actions may have partly run, other running work may have stopped too, "
r"and actions that finished keep their effects\.\n\n[^\s].*\Z", re.DOTALL)
_RUNTIME_INFO = r"<runtime_info>(?:(?!</?runtime_info>).)*</runtime_info>"
_RUNTIME_SUFFIX = r"\s*(?:" + _RUNTIME_INFO + r"\s*)?\Z"
_COMPLETION = re.compile(
r"\ADelegated task (?P<id>[^\s]+) reached a terminal state\. "
r"Use task_status with taskId (?P=id) to read the result\." + _RUNTIME_SUFFIX, re.DOTALL)
_COMPLETIONS = re.compile(
r"\ADelegated tasks [^\n]+ reached terminal states\. "
r"Use task_status with each taskId to read the results\." + _RUNTIME_SUFFIX, re.DOTALL)


def ensure_table(con):
con.execute(
"CREATE TABLE IF NOT EXISTS intervention_prompt_evidence("
" native_id TEXT PRIMARY KEY REFERENCES user_messages(native_id)"
" ON UPDATE CASCADE ON DELETE CASCADE,"
" transport_kind TEXT CHECK(transport_kind IN ('steer','completion')),"
" actor_kind TEXT CHECK(actor_kind IN ('client','server')),"
" actor_reason TEXT NOT NULL)")


def record(con, native_id, text, *, sdk=False, allowed=True):
"""Parse only anchored transport envelopes, excluding Claude scaffolding/children.

The envelope is sender evidence even when a native provider mislabeled the
input as user-authored. Ordinary SDK inputs need a matching origin, never
an inference from the thread opener or a preceding conversational turn.
"""
kind, actor = None, None
reason = ('native_sdk_prompt_without_matching_origin' if native_id.startswith('claude:')
else 'native_prompt_without_matching_origin') if sdk else 'actor_evidence_missing'
if allowed:
steer = _STEER.match(text) if sdk else None
if steer:
kind = 'steer'
actor = 'client' if steer[1] == 'user' else 'server'
reason = 't3_steer_sender'
elif _COMPLETION.match(text) or _COMPLETIONS.match(text):
kind, actor, reason = 'completion', 'server', 'delegated_task_notice'
con.execute(
"INSERT INTO intervention_prompt_evidence VALUES(?,?,?,?)"
" ON CONFLICT(native_id) DO UPDATE SET transport_kind=excluded.transport_kind,"
" actor_kind=excluded.actor_kind,actor_reason=excluded.actor_reason",
(native_id, kind, actor, reason))


def backfill_root_reasons(con):
"""Keep a stored gap even for harnesses without a transport-envelope parser."""
ensure_table(con)
con.execute(
"INSERT OR IGNORE INTO intervention_prompt_evidence"
" SELECT u.native_id,NULL,NULL,'native_prompt_without_matching_origin'"
" FROM user_messages u JOIN sessions s ON s.session_key=u.session_key"
" JOIN t3_threads t ON t.native_session=s.native_id"
" WHERE t.thread_id=t.root_thread_id AND t.opened_by='client'")
Loading
Loading