Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions packages/std/evals/inspector.prose.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,8 @@ Read the run's artifacts and produce a structured extraction suitable for evalua
- (deep only) manifest_summary: execution order and wiring from forme.manifest.json
- (deep only) service_outputs: map of service name to first 500 chars of each binding
- (deep only) workspace_artifacts: map of service name to list of files in workspace
- (deep only) handoff_provenance: map of delegate name to pre-dispatch handoffs (supplied inputs, versions, sequence) and reported review scopes
- (deep only) observed_telemetry: map of delegate name to observed tool access events, or "unknown" if host telemetry was not captured
- (deep only) error_details: contents of any `__error.md` files

### Errors
Expand Down Expand Up @@ -164,6 +166,10 @@ Apply judgment to the extraction. Score runtime fidelity and task effectiveness
- when scoring: use the full 0-100 range. A perfect run scores 95-100, not 100 (reserve 100 for extraordinary cases). A run with minor issues scores 70-85. A run with significant problems scores 40-69. A fundamentally broken run scores below 40.
- when a run failed but produced partial output: evaluate what exists. A failed run can still have high task effectiveness if the partial output is useful.
- when evidence conflicts: note the conflict explicitly in flags rather than silently resolving it
- when evaluating provenance: distinguish supplied inputs (from handoffs/*.json) from worker-reported review scope ({id}-response.md) and observed access events (observed_access.jsonl). Never treat a supplied input tuple or tool-level file read as proof of understanding, and never treat an agent report as host authentication.
- when a delegate output cites an input missing from contemporaneous pre-dispatch handoff records: flag unrecorded-handoff (critical). Expose the missing handoff gap rather than retroactively backfilling it.
- when an input is declared not_read or integrity_check_only: verify downstream tasks did not treat it as semantically evaluated; flag supplied-unreviewed if subsequent decisions falsely assume vetting.
- when host tool telemetry is absent: record telemetry status as unknown; do not synthesize or assume tool events from worker return text.

---

Expand Down
23 changes: 15 additions & 8 deletions skills/open-prose/prose.md
Original file line number Diff line number Diff line change
Expand Up @@ -754,17 +754,20 @@ The render writes its request payload to the specified path before yielding. The
The VM:

1. Reads the delegation request
2. Spawns the delegate as a new session (same mechanics as Step 4b — the delegate's source, inputs, workspace, and output instructions come from the manifest)
3. Passes the request file as the delegate's input
4. Waits for the delegate to complete normally (writes outputs, returns confirmation)
5. Writes the delegate's output to `workspace/{node}/__delegate/{delegate}/{id}-response.md`
6. Resumes the original render with a pointer to the response:
2. Writes a contemporaneous pre-dispatch handoff record to `workspace/{node}/__delegate/{delegate}/handoffs/{seq}.json` recording exact task identity, instructions, and supplied input versions (see `state/execution-provenance.md`)
3. Spawns the delegate as a new session (same mechanics as Step 4b — the delegate's source, inputs, workspace, and output instructions come from the manifest)
4. Passes the request file as the delegate's input
5. Waits for the delegate to complete normally (writes outputs, records reported review scope, returns confirmation)
6. Writes the delegate's output and reported review scope to `workspace/{node}/__delegate/{delegate}/{id}-response.md`
7. Resumes the original render with a pointer to the response:

```
Delegation complete: {delegate-name}/{id}
Response: workspace/{node}/__delegate/{delegate}/{id}-response.md
```

If follow-up or reassignment handoffs occur (such as supplying auxiliary input `D`), each subsequent handoff is recorded as a distinct, monotonically sequenced pre-dispatch record (`handoffs/002.json`) before dispatch. Retrospective summaries must never be labeled contemporaneous dispatch records.

The render reads the response and continues execution.

### Parallel Delegation
Expand All @@ -778,7 +781,7 @@ Delegate: validator
Request: workspace/server/__delegate/validator/req-001.md
```

The VM spawns all delegates concurrently, waits for all to complete, and resumes the render once with all response paths.
The VM writes a pre-dispatch record for each delegate, spawns all delegates concurrently, waits for all to complete, and resumes the render once with all response paths.

### State Markers

Expand All @@ -799,8 +802,12 @@ Delegation state lives in the delegating node's workspace:

```
workspace/{node}/__delegate/{delegate}/
├── {id}.md # Request payload (written by render before yield)
└── {id}-response.md # Response payload (written by VM after delegate completes)
├── handoffs/
│ ├── 001.json # Pre-dispatch record (supplied inputs + task)
│ └── 002.json # Permitted follow-up handoff (e.g. additional input D)
├── {id}.md # Request payload (written by render before yield)
├── {id}-response.md # Response payload + reported review scope (written by VM)
└── observed_access.jsonl # Optional host tool telemetry (if supported)
```

### Interaction with Persistent Responsibilities
Expand Down
1 change: 1 addition & 0 deletions skills/open-prose/state/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ history — still canonical, just ephemeral.
## Contents

- `filesystem.md` — file-based state; the **normative reference** for the canonical world-model artifact layout, deterministic serialization, and the receipt ledger
- `execution-provenance.md` — host-agnostic execution-provenance profile; normative rules distinguishing supplied inputs, observed access telemetry, and worker-reported review scope across initial and follow-up handoffs
- `in-context.md` — ephemeral state held in the LLM context window; canonical world-model + ledger live in conversation history, lost when the session ends
- `sqlite.md` — SQLite-backed persistence; receipt ledger + content-addressed world-model versioning, with SQL as a derived query projection
- `postgres.md` — PostgreSQL-backed persistence; the same ledger + world-model versioning for multi-agent and multi-host systems, with SQL/vector projections for query
226 changes: 226 additions & 0 deletions skills/open-prose/state/execution-provenance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@
---
role: execution-provenance-specification
summary: |
Host-agnostic execution-provenance profile for inspectable handoffs. Defines the
distinction between supplied inputs, observed access events, and worker-reported
review scope across initial dispatch and permitted follow-up handoffs.
see-also:
- filesystem.md: Filesystem state layout and delegation directory structure
- README.md: State backend router
- ../prose.md: VM runtime delegation and yield/resume protocol
- ../primitives/session.md: Render context and input-by-reference guidelines
---

# Execution Provenance: Supplied Inputs, Observed Access, and Reported Review Scope

This document defines the **execution-provenance profile** for OpenProse handoffs,
delegations, and multi-turn workflows. It establishes a host-agnostic convention
for recording what an agent was given, what tool-level access was observed, and
what reading scope the agent claimed to perform.

---

## 1. The Provenance Gap

In OpenProse, a node's receipt records `input_fingerprints`—a bound tuple of content
hashes for the inputs declared in `### Requires` or bound by the caller.

> **Crucial Invariant:** `input_fingerprints` exists solely for **reconciliation
> and memoization** (determining whether upstream state changed). A bound input
> tuple is an opportunity to read, **not** proof of semantic inspection.

When a coordinator delegates work to a subagent or worker across multiple cycles,
auditors and inspectors require answers to three distinct questions:
1. *What was the worker actually handed before dispatch?*
2. *What physical file or tool operations did the host environment witness?*
3. *What scope of review did the worker attest to having conducted?*

Conflating these questions leads to critical audit failures: treating a path list
as proof of reading, assuming missing telemetry means non-access, or backfilling
an omitted follow-up handoff from a retrospective worker summary.

---

## 2. The Three Evidentiary Planes

Execution provenance separates evidence into three independent, non-interchangeable
planes:

| Plane | Authority | Form | What It Proves | What It Does NOT Prove |
| :--- | :--- | :--- | :--- | :--- |
| **1. Supplied Inputs** | Coordinator / Dispatcher | Immutable pre-dispatch handoff records (`handoffs/{seq}.json`) | What was authoritatively made available to the worker prior to dispatch. | Does not prove the worker opened or processed any of it. |
| **2. Observed Access** | Host Tool Telemetry | Intercepted tool events (`observed_access.jsonl`) | Physical I/O operations witnessed by the host environment. | Does not prove semantic comprehension or retention. |
| **3. Reported Review Scope** | Worker Agent | Worker return attestation (`review_scope` in response) | What the worker claims to have inspected and how it claims to have read it. | Is an agent claim; not host authentication. |

### Core Invariants

1. **A hash check or file read is not proof of understanding.** A tool event
demonstrating that bytes were transferred or a checksum was calculated proves
only physical access, never cognitive evaluation.
2. **An agent report is not host authentication.** A worker claiming "I thoroughly
reviewed Appendix C" is an attestation, not an empirical proof. It must be
recorded as worker-reported scope, never as observed access.
3. **Missing telemetry must stay unknown.** If a host does not capture tool
events, `telemetry_status` is `"unknown"`. It must never be recorded as `"none"`
(implying non-access) or synthesized from the agent's prose response.
4. **Retrospective reconstruction must not be labeled contemporaneous.** If a
follow-up handoff occurred without a contemporaneous pre-dispatch record, a
later inspector may note the gap, but must never generate a retroactive
dispatch record as if it were captured at dispatch time.
5. **Host-agnostic and unsigned baseline.** The baseline profile is an unsigned,
portable JSON/Markdown convention. It does not require host cryptographic
signatures, hidden chain-of-thought recording, live socket protocols, or
domain-specific scientific schemas.

---

## 3. Pre-Dispatch Handoff Records

Before any worker session or delegate is spawned, the coordinator or VM writes an
**immutable, contemporaneous pre-dispatch record**.

### Directory Structure

Within the delegating node's workspace:

```text
workspace/{node}/__delegate/{delegate}/
├── handoffs/
│ ├── 001.json # Initial dispatch handoff
│ └── 002.json # Permitted follow-up handoff (e.g. additional input D)
├── {id}.md # Request payload
└── {id}-response.md # Response payload (carrying worker review scope)
```

### Pre-Dispatch Record Schema (`handoffs/{seq}.json`)

```json
{
"handoff_id": "hnd-20260906-001",
"sequence": 1,
"parent_node": "coordinator",
"target_delegate": "auditor",
"dispatched_at": "2026-09-06T18:30:00Z",
"task_identity": "audit-security-controls",
"instructions": "Evaluate controls against incident telemetry.",
"supplied_inputs": [
{
"ref": "incident-log.md",
"path": "workspace/coordinator/incident-log.md",
"version": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
},
{
"ref": "control-matrix.md",
"path": "state/world-model/controls/published/matrix.md",
"version": "sha256:7a5e8f..."
},
{
"ref": "reference-architecture.md",
"path": "sources/arch.md",
"version": "sha256:1b2c3d..."
}
],
"permitted_followup": true
}
```

### Permitted Follow-Up & Reassignment Handoffs

In multi-turn workflows, a coordinator may supply auxiliary data (e.g., input `D`)
or adjust scope after a partial return.
- Each follow-up dispatch is written as a distinct, monotonically sequenced
record (`handoffs/002.json`).
- If a worker mentions or uses an input not present in any pre-dispatch record in
`handoffs/`, inspectors flag an **unrecorded-handoff gap**.
- The system never merges follow-up inputs back into `001.json` or invents prior
dispatch records post-hoc.

---

## 4. Worker-Reported Review Scope

When completing a task or yielding a response, the worker links a structured
attestation of its review scope in its return payload (`{id}-response.md` or
structured confirmation).

### Review Scope Modes

Each supplied input is categorized under one of five review modes:

| Mode | Meaning | Required Fields |
| :--- | :--- | :--- |
| `full` | Complete reading and evaluation of the entire input. | None. |
| `partial` | Selective reading of specific sections, tables, or line ranges. | `scope`: description or line/section ranges read. |
| `integrity_check_only` | Checksum or metadata verified; content not read for semantics. | `hash_verified`: boolean. |
| `reused_prior` | Reused conclusions from a prior review of the same version. | `prior_handoff_id` or `prior_receipt`. |
| `not_read` | Input was supplied but explicitly not opened or evaluated. | `reason`: explanation for non-reading. |

### Attestation Example

```json
{
"reported_review_scope": {
"incident-log.md": {
"mode": "partial",
"scope": "Sections 1.2 through 2.0 (lines 45-130)",
"reused_prior": false
},
"control-matrix.md": {
"mode": "integrity_check_only",
"hash_verified": true
},
"reference-architecture.md": {
"mode": "not_read",
"reason": "Not required after incident log established scope"
},
"supplemental-patch-notes.md": {
"mode": "full",
"scope": "entire_document"
}
}
}
```

---

## 5. Observed Access Telemetry (Adapter Seam)

Where the host environment or harness adapter provides tool call interception,
empirical access events are written to `observed_access.jsonl` in the delegation
directory:

```json
{"event": "tool_call", "timestamp": "2026-09-06T18:30:12Z", "tool": "view_file", "path": "workspace/coordinator/incident-log.md", "start_line": 45, "end_line": 130}
{"event": "tool_call", "timestamp": "2026-09-06T18:30:15Z", "tool": "hash_file", "path": "state/world-model/controls/published/matrix.md", "algorithm": "sha256"}
```

If the host does not capture tool-level telemetry:
- `observed_access.jsonl` is omitted or marked with `"status": "telemetry_unavailable"`.
- Inspectors record `telemetry_status: "unknown"`.
- Under no circumstances may an inspector convert a worker's reported review scope
into observed access entries.

---

## 6. Inspector Evaluation Rules

Post-run inspection (e.g. `std/evals/inspector`) evaluates the consistency of the
execution trace against the following rules:

1. **Supplied Input Completeness**: The union of all `supplied_inputs` across all
`handoffs/*.json` defines the total supplied input set.
2. **Review Scope Coverage**: Every supplied input must have a corresponding entry
in `reported_review_scope`. If an input is omitted from the report, it is
flagged as `unattested-consumption`.
3. **Supplied-Unreviewed Transparency**: An input marked `not_read` or
`integrity_check_only` is verified as supplied-but-unreviewed. This prevents
downstream tasks from falsely assuming the input was semantically vetted.
4. **Unrecorded Handoff Detection**: If the worker's output cites or demonstrates
intimate knowledge of an input `D` that has no contemporaneous pre-dispatch
record in `handoffs/`, the inspector flags `unrecorded-handoff`. The inspector
must expose the gap rather than synthesizing a missing pre-dispatch event.
5. **Telemetry Discrepancy Flagging**: If observed access is available:
- Tool calls to files never declared in any `handoffs/*.json` are flagged as
`undeclared-access`.
- Claims of `full` review with zero observed access events are flagged as
`unverified-reading-claim`.
10 changes: 8 additions & 2 deletions skills/open-prose/state/filesystem.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,12 @@ artifact, never the truth.
│ │ │ ├── raw-results.md # Intermediate data (e.g. raw poll w/ fetched_at)
│ │ │ └── __delegate/ # Runtime delegation state (if any)
│ │ │ └── {delegate}/
│ │ │ ├── handoffs/ # Pre-dispatch provenance records (execution-provenance.md)
│ │ │ │ ├── 001.json # Initial dispatch handoff (supplied inputs + task)
│ │ │ │ └── 002.json # Permitted follow-up handoffs (incremental inputs)
│ │ │ ├── {id}.md # Request payload
│ │ │ └── {id}-response.md # Response payload
│ │ │ ├── {id}-response.md # Response payload + reported review scope
│ │ │ └── observed_access.jsonl # Optional host tool telemetry (if supported)
│ │ ├── critic/
│ │ │ └── evaluation.md
│ │ └── synthesizer/
Expand Down Expand Up @@ -535,8 +539,10 @@ To resume an interrupted run:
| `sources/*.prose.md` | the compile phase | Before execution |
| `world-model/caller/*.md` | VM | At entry / gateway boot |
| `workspace/{node}/*` | the render | During the render |
| `workspace/{node}/__delegate/{delegate}/handoffs/{seq}.json` | the render / VM | Contemporaneously before dispatch (pre-dispatch record) |
| `workspace/{node}/__delegate/{delegate}/{id}.md` | the render | Before delegation yield |
| `workspace/{node}/__delegate/{delegate}/{id}-response.md` | VM | After delegate completes |
| `workspace/{node}/__delegate/{delegate}/{id}-response.md` | VM | After delegate completes (includes worker review scope) |
| `workspace/{node}/__delegate/{delegate}/observed_access.jsonl` | Host adapter | As tool calls occur (optional observed access telemetry) |
| `world-model/{node}/*` + `.version` | VM (`commit_world_model`) | On a `rendered` receipt with a moved fingerprint |
| `receipts/{node}.jsonl` | VM | After each render or skip |
| `vm.log.md` | VM | After each event |
Expand Down
Loading