Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 45 additions & 1 deletion INTERFACES.md
Original file line number Diff line number Diff line change
Expand Up @@ -689,7 +689,8 @@ A 200 response is strict, bounded UTF-8 JSON with only `created`, `data`, and op
exactly one `data` item. Base64 is decoded strictly and bounded to the registered 16 MiB
encoded-image limit. The receipt
records exact raw-response and output BLAKE3/SHA-256/count identities, optional provider media-type
claim, reported finite nonnegative USD cost when present, and measured adapter latency. The Image
claim, cost telemetry when it conforms to the receipt contract (explicit unavailability otherwise,
never response rejection), and measured adapter latency. The Image
response does not attest the actual model or upstream provider, so receipts honestly record
`actual_model:undisclosed` and `upstream_route:unknown`; a request pin is not rewritten as response
provenance.
Expand Down Expand Up @@ -719,6 +720,49 @@ Content-Length before treating an envelope as complete, and returns no headers.
data URLs, request bytes, response headers/body, and decoded output
bytes are excluded from adapter representations and stable errors.

## `eval/openrouter_real_e2e.py`: confirmation-bound one-call evaluation

The real-provider evaluation is deliberately outside the reusable adapter. Its first phase,
`prepare_openrouter_real_e2e`, is credential-free and dispatch-free. It freezes exact discovery,
source, authority, mask, visible overlay, compact-summary, packet projection, and wire identities
in an owner-only directory. Challenge and compact-summary versions are evaluation-local artifact
indexes; they do not expand the provider schema registry or claim a reusable Studio authority.

`execute_openrouter_real_e2e` accepts that directory plus a separate closed confirmation context.
It validates the exact challenge/summary identities, canonical principal/session/time fields,
artifact SHA-256/counts, path/device/inode binding, expiry, reconstructed packet and wire, and a
fresh byte-identical discovery response before credential access. The self-hash on the context is
integrity, not authorization: a caller must also supply a trusted Studio confirmation consumer
that atomically verifies and durably spends that exact authorization, and the production default
rejects execution. The retired Boolean one-shot function cannot reach
I/O, and the evaluation CLI only reports the missing trusted-authority prerequisite.

After validation, the caller-supplied confirmation consumer must atomically and durably spend the
authorization; this repository has no production Studio ledger. The local `O_EXCL` record proves
only same-directory concurrency behavior in the injected offline harness. The adapter journal
remains the authoritative non-idempotent send and provider-evidence boundary. `$0.05` is only an
exact discovery-quote admission limit, never a provider-side hard spend cap. Post-response cost is
non-gating telemetry. A terminal provider occurrence is not an aesthetic judgment; the sanitized
result separately reports media admission, raw structure/locality, localized-edit gate status,
workflow acceptance `not_recorded`, semantic/aesthetic `not_run`, and compositor `not_run`.

The offline authority helper reads explicitly supplied board and Pixel-RAG artifacts through public validators
and derives content-bound collection-gate identities. It never repairs stale bytes. The current
local evidence belongs to a retired projection and fails when explicitly supplied to the current
reader. Evidence republication, trusted authority-to-session integration, a durable Studio
confirmation boundary, and a credential-free idempotent post-response finalizer are explicit
prerequisites to any paid run; this slice performs no provider call and makes no real-run claim.

The helper's identity domains are evaluation-local. `eligible_corpus_sha256` hashes RFC 8785 of
`{schema_version, source_manifest:{catalog_sha256,dataset_id,manifest_sha256}, field, operator,
value, assets:[{asset_id,content_ref}]}` under
`moodboard.openrouter-real-e2e.eligible-corpus.v1`; `assets` is sorted by
`(asset_id, content_ref)`. `route_policy_id` hashes RFC 8785 of
`{schema_version, eligible_corpus_sha256, namespace, field, operator, value,
empty_result_policy, interpretation}` under `moodboard.openrouter-real-e2e.route-policy.v1`.
These hashes provide integrity only. A future trusted Studio integration must cross-bind them to
the exact validated board/Pixel bytes and enrolled creative session.

## `report.py`

### The axis vocabulary
Expand Down
61 changes: 61 additions & 0 deletions eval/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,3 +77,64 @@ outcome, since the two differ by how far apart the styles are. It does not make
worthless, and it does make the claim narrower. The rule is that the record is not accepted
as written, the claim is narrowed to the domain that passed, and the narrowing goes in the
README where a reader will see it rather than in a footnote.

## OpenRouter real-provider evaluation is a two-stage confirmation

`openrouter_real_e2e.py` is an opt-in evaluation harness, not a general provider CLI. Preparation
fetches and freezes exact discovery and public source bytes, compiles the authoritative rectangle
mask and visible overlay, binds the board/retrieval authority, computes the exact provider wire
identity, and writes an owner-only confirmation challenge. Preparation has no credential or
transport parameter and cannot dispatch.

Execution requires a separate, closed confirmation context that names the exact challenge and
compact summary, one enrolled principal, one Studio session, the same creative session, and one
fresh explicit approval. A document hash alone is not Studio authority: the production API has no
default confirmer and fails `confirmation_authority_unavailable` until a trusted Studio boundary
that atomically verifies and consumes the authorization is supplied. The retired Boolean
`authorize_one_paid_call` entry point always fails
`two_phase_confirmation_required`; the command line only reports the missing trusted-authority
integration and cannot prepare or dispatch a live challenge.

After all frozen bytes, identities, timestamps, directory inode, reconstructed wire, and a fresh
byte-identical discovery response agree, a caller-supplied Studio boundary must atomically and
durably consume the confirmation before Keychain access. This repository does not implement that
ledger: the default rejects execution, while the injected offline seam proves one-process CAS and
local `O_EXCL` concurrency behavior only. Credential-bearing work is contained in a non-raising
inner scope, core dumps are disabled before Keychain access, and private response and output bytes
remain in the owner-only journal/run directory.

The fixed `$0.05` value is a **quote-admission limit**, not a provider-enforced spending cap. It is
checked against the exact live discovery pricing before source access. Reported cost is post-hoc
telemetry: missing, differently reported, or unexpectedly high telemetry cannot undo a charge and
therefore does not strand an otherwise valid provider response before terminal media admission.
A receipt distinguishes the absence of a legible cost value (`not_reported`, covering absent or
non-object telemetry) from a present cost value the adapter could not certify
(`reported_uncertifiable`); the raw response bytes always retain the original. This covers telemetry the adapter could parse: a response whose JSON number lexemes
exceed the adapter's structural budgets is rejected as a malformed document by the bounded parse,
which is a document-integrity bound, not a telemetry judgment.
Reports distinguish provider lifecycle state, media admission, raw structural/locality evidence,
localized-edit gate status, workflow acceptance (`not_recorded`), semantic/aesthetic judgment
(`not_run`), and compositor execution (`not_run`).

No paid call is currently authorized. The available local Pixel-RAG evidence uses a retired
projection and fails the current public reader when supplied explicitly. A separately governed
evidence republication, a trusted authority-to-creative-session integration, a durable Studio
confirmation consumer, and a credential-free idempotent post-response finalizer are prerequisites
to a live run. Until those exist, the two-stage functions are an injected offline contract harness.

The executable acceptance map for this slice is:

| Condition | Evidence test |
| --- | --- |
| Prepare has no credential, transport, or Boolean authorization surface | `test_prepare_api_has_no_credential_or_transport_and_returns_frozen_challenge` |
| Exact discovery/source/authority/mask/overlay/summary bytes are bound | `test_prepare_freezes_exact_content_bound_snapshot_summary_and_overlay` and the artifact-drift matrix |
| Self-minted confirmation is insufficient without Studio authority | `test_production_default_rejects_self_minted_context_before_discovery_or_key` |
| Context, expiry, inode, fresh discovery, and rebuilt wire gate Keychain | confirmation-context, expiry, directory-swap, discovery-drift, and wire-drift tests |
| One injected in-process consumption winner can reach one fake POST | replay, ambiguity, and concurrent-executor tests |
| Quote arithmetic is exact and `$0.05` is pre-dispatch only | ambient-Decimal and over-quote tests |
| Missing post-paid cost telemetry does not strand valid media evidence | `test_missing_reported_cost_remains_terminal_success_after_paid_response` |
| Non-conforming cost telemetry degrades to explicit unavailability, never rejection | `test_nonconforming_cost_telemetry_degrades_to_unavailable_without_stranding` |
| A cost lexeme past the structural budget rejects the document (integrity bound, not telemetry) | `test_cost_number_is_bounded_before_decimal_expansion` |
| Credentials cannot survive public exceptions or local artifacts | real-E2E transport exception-graph tests |
| An unanticipated post-response failure still scans the private artifacts | `test_post_response_failure_still_scans_the_private_artifacts` |
| Real board/retrieval identities are derived, never label hashes | `test_openrouter_real_e2e_authority.py` |
Loading
Loading