feat(python): complete runtime parity harness

[sethjuarez]

Summary

• add an emitted-type durable Python turn engine with retries, cancellation, reconciliation, replay-safe checkpoints, permission handling, ordered tool execution, deterministic invalid-output handling, and safe sync/async boundaries
• align OpenAI, Foundry, and Anthropic model discovery/enrichment with shared capability data and discovery vectors
• harden shared load/render/parse/process/wire/agent/harness/engine contracts, structured output, streaming, replay verification, and terminal failure behavior
• require explicit opt-in for paid live image-model coverage

Validation

• uv run pytest tests/ --ignore=tests/model/agent/test_prompty.py -q --tb=short --cov=prompty --cov-report=term --cov-report=json — 1,514 passed, 17 skipped, 56 deselected; 80% coverage
• uv run pytest tests/integration -v -o "addopts=" — 22 passed, 34 skipped against configured real OpenAI and Anthropic providers
• uvx ruff@0.16.1 check . and uvx ruff@0.16.1 format --check . — clean
• GitHub Actions — Linux Python 3.11/3.12/3.13, Windows Python 3.11, CodeQL, repo hygiene, and artifact build all green

Typra follow-up

Python was regenerated and probed without hand-editing generated files against both published @typra/emitter 0.4.5 and 0.4.6. Both still fail all required adoption probes:

1. ModelInfo.load({"id": "x"}) hydrates omitted input/output modalities as []; explicit [] also remains [], so omitted-versus-empty tri-state is lost.
2. Default EngineCheckpoint.save() for duplicate ordered requests named same emits {"same": {"id": "2"}}, dropping the first request and order.
3. tests/model/agent/test_prompty.py reports 16 failed, 24 passed from generated multiline trailing-space mismatches.

All probe/regeneration churn was restored to the committed 0.4.2 baseline. This PR retains explicit None construction and SaveContext(collection_format="array"), and temporarily excludes only the single failing generated multiline fixture file; all other generated model tests remain in the gate. Keep this PR open and unmerged until the emitter fixes are available and all three probes pass.

[Copilot]

Pull request overview

This PR brings Python’s harness layer to parity with the shared cross-runtime “durable turn” contracts by adding an emitted-type ReferenceTurnEngine, aligning provider model discovery/enrichment with shared capability data and vectors, and tightening vector-based test enforcement (fail-fast on unknown/unsupported cases). It also adjusts processing semantics so null model content is treated consistently as an empty string and requires explicit opt-in for live image-model integration coverage.

Changes:

• Add a durable, replay-safe Python turn engine (ReferenceTurnEngine) with checkpoint round-tripping and resume semantics validated against shared engine vectors.
• Introduce shared model capability enrichment (prompty.core.model_capabilities) + vendored dataset and refactor OpenAI/Foundry/Anthropic model discovery to converge on shared discovery/enrichment vectors.
• Harden spec/vector tests (skip → fail) and align processing/test expectations around null content and tool/missing-tool behaviors; require explicit opt-in for OpenAI image integration tests.

Show a summary per file

File	Description
runtime/python/prompty/tests/test_turn_runner.py	Adds new runner correctness tests for failure propagation, zero-iteration behavior, and event-id uniqueness; tightens event ordering assertions.
runtime/python/prompty/tests/test_spec_vectors.py	Makes vector suites fail-fast (skip → fail) and updates thread nonce expansion test to exercise pipeline marker injection/expansion.
runtime/python/prompty/tests/test_responses.py	Updates JSON Schema expectations to allow nullable types while keeping fields required.
runtime/python/prompty/tests/test_replay_verifier.py	Adds replay-verifier coverage for missing/extra trailing records and ordered mismatch reporting.
runtime/python/prompty/tests/test_processor.py	Aligns processor contract/tests so null chat content returns empty string (including when outputs are configured).
runtime/python/prompty/tests/test_models.py	Refactors OpenAI/Foundry model discovery tests to focus on raw-wire mapping helpers and richer Foundry shape handling.
runtime/python/prompty/tests/test_model_capabilities.py	New unit tests for capability dataset drift guard + lookup/enrich semantics (prefix matching, token boundaries, fill-only-missing).
runtime/python/prompty/tests/test_enrichment_vectors.py	New vector-driven enrichment parity tests consuming spec enrichment vectors exhaustively with drift guards.
runtime/python/prompty/tests/test_engine_vectors.py	New vector-driven engine parity tests consuming shared turn engine vectors and validating resume/cancel/retry/reconcile behaviors.
runtime/python/prompty/tests/test_discovery_vectors.py	New vector-driven discovery parity tests consuming shared discovery vectors with dispatch/drift guards.
runtime/python/prompty/tests/test_anthropic.py	Extends Anthropic processor streaming tests for mixed text/tool-use and ordered multi-tool streaming cases.
runtime/python/prompty/tests/test_anthropic_models.py	New tests for Anthropic model discovery wire mapping and mocked client orchestration (sync/async).
runtime/python/prompty/tests/integration/conftest.py	Requires explicit OPENAI_IMAGE_MODEL opt-in for image integration tests (avoids retired defaults).
runtime/python/prompty/prompty/providers/openai/processor.py	Changes chat completion processing to normalize null content to empty string.
runtime/python/prompty/prompty/providers/openai/models.py	Replaces built-in known-model table with shared capability enrichment + adds model_info_from_wire mapping.
runtime/python/prompty/prompty/providers/foundry/models.py	Adds shared enrichment, expands Foundry wire mapping helpers (deployment/catalog), and tightens coercion semantics.
runtime/python/prompty/prompty/providers/anthropic/models.py	Adds Anthropic model discovery implementation + wire mapping helper with enrichment fallback.
runtime/python/prompty/prompty/providers/anthropic/init.py	Exposes Anthropic model discovery APIs from the provider package.
runtime/python/prompty/prompty/harness/turn_runner.py	Records a tool_result event for permission-denied outcomes (closing event/journal parity gap).
runtime/python/prompty/prompty/harness/engine.py	Introduces the new durable, emitted-type ReferenceTurnEngine with retries, cancellation, reconciliation, checkpointing, and resume.
runtime/python/prompty/prompty/harness/init.py	Re-exports new engine APIs and checkpoint helpers from the harness package.
runtime/python/prompty/prompty/data/model_capabilities.json	Adds vendored capability dataset snapshot used for cross-runtime discovery enrichment.
runtime/python/prompty/prompty/core/model_capabilities.py	Adds shared capability lookup/enrichment primitive with token-boundary longest-prefix matching.
runtime/python/prompty/prompty/core/init.py	Re-exports model capability enrichment symbols at the core package level.
runtime/python/prompty/prompty/init.py	Re-exports engine/checkpoint helpers at the top-level public API.
runtime/python/prompty/.env.example	Updates example image model env var to a non-retired image model and aligns with opt-in behavior.

Review details

• Files reviewed: 27/27 changed files
• Comments generated: 1
• Review effort level: Lite

[sethjuarez]

Fixed in 59000dc by checking message.content is None explicitly. Added regression coverage that preserves empty list and dict content by identity.

[Copilot]

Review details

Suppressed comments (2)

runtime/python/prompty/prompty/harness/engine.py:194

• resume() uses asyncio.run(), which will fail if called from an existing event loop. Consider mirroring run() by detecting a running loop and raising a targeted error telling callers to use resume_async().

        """Resume a durable checkpoint synchronously without repeating committed effects."""
        return asyncio.run(self.resume_async(context, cancellation=cancellation))

runtime/python/prompty/prompty/harness/engine.py:148

• Using asyncio.run() directly in a library sync wrapper will raise RuntimeError when called from an existing event loop (e.g., Jupyter/async frameworks). Add an explicit running-loop guard with a clear error directing callers to run_async().

This issue also appears on line 193 of the same file.

        """Run a new turn synchronously."""
        return asyncio.run(
            self.run_async(
                session_id,
                turn_id,

• Files reviewed: 27/27 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (3)

runtime/python/prompty/prompty/providers/openai/processor.py:143

• message.content or "" will coerce any falsey-but-valid content (e.g., an empty list for multimodal content parts) into an empty string. The intent here seems to be only to normalize None to "" while preserving non-None content verbatim.

    refusal = getattr(message, "refusal", None)
    if message.content is None and isinstance(refusal, str):
        return refusal

    return message.content or ""

runtime/python/prompty/prompty/harness/engine.py:195

• asyncio.run(...) has the same running-event-loop failure mode in resume(). This should produce a clearer error directing callers to resume_async() when invoked from async contexts.

    ) -> TurnEngineResult:
        """Resume a durable checkpoint synchronously without repeating committed effects."""
        return asyncio.run(self.resume_async(context, cancellation=cancellation))

runtime/python/prompty/prompty/harness/engine.py:148

• asyncio.run(...) raises RuntimeError: asyncio.run() cannot be called from a running event loop when ReferenceTurnEngine.run() is used from async contexts (e.g., notebooks, async frameworks). Since this is a public sync entrypoint, it should fail with a clearer error (or otherwise handle the running-loop case) and direct callers to run_async().

This issue also appears on line 192 of the same file.

        """Run a new turn synchronously."""
        return asyncio.run(
            self.run_async(
                session_id,
                turn_id,

• Files reviewed: 28/28 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

runtime/python/prompty/prompty/harness/engine.py:923

• _tool_result_messages calls json.dumps(result.output) for any non-string output. If a tool returns a non-JSON-serializable value (e.g., bytes, datetime, custom objects), this will raise TypeError while constructing the tool message and can fail the entire turn after the tool has already run. Consider serializing with a safe fallback and a stable/compact encoding, and avoid dumping None before the explicit check.

            value = result.output if isinstance(result.output, str) else json.dumps(result.output)
            if result.output is None:
                value = ""

• Files reviewed: 28/28 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (5)

runtime/python/prompty/prompty/harness/engine.py:194

• ReferenceTurnEngine.resume() has the same asyncio.run(...) nested-event-loop failure mode as run(). Consider detecting an already-running loop and raising a clearer error (or requiring callers to use resume_async() in async contexts).

        """Resume a durable checkpoint synchronously without repeating committed effects."""
        return asyncio.run(self.resume_async(context, cancellation=cancellation))

runtime/python/prompty/prompty/harness/engine.py:148

• ReferenceTurnEngine.run() unconditionally uses asyncio.run(...), which raises RuntimeError: asyncio.run() cannot be called from a running event loop in common environments (e.g., Jupyter/IPython, async web frameworks). Since this is a public sync API, it should either detect an active loop and raise a clearer Prompty-specific error instructing callers to use run_async(), or avoid asyncio.run entirely.

This issue also appears on line 193 of the same file.

        """Run a new turn synchronously."""
        return asyncio.run(
            self.run_async(
                session_id,
                turn_id,

.github/workflows/prompty-python.yml:42

• The PR description and validation commands say generated conversion tests under tests/model remain excluded from the gate, but this workflow only ignores a single file (tests/model/agent/test_prompty.py). If the intent is to keep all generated model tests out of CI until the emitter fixes whitespace, the workflow should ignore tests/model (or the PR description should be updated to match the narrower exclusion).

      - name: Run tests
        working-directory: ./runtime/python/prompty
        # Typra-generated multiline conversion fixtures remain outside the gate until the emitter preserves whitespace.
        run: python -m pytest tests/ --ignore=tests/model/agent/test_prompty.py -q --tb=short

.github/workflows/prompty-python-check.yml:56

• Same gating mismatch as prompty-python.yml: this job says conversion fixtures are outside the gate, but only ignores tests/model/agent/test_prompty.py. If the intended exclusion is the full generated suite, ignore tests/model here as well (or update the PR description to reflect the narrower exclusion).

      - name: Run tests
        working-directory: ./runtime/python/prompty
        # Typra-generated multiline conversion fixtures remain outside the gate until the emitter preserves whitespace.
        run: python -m pytest tests/ --ignore=tests/model/agent/test_prompty.py -q --tb=short --cov=prompty --cov-report=term --cov-report=json

runtime/python/prompty/.env.example:12

• This PR’s stated goal is to require explicit opt-in for live image model coverage, but .env.example sets OPENAI_IMAGE_MODEL to a non-empty default. Users who copy it verbatim will unintentionally enable live image integration tests (and associated cost). Consider leaving it blank with a comment indicating it’s optional.

OPENAI_MODEL=gpt-4o-mini
OPENAI_EMBEDDING_MODEL=text-embedding-3-small
OPENAI_IMAGE_MODEL=gpt-image-1

• Files reviewed: 30/30 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 30/30 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

runtime/python/prompty/tests/test_turn_runner.py:442

• This test hard-codes the final event id ("session-event-7"), which effectively locks in the exact number/order of events emitted by ReferenceTurnRunner. Any legitimate new event type (or reordering) will break the test even though the underlying guarantee being tested is uniqueness / monotonic id generation.

Consider asserting that ids are unique and sequential (based on the observed list length) instead of asserting a specific final id value.

    event_ids = [record["event"]["id"] for record in _records(journal_path) if record["kind"] != "summary"]
    assert len(event_ids) == len(set(event_ids))
    assert event_ids[0] == "session-event-1"
    assert event_ids[-1] == "session-event-7"

• Files reviewed: 30/30 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 30/30 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Files not reviewed (1)

• schema/package-lock.json: Generated file

Suppressed comments (4)

schema/package-lock.json:481

• This lockfile entry is for @typra/emitter 0.4.7, but the PR description says regeneration was restored to the committed 0.4.2 baseline until upstream issues are fixed. If 0.4.2 is still the intended pinned version, this section should be reverted (or, if 0.4.7 is intended, the PR description should be updated accordingly).
  schema/tsp-output/.typra-generated/export-surfaces.json:22
• The PR description says the repo was restored to the committed Typra emitter 0.4.2 baseline (keeping the PR unmerged until upstream fixes land), but this generated toolchain metadata now reports @typra/emitter 0.4.7 as the supported version/range. This makes it unclear which emitter version is actually supported for regeneration and may cause contributors to regenerate with an incompatible emitter.
  schema/package.json:16
• The PR description indicates Typra regeneration was rolled back to the 0.4.2 baseline due to failing probes, but schema/package.json pins @typra/emitter to 0.4.7. Either the description needs updating to reflect that 0.4.7 is now acceptable, or the dependency should remain at 0.4.2 to match the stated gating strategy.
  schema/package-lock.json:11
• schema/package-lock.json is updated to @typra/emitter 0.4.7, which conflicts with the PR description’s statement that the repo was restored to the 0.4.2 baseline until emitter probes pass. If the baseline is still 0.4.2, the lockfile should reflect that to avoid accidental regeneration/toolchain drift.

This issue also appears on line 479 of the same file.

• Files reviewed: 60/61 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Files not reviewed (1)

• schema/package-lock.json: Generated file

Suppressed comments (5)

schema/package-lock.json:481

• The resolved @typra/emitter tarball and integrity hash are pinned to 0.4.8 here, but the PR description says the repo should stay on 0.4.2 until the emitter adoption probes pass. This entry should match whatever version is actually intended (and should be regenerated by npm to keep the lockfile consistent).
  schema/package.json:17
• The PR description says schema generation was restored to the committed @typra/emitter 0.4.2 baseline due to failing adoption probes, but this change bumps the dependency to 0.4.8. This makes the repo state inconsistent with the stated Typra follow-up plan and may reintroduce the probe failures described in the PR.
  schema/tsp-output/.typra-generated/export-surfaces.json:22
• This generated export-surface metadata now reports @typra/emitter 0.4.8, but the PR description indicates the toolchain should remain pinned to 0.4.2 until the emitter fixes land. If the schema dependency is reverted, this file should reflect the same pinned version/range.
  schema/package-lock.json:12
• schema/package-lock.json is updated to @typra/emitter 0.4.8, which conflicts with the PR description’s statement that regeneration churn was restored to the committed 0.4.2 baseline. If the project remains pinned to 0.4.2, the lockfile should be consistent (typically via re-running npm install after reverting schema/package.json).

This issue also appears on line 479 of the same file.
runtime/python/prompty/tests/test_model_capabilities.py:38

• The drift-guard failure message suggests a PowerShell-only Copy-Item command, but this test suite runs cross-platform. Including a platform-neutral command (and optionally PowerShell as an alternative) makes the failure actionable on Linux/macOS CI and local dev environments.

• Files reviewed: 69/70 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Files not reviewed (1)

• schema/package-lock.json: Generated file

Suppressed comments (5)

schema/package-lock.json:481

• Lockfile entry for the installed @typra/emitter package is bumped to 0.4.8. If this PR is intended to stay on the 0.4.2 baseline per the description, this section should also be reverted/regenerated so the resolved tarball and integrity match 0.4.2.
  schema/package.json:17
• PR description says regeneration churn was restored to the committed @typra/emitter 0.4.2 baseline until upstream fixes land, but this change pins @typra/emitter to 0.4.8. Unless the PR description is updated (and the probe failures are re-validated for 0.4.8), the version bump should be reverted to keep the repo on the stated baseline.
  schema/package-lock.json:12
• schema/package-lock.json is also updated to @typra/emitter 0.4.8, which contradicts the PR description’s stated 0.4.2 baseline. If the baseline should remain 0.4.2, revert these lockfile entries (or regenerate the lockfile after pinning package.json back).

This issue also appears on line 478 of the same file.
schema/tsp-output/.typra-generated/export-surfaces.json:22

• This generated toolchain manifest now reports @typra/emitter 0.4.8, which conflicts with the PR description’s claim that Typra churn was restored to the 0.4.2 baseline. If the intent is to stay on 0.4.2 until the upstream probes pass, revert this manifest version accordingly (it should be regenerated from the pinned dependency).
  runtime/python/prompty/prompty/harness/engine.py:761
• _checkpoint() currently forces optional list fields to empty lists ([]) even when there are no pending tool requests/results. Since the emitted save() logic serializes a collection whenever it is non-None, this makes checkpoints always include empty pendingToolRequests / completedToolResults, diverging from other runtimes that omit empty collections (e.g., Rust only writes these keys when non-empty). Preserve the omission semantics by leaving these fields as None unless they have items.

            pending_tool_requests=list(pending_tool_requests or []),
            completed_tool_results=list(completed_tool_results or []),

• Files reviewed: 69/70 changed files
• Comments generated: 0 new
• Review effort level: Lite