Add a Swift runtime for Prompty - #439
Open
sethjuarez wants to merge 47 commits into
Open
Conversation
Add a Swift emit target so the canonical Prompty types are emitted from
schema/model/**/*.tsp like every other runtime, rather than hand-written.
The generated package lands at runtime/swift/prompty-model as a standalone
SwiftPM package because the emitter owns Package.swift.
test-dir is deliberately omitted from the emit target: the emitter's
generated Swift tests do not compile (297 errors across 4 further defects),
and driver.js treats testRoot as optional, so a library-only package is a
supported configuration.
@typra/emitter@0.4.2 has defects that make its Swift output either
uncompilable or lossy, so generation is followed by a scripted, idempotent
normalization pass. It is never a hand edit of generated files: the shim runs
as part of `npm run generate` and is designed to retire itself once the
emitter is fixed. Change requests for all of these have been sent upstream.
1. Wildcard enum cases referenced but never declared, with no save branch
2. Recursive enums not marked `indirect` (Property <- ArrayProperty.items)
3. Convenience factories built from raw literals instead of enum values
4. Unmapped placeholder type names in protocol signatures
5. Array/optional suffixes dropped from protocol signatures
10. `extends` base fields dropped from derived structs -- silent data loss
across 3 Property subtypes and all 5 Tool subtypes
Defect 10 is the dangerous one, so it is patched structurally rather than by
literal find/replace: base declarations, load branches and save branches are
injected at known anchors in each derived struct. The shim pins the emitter
version, throws when the Swift output root is missing, and refuses to run
against partially-patched or ambiguous input, so a future emitter release
cannot silently produce half-patched code.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Introduces the `Prompty` module: loading, rendering, parsing, preparation,
and the invoker registry, built on the Typra-generated `PromptyModel`
package. The generated model is the only domain type layer -- this package
adds behavior, not a parallel set of types, and conforms to the pipeline
protocols the emitter already produces rather than declaring its own.
The Rust runtime is the behavioral reference. Loading resolves `${env:}`
and `${file:}` references, migrates legacy frontmatter, and injects the
prompt discriminator. Preparation stamps role markers with a nonce before
rendering and requires that nonce when parsing, so markers that appear only
after interpolation are rejected instead of silently becoming new turns.
Nonce substitution happens in exactly one place for the same reason.
Templates are evaluated by a real tokenizer and precedence-climbing parser
rather than shape matching, so an expression the runtime does not support
is reported instead of quietly resolving to nothing.
Conformance runs against the shared spec vectors -- 25 load, 23 render,
and 15 parse -- plus round-trip guards over every base field the Swift
generation shim injects, and regression tests covering defects that the
vectors do not reach.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Adds `PromptyOpenAI`: the executor that turns a prepared conversation into an OpenAI request, and the processor that turns the response back into a result. Connection details, structured output, and tool declarations come from the generated model, so the provider translates between that model and the wire rather than defining shapes of its own. Providers register themselves. Importing the module and calling `registerOpenAI()` installs the executor and processor, which keeps the core runtime free of provider dependencies. Both directions are checked against the shared spec vectors: `spec/vectors/wire` for the request shape and `spec/vectors/process` for response handling, including structured output and streamed responses. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Adds the reference harness adapters and a turn runner that drives the loop the durability contract describes: ask the model, checkpoint, resolve permission, execute the tool, report the result, and go again until the model answers or the iteration budget runs out. Every event is handed to the sink before it is journaled, so no observer can see an event that was never recorded. The clock and id factory are injected, which makes a run byte-reproducible and lets replay compare against a recorded journal rather than a re-derived one. The journal writer appends and flushes per record, so a run interrupted mid-flight still leaves a readable prefix -- covered by a test that reads the file while the run is in progress. Tool failures are contained and reported as results rather than propagated, since a failing tool is an outcome the model has to see, not a crash. Verified against `spec/vectors/harness` across all five replay scenarios, with adapter-level tests for crash durability, checkpoint ordering and session scoping, permission decisions, tool failure containment, and verifier drift detection. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Swift treats a CRLF pair as a single Character, so the obvious line
splits silently do nothing: for "a\r\nb" both components(separatedBy:)
and split(separator:) return the whole string as one element, and
text.contains("\r") is false. A prompt assembled in memory with Windows
endings therefore missed every role marker and collapsed into one lumped
message, a CRLF journal replayed as zero records, and the buffered SSE
path yielded a single unparseable line.
Rust splits at the byte level and degrades gracefully, so this hazard has
no counterpart in the reference runtime and no LF-only spec vector can
catch it.
Add Lines, a scalar-based splitter, and route the parser, the JSONL
journal reader and the SSE fallback through it. Scanning unicodeScalars
rather than Characters is what makes this correct: normalizing first and
splitting on characters afterwards still fails, because rewriting
"a\r\r\nb" yields "a\r\nb" whose CR and LF are now adjacent and cluster
into one grapheme again.
Character.isNewline is deliberately not used. It also matches U+0085,
U+2028 and U+2029, which are legal unescaped inside JSON strings, so
splitting on them tore journal records in half and discarded them
silently. Only CR, LF and CRLF terminate a line. Normalization rewrites
CRLF and nothing else, matching the Rust loader, so a lone CR stays as
content.
Also add seven end-to-end tests against the real OpenAI API covering
chat, token caps, streaming, streamed accumulation, structured output,
tool calling and loading a .prompty file with ${env:} references. They
skip when OPENAI_API_KEY is absent so a keyless CI stays green. The
assertions avoid vacuous passes: the token cap is a relative two-call
comparison, structured output is proven by a prompt that demands prose
and forbids JSON, and tool use is proven by an opaque run-time station
code the model cannot guess.
Every fix here was mutation-verified by reverting it and confirming the
predicted failure.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Cover the two-package layout and why it is split, the rule that nothing under prompty-model/Sources is ever hand-edited, the Windows git escape hatch SwiftPM needs, how to run the live tests, and the emitter shim — including that it is pinned, fails loudly, and is meant to shrink to nothing as upstream fixes land. The snippets are type-checked by ReadmeSnippetTests rather than trusted: an earlier draft documented a Prompty.load(path:) entry point that does not exist. Documented calls are API surface and should break the build when they drift. State plainly that spec/vectors/engine/turn_vectors.json describes a turn engine this port does not implement, so those vectors are not run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ArrayProperty, ObjectProperty and UnionProperty each extend Property, so they inherit description, required, nullable, default, example and enumValues, plus name via the Named<Property> alias. The Typra emitter drops these fields from derived Swift structs and the loss is silent: the code compiles and the values simply disappear. The build-time shim restores them, but the shared spec vectors only exercise inherited fields on scalar properties, so a regeneration that reintroduced the defect on the composite subtypes would go unnoticed along several axes. Add an acceptance gate covering the axes the existing round-trip suite does not reach: JSON and YAML text round trips, values constructed in Swift rather than parsed, repeated round trips reaching a fixed point, non-scalar values in default/example/enumValues, falsy values that the save() guards could quietly discard, and inherited fields on nested object children, union branches and three-level composites. The assertions target individual inherited fields rather than comparing whole dictionaries, because a corrected emitter may legitimately begin materializing schema defaults such as required: false, which a strict key-count comparison would reject for reasons unrelated to this defect. These tests assert the required behaviour rather than the shim, so they are expected to stay green once the emitter is fixed. Verified by mutation: with the shim's Property injection disabled the library still builds with zero errors, confirming the defect is silent, while the construction test fails to compile and the nine runtime tests report 268 assertion failures spread across every inherited field. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The 135 generated PromptyModel files already pass `swift-format lint --strict` with zero findings, so swift-format defaults are the house Swift style in this repo. The hand-written runtime did not conform. Applied `swift-format format --in-place` across Sources and Tests, then hand-fixed one multi-line string literal in LiveOpenAITests that the formatter reindented only at its closing delimiter -- Swift strips the closing delimiter's indentation from every line, so the body had to be indented uniformly to match. The literal's string content is unchanged, which testLivePromptFile confirms by still round-tripping the same .prompty document through a live call. No generated model file is touched. Verified: swift-format lint --strict => 0 findings; swift build --build-tests => 0 errors, 0 warnings; swift test => 64 tests, 0 failures, including all 7 live OpenAI E2E tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
runtime/swift carries a temporary post-processing shim that repairs known Swift emitter defects, and that shim asserts every one of its find/replace patches still applies. That is the right default -- it fails loudly instead of rotting silently -- but it also means a candidate emitter build cannot be measured at all, because generation aborts as soon as the candidate stops matching the patches. This flag emits raw, unpatched Swift so a candidate can be evaluated against the defects the shim compensates for. It was used to validate two emitter candidates and to produce the defect reports filed upstream. The comment records the retirement criterion. A clean `swift build` is deliberately not sufficient: dropped inherited fields on Property subtypes are read back through save(), so they vanish at runtime with no compile error at all. `swift test` and InheritedPropertyFieldTests are the gate for that. Verified: with the flag unset, `npm run generate` reproduces runtime/swift byte-identically to HEAD. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Every other runtime has a prompty-<lang>-check.yml; Swift had none, so nothing in CI compiled or ran the new runtime. Matrix is ubuntu-latest and macos-latest. Only Linux installs a toolchain -- macOS runners ship Swift with Xcode. windows-latest is deliberately omitted: swift-actions/setup-swift maps Windows releases only through 5.6.3, and this needs Swift 6.x for the `swift format` subcommand. Windows is covered by local development for now. `spec/**` is in the trigger paths because the Swift tests are validated against the shared cross-runtime vectors under spec/vectors, so a spec change has to re-run them. The other runtimes' check workflows watch only their own runtime directory and have the same blind spot; fixing those is out of scope here. The SwiftPM cache keys on a stamp of `swift --version` as well as the manifests, so a runner-image or toolchain change can never restore artifacts produced by a different compiler. Package.resolved is untracked (a library must not pin its consumers), so it cannot contribute to the key. Only the hand-written runtime is format-checked. The generated PromptyModel sources pass swift-format today, but they are the emitter's output and an upstream formatting change must not break CI. swift test is hermetic here: LiveOpenAITests calls XCTSkipIf when OPENAI_API_KEY is unset. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
`Frontmatter.split` located its delimiters with `firstIndex(of: "\n")`. Swift clusters a CRLF pair into a single `Character` that is not equal to `"\n"`, so in a Windows-authored document the search found nothing, the opening `---` looked unterminated, and the function returned empty frontmatter and an empty body. No error was raised. `Loader` normalized before calling in, so every existing test and every shared vector passed and the defect was invisible; only direct callers of the public API lost their document. Rewrite `split` onto `Lines.splitLineFeeds`. Normalizing and then splitting on characters is not sufficient -- as `Lines` documents, one pass over `a\r\r\nb` yields `a\r\nb`, whose CR and LF re-cluster into a single grapheme. Only a unicode-scalar scanner is safe. That makes `split` the single normalization pass in the load path, so `Loader` no longer normalizes first: two passes would read the residual CR + LF of `\r\r\n` as a terminator and delete a lone CR that `Lines` and the Rust reference both preserve. Blank-line skipping uses `allSatisfy(isWhitespace)` rather than `CharacterSet.whitespaces`, which is horizontal-only and would have demoted a document opening with a vertical tab, form feed, NEL, or a Unicode separator to body-only. Equivalence with the replaced algorithm was verified by a temporary differential harness fuzzing both implementations over a 54-sample corpus, including a negative control confirming it detects that divergence. Also widen the SSE payload trim from `.whitespaces` to `.whitespacesAndNewlines`; the narrow set is space and tab only, so a surviving CR left `"[DONE]\r"` and defeated the sentinel comparison. Add 12 line-ending tests covering the regression, CRLF/LF equivalence, .prompty files with CRLF bytes on disk, lone-CR survival end to end, vertical whitespace before the delimiter, and CR-tolerant SSE framing. Correct the README conformance section: report the partial OpenAI-only coverage of the wire and process vectors, and stop claiming the port has no turn engine -- `ReferenceTurnRunner` implements the loop, permission mediation, and checkpointing. The genuine gaps are delegated provider state and cancellation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
0.4.3 ships the Swift source-generator fixes this repo requested. Determine
per-defect what actually landed by generating unpatched Swift with both 0.4.2
and 0.4.3 (PROMPTY_SKIP_SWIFT_SHIM=1) and diffing the two, rather than
inferring from release notes: 10 files, 18 insertions, 16 deletions.
Fixed upstream, so the corresponding 13 patch entries are deleted:
- Tool wildcard `case customTool(CustomTool)` plus load/save arms
- `indirect` on recursive polymorphic enums
- typed factory cases (`role: .assistant`, `.textPart(TextPart(...))`)
- placeholder types (`Unknown` -> `Any?`, `RecordUnknown` -> `[String: Any]`)
- array/optional suffix loss in protocol signatures
Still broken at 0.4.3, so those patches are retained and loudly asserted:
- `Connection` has no `unknown` case although tool.swift defaults six fields
to `.unknown([:])` (hard compile error)
- fields inherited via `extends` are dropped from derived structs (silent
data loss, zero compile errors -- so a clean build cannot be the retirement
criterion for this shim)
- map-form `bindings` is never emitted
Net effect: the shim drops from 419 to 345 lines while producing generated
Swift that is byte-identical to the 0.4.2 output except two cosmetic lines.
Also harden the shim against emitter drift, since its whole value is being
deterministic. Patch application now requires exactly one anchor: a duplicated
patched form, or a patched form sitting alongside a stray raw anchor, both
fail instead of silently producing a half-patched file. Single-site replacement
via indexOf/slice replaces split/join. Missing-manifest version assertion now
throws rather than returning silently.
The generated Swift test directory stays disabled. Measured, not assumed:
enabling test-dir emits 125 files and 1184 compile errors in four classes --
unwrapped optionals, struct-style member access on polymorphic enums, compound
values compared to their scalar shorthand, and references to helpers that are
never emitted (`FixtureRoot`). These are test-generator defects, disjoint from
the source-generator defects 0.4.3 fixed; the tspconfig comment now records
that instead of the stale 0.4.2 note.
Cross-runtime effect of the bump, all measured against HEAD rather than
assumed: Go improves from 44 failures to 17, C# is unchanged at 17/1089,
Rust stays fully green. Swift is 76 passed / 0 failed both with live
credentials and without (7 E2E skipped), build and strict-lint clean, and
regeneration is deterministic.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Reverts the generated-output and pin changes from 66b60ec. CI is the authority here: the C# suite was green at 2a724c0 and fails on all three platforms at 66b60ec, with 16 `LoadYamlInput*`/`RoundtripYaml*` tests in Prompty.Core.Tests newly broken by the reshaped generated-test string literals. That is the same regression that already disqualified 0.4.6. The local pre-merge measurement missed it, and the reason is worth writing down. Locally the same 16-test class fails at HEAD too -- but as `LoadJsonInput*`/`RoundtripJson*`, a Windows CRLF artifact of this checkout. Counts are 16 before and 16 after, so a count comparison reports "unchanged"; the failing identities are in fact completely disjoint. This is precisely the trap recorded for 0.4.6, and it was walked into anyway by comparing totals instead of names. The pin note now says so explicitly. 0.4.3 remains a genuine Swift improvement -- it fixes the Tool wildcard, `indirect`, typed factories, placeholder types and suffix loss, and would cut this shim from 419 to 345 lines. But the emitter is shared, so a bump regenerates every runtime, and Swift gains cannot be bought with a C# break. Both evaluated releases and their exact costs are now recorded next to PINNED_EMITTER_VERSION so the next reader does not repeat the probe. Kept from the reverted commit, because none of it depends on the emitter version: - Patch application is now fail-closed on ambiguity. It previously used `includes()` plus `split/join`, which would silently accept a file that was half patched, or rewrite several sites when the anchor was not unique. It now requires exactly one occurrence of either the raw anchor or the patched form, and rejects a patched form sitting next to a stray raw anchor. Both failure modes are exercised directly; all 16 patches satisfy the invariant. - The missing-manifest branch of the version assertion throws instead of returning silently, so a shim run without `npm install` can no longer patch against an unverified emitter. - The omitted Swift `test-dir` is documented from measurement rather than memory: enabling it emits 125 files and 1184 compile errors in four classes -- unwrapped optionals, struct-style member access on polymorphic enums, compound values compared to their scalar shorthand, and references to helpers that are never emitted. All four are test-generator defects, still present at 0.4.3 and disjoint from the source-generator defects it fixes, so those fixes could not have made these compile. Generated output is byte-identical to 2a724c0. Swift: build clean, strict lint clean, 76 passed / 0 failed with live credentials. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Defect 10 in the shim header listed `name` among the fields that `ArrayProperty` / `ObjectProperty` / `UnionProperty` and the `Tool` subtypes lose to the emitter's dropped `extends` inheritance. That is inaccurate: neither `model Property` nor `model Tool` declares `name`. It arrives through the `Named<...>` spread in `schema/model/core/core.tsp`, which the emitter drops by a separate mechanism. The distinction is not cosmetic. It decides part of the shim retirement criterion this PR is blocked on: an upstream fix to `extends` inheritance restores every other field in that list but leaves `name` missing, so removing the `name` injection on the strength of such a fix would reintroduce silent data loss with no compile error to catch it. The header now says so explicitly and directs the reader to confirm against regenerated output. Comment-only; generated output is unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Probed the current registry frontier with the shim disabled, so every claim below is a measurement of native emitter output rather than a reading of release notes. Swift source generation is close to clean at 0.4.8. Native output compiles to exactly 30 errors, all `Connection has no member 'unknown'`, all in tool.swift; probe-patching only that enum case makes the model package compile. Inherited `extends` fields are fixed too, including `name` -- which matters, because `name` arrives via the `Named<...>` spread rather than `extends`, so a narrower upstream fix would have restored every other base field and silently dropped that one. That is still not grounds to retire a patch on a clean build alone. Dropped fields produce no diagnostics, so the retirement criterion in defect 10 stands: `swift build --build-tests` and `swift test`. 0.4.8 is rejected all the same. The C# `*Yaml*` break reproduces, and the cause is now identified: agent.tsp:166 genuinely ends in a trailing space, which the 56 escaped expected-value literals preserve and the 24 verbatim input-YAML literals drop. At 0.4.2 both forms dropped it consistently, so the suite passed. Also re-measures the generated test-dir, which the library must compile before it is observable at all: 180 errors, down from 1184 at 0.4.3, in four test-generator classes that are now described from the diagnostics they actually produce. Comments only -- generation output is byte-identical. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The shim header doubles as the upstream defect report, and it bundled two
unrelated problems into a single 'Defect 1'. That conflation had a real cost:
successive emitter candidates fixed the Tool wildcard, were reported as
fixing Defect 1, and the Connection build errors persisted unchanged.
Split into two entries with distinct owners:
1a (emitter) Tool is missing a customTool case even though tool.tsp
declares CustomTool { kind: '*' }.
1b (emitter) The emitter synthesises Connection = .unknown([:]) defaults
for six required fields in tools/tool.swift while never declaring the
case, so raw output does not compile on its own terms. Verified by
deleting only the injected case declaration and rebuilding: 16 errors,
all
"
type 'Connection' has no member 'unknown'
"
, at
tool.swift:145/152/232/242/344/351 plus the two injected arms.
Schema gap connection.tsp closes the union over six kind literals, so
throwing on an unrecognised discriminator is correct emitter output.
Overriding it is a deliberate choice, not a defect report.
Three claims were corrected after review rather than shipped:
- The header cited spec.md 2.5 as requiring lossless unknown-kind
round-trips. It does not; 2.5 only tabulates the six known kinds on this
branch. The override is now described as extending the 2.3 unknown-
property rule pending a 2.5 amendment, matching what
ConnectionRoundTripTests already documents.
- It claimed fixing 1b retires none of the three patches. This file's own
0.4.10 log records that release emitting the declaration and save arm
natively, which retires two. Retirement now depends on the shipped fix.
- It cited Rust as precedent for lossless round-trips. Rust maps unknown
kinds to ConnectionKind::default() (connection.rs:258) and kind_str has
only six arms (274-283), so it rewrites the discriminator and drops the
payload. It is precedent for not throwing only; Swift is stronger, and
that divergence is unresolved rather than settled parity.
The exit condition no longer says 'regenerate and delete'. Defect 1a proves a
declared wildcard does not guarantee correct projection, so it now requires
regenerating with the patches removed and measuring that raw output builds,
preserves an unknown kind's discriminator and payload, and passes the suite.
Regenerated at the pinned 0.4.2 so the emitted comment matches the script.
Swift regeneration is otherwise byte-identical. 148 tests, 3 skipped,
0 failures.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
An open schema question is how to open the closed
"
Connection
"
union: add a
typed wildcard subtype (as
"
CustomTool { kind:
"
*
"
}
"
does for
"
Tool
"
), or open
the discriminator so the raw payload survives. The two are routinely treated
as interchangeable. They are not, and nothing measured that until now.
Adds WildcardPreservationTests (4 tests) covering:
- the typed subtype drops unknown top-level fields, because CustomTool
declares only named fields and no raw catch-all;
- raw passthrough preserves every field, unrecognised discriminator
included;
- the two disagree on identical unknown fields, pinned as one assertion so
the divergence cannot close in either direction unnoticed;
- a missing *required* connection is accepted without diagnostic and saved
as an empty, kind-less connection.
That last one sharpens emitter defect 1b. It was reported as a compile break
the shim absorbs; it also produces invalid output. CustomTool.connection is
required (tool.tsp:98), but the loader assigns it only when present
(tool.swift:167-169), so the synthesised .unknown([:]) placeholder survives
and save writes it unconditionally (tool.swift:198). Rust guards the same
write with !connection.is_null() (tool.rs:327-329) and omits the key, so this
is a Swift-specific divergence rather than agreed behaviour.
Scoped deliberately narrowly. What is shown is that a wildcard subtype
declaring only named fields loses unknown ones -- corroborated cross-runtime,
since Rust's ToolKind::Custom captures only connection/options/kind_name
(tool.rs:175-185). It does not follow that any conceivable Connection
wildcard must, as one could declare catch-all storage. And dropping unknown
properties is not a spec violation: spec.md 2.3 permits them to be preserved
or ignored. The transferable point is that the choice is not neutral and the
declared shape decides it.
Header comments in the shim are updated to match, replacing an earlier
categorical claim that mirroring CustomTool onto Connection would be a
regression. Comment-only; generated output verified byte-identical.
All 6 new assertions mutation-proved. Suite 152 tests, 3 skipped, 0 failures.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
parseAttributes coerced every attribute value to Int or Double before falling back to String. The strict-mode nonce is 8 random bytes rendered as 16 hex characters, and hex is a superset of decimal, so the nonce was sometimes numeric-looking and did not survive the round trip: 9677e80871924237 -> inf (exponent overflows Double) 0419856025378190 -> 419856025378190 (leading zero lost to Int) 9789350921772800 -> 9.7893509217728e+15 (normalized on stringify) buildMessage then compared the corrupted value against the nonce that produced it, so the parser rejected its own untampered output with "possible prompt injection detected". Measured 165 failures in 200,000 trials, about 1 in 1,200. This surfaced as an intermittent failure of testLiveToolCalling, which has no template inputs at all -- the nonce was the only thing varying between runs, and the throw happened in about a millisecond, before any network call. CI never saw it because live tests need credentials CI does not have. Exempt nonce from coercion. It is already reserved: it is generated by the parser, consumed by the parser, and stripped from metadata rather than surfaced to callers, so coercing it was never meaningful. Coercion is preserved for every other attribute, since attributes such as [index=1] are documented to arrive typed. Rust hit the same problem and worked around it at the comparison site (parsers/prompty.rs:174-182, "parse_attrs may coerce all-digit hex nonces to Number"), mapping Number back through to_string. That workaround is incomplete: parse_attrs strips leading zeros, so a nonce like 0123456789012345 still fails to compare equal. Fixing the root cause is strictly stronger and behaves identically wherever Rust behaves correctly, so this diverges from the reference only where the reference is broken. Reported upstream separately. The new tests use crafted numeric-looking nonces, one per corruption mode, so a regression fails deterministically rather than once in 1,200 runs, plus a 20,000-trial sweep of real generated nonces to catch modes nobody anticipated. Both directions are mutation-proved: removing the exemption fails 6 of 7 tests, and "fixing" it by dropping coercion wholesale fails the coercion-preservation test. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The named-collection vector suite compares with deliberate subset semantics, which is correct: fields the runtime adds beyond the fixture are tolerated while the canonical file is still being revised upstream. The gap was one step earlier. `canonicalizeExpected` rewrites a fixture's nested name-keyed `properties` map into the ordered list the wire uses so the semantic assertions can run at all. That same adaptation erases the shape difference, so a nested collection saved as the array fallback compared equal to a fixture asking for the canonical object form. The top-level fallback was already recorded, and NamedCollectionSaveFormTests covers duplicate and empty names, but nothing saw a nested one. Walk the fixture and the saved wire in parallel and record nested object-form gaps into the existing `blocked` baseline, which is tied to the emitter pin, so bumping the pin without closing the gap fails rather than resting on stale prose. Only the fixture's own structure drives it: a nested map is read as a request for the object form, a nested list as a request for the array form, and nothing is inferred about collections the fixture does not mention. Verified end-to-end by authoring the absent fixture locally: the gap reached `blocked` and the pin check named the exact path. The fixture was then removed, so the vector test still skips on absence. Review follow-ups folded in: - Pairing children by name alone dropped a saved child that omits `name` and reported nothing beneath it. That unnamed composite is precisely the canonical case, so pair positionally when both sides are ordered lists, and fall back to position when a name lookup fails. - Every test drove the helper directly, so deleting the call in runLoadSaveReload left them all green. Added a test that drives the real vector path and asserts the gap arrives in `blocked`. - Renamed the justification test: `compare` on its own would see the shape mismatch. It is the canonicalised pipeline that goes blind, and the name now says so. - Pinned the alphabetical ordering of nested named collections, which both the canonicalisation and the positional fallback rely on and neither stated. 169 executed, 3 skipped, 0 failures, 7 live OpenAI E2E. Mutating away the wiring and the unnamed-child fallback each kill exactly one test. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The spec coordinator ruled that object-form named collections are keyed
rather than order-bearing, and that only the array fallback carries order.
Post-save positional lookup is therefore invalid for object-form
collections.
`nestedObjectFormGaps` descended by name but fell back to indexing the
saved array whenever a key missed, which is exactly that invalid lookup.
It had two concrete failure modes:
- misattribution: a differently-named child bound to the wrong key, so
that child's nested gap was reported against a key it does not belong
to;
- silent loss: when a named child resolved to a later index, the earlier
nameless child was never reached and its gap disappeared.
Pair by elimination instead. Names address every child they can; the one
child a key cannot address is an entry that omits `name` in the array
fallback, and that entry is matched only when a single unmatched key faces
a single nameless child. Anything more ambiguous is left unpaired rather
than guessed, since attributing a gap to an unproven key is the same
misattribution keyed pairing exists to prevent. Nameless now means the
field is absent, so a malformed `name` is treated as a mismatch.
Both guards are load-bearing, and both are caught by the real canonical
vector test rather than only by the synthetic ones:
- disabling elimination fails testEliminationPairingIgnoresPosition,
testNestedObjectFormGapIsFoundWhenTheSavedChildOmitsItsName, and
testCanonicalNamedCollectionVector;
- removing the ambiguity guard fails
testAmbiguousNamelessChildrenAreNotGuessed and
testCanonicalNamedCollectionVector.
Also corrects comments that justified the positional fallback by asserting
both sides order collections alphabetically. The alphabetical-order test
keeps earning its place, but for the array save form, which does carry
order, and because canonicalizeExpected sorts a keyed expectation into a
list that compare then pairs positionally.
172 executed / 3 skipped / 0 failures, including 7/7 live OpenAI E2E.
Emitter pin 0.4.2 unchanged; no generated files touched.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Canonical rule: an absent optional collection must stay omitted through save, an explicitly present empty collection may save as empty, and save must never synthesize an empty collection from absent input. Adds OptionalCollectionPresenceTests covering both halves of the rule and the distinction between them, over all three composite Property kinds (array/object/union) plus the document-level `tools` collection. Swift conforms at emitter pin 0.4.2: all five gates pass unmodified. Each absence assertion is paired with a positive control, because an empty save output or a silent fall-through to the `.unknown` passthrough - which echoes its source dictionary verbatim - would otherwise satisfy every "must be nil" assertion while exercising no composite code at all. The controls assert the loaded enum case, not just the saved shape, since only the case distinguishes a real composite from the passthrough. Also removes the `isMaterializedEmptyDefault` exemption from InheritedPropertyFieldTests. It tolerated an empty array standing in for an absent value on the rationale that "a corrected emitter materializes the schema default for enumValues" - which this rule inverts: doing so is itself the defect. The new gates prove the pinned emitter does not synthesize, so the exemption was dead code, but it would have silently accepted a future regression. Full suite green. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Canonical rule 2 is shared, not a Java special-case: a duplicate-name save must pre-scan exact uniqueness and fall back to the whole array *before* any map construction. Applying it to the Swift comparators found two places that built the map first and let a later entry overwrite an earlier one - the same silent-loss class as the positional pairing bug, arriving through map collapse instead of indexing. NamedCollectionVectorTests.namedChildren now pre-scans and returns an optional, where nil means "not soundly name-addressable". Refusal is nil rather than an empty index because the two mean opposite things to the caller: an empty index says the collection has no named children, which leaves every expected key unmatched and therefore eligible for the nameless-child elimination pairing. A duplicate would then bind an unrelated child to the very key it was hiding - "no proven owner" decays into "sole remaining candidate". Refusal now suppresses lookup and elimination together. A present-but-non-string `name` refuses the index for the same reason: it falls out of the uniqueness scan while also not counting as nameless, so it would vanish from both pairing paths without trace. The expected-list against actual-object path now requires unique expected names too. Uniqueness on the actual side alone still lets several expected entries be attributed to one actual child, reporting a nested gap once per duplicate against entries never separately observed. LoadVectorTests.bindingPairs throws instead of collapsing duplicates. Its sole consumer pins exactly one binding and its [String: String] shape cannot represent duplicates, so collapsing would shrink a spec-vector expectation and keep the vector passing while checking fewer bindings than it declares. This is correct for that pin only - under rule 2 duplicate entries in the array fallback are legitimate ordered entries, so it is not a general binding-list interpretation. Three tests cover the refusal modes. All five guards are mutation-proved: each mutation compiles cleanly and kills exactly one test, including `return nil` -> `return [:]`, which is what makes the sentinel choice load-bearing rather than cosmetic. Full suite 181 executed / 3 skipped / 0 failures; live OpenAI E2E 7/7. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The Swift CI job runs `swift format lint --strict`, where warnings are errors. Two spots in the new tests tripped it: a dictionary literal over the line limit and an XCTAssertEqual whose first argument was not on its own line. Reproducing this locally needs care - a Windows checkout carries CRLF, and the trailing CR is itself counted as trailing whitespace, so a local run reports failures in untouched files that CI never sees. Both remaining local hits are that artifact; only these two were real. No behavioural change. Full suite 181 executed / 3 skipped / 0 failures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ections The coordinator ratified the save-form gate design in 393d274 and fixed the canonical semantics: where the ordered-array fallback is mandatory the serialized form must be asserted directly, name-payload association must be checked with composite entries, and neither form nor order may be asserted for collections that qualify for the object form. Reviewing the file against that ruling surfaced three gaps. The property tests read payload only from a *second* save, because a scalar Property resolves to the .unknown passthrough so the saved dictionary is the only place the payload is legible. That makes them blind to any involutive defect: a serializer that reversed entries would reverse them once, reload in that order, then reverse them back on the re-save, cancelling out and leaving the assertion green while the first serialized output violated canonical order. Both now assert the first save as well, keeping the re-save assertion for reload-side corruption that the first cannot see. The empty-name disqualifier was proven only on tools. inputs serializes through a different path and can fail independently, and this is the disqualifier least likely to be caught by accident: "" is a legal JSON object key, so an illegal object encoding round-trips cleanly and satisfies every payload assertion. Only the form check rejects it. The duplicate-name inputs comparison used a sorted multiset. Both entries are named dup, so exchanging their defaults leaves the sorted multiset unchanged and passed; the association is observable only positionally. Asserting order is legal here because the array form is required for this fixture and the array fallback is the one representation canonical promises an order for. Mutation-proved: reordering either expected array kills exactly its own test. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The optional-collection presence rule was gated at document level only for tools. inputs and outputs serialize through different paths and can fail independently, so an implementation can synthesize one while correctly omitting another. Verified non-vacuous rather than assumed: populating the fixture and inverting both assertions passes, which proves both key names are real and do appear in the saved output when present. Without that check a misspelled key would satisfy XCTAssertNil forever and gate nothing. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… alone validateBindings resolved every expected binding with actual.first(where:), which is unsound once names repeat. Two opposite defects followed: entries sharing a name and an input satisfied both expectations from the first entry, leaving the rest unverified; and a correctly ordered duplicate pair was falsely rejected, because the second expectation also matched the first entry. Object form cannot carry one key twice, and an empty key disqualifies it as well, so either one proves the source used the array fallback - the only ordered representation. Those entries are now compared positionally, by name and input at each index. Collections that qualify for object form stay name-addressed and order-agnostic, since both forms are legal for them and asserting order there would reject a conforming loader. No current vector declares a duplicate or empty binding name, so none of this is reachable through testLoadVectors; the new tests drive validateBindings directly to pin the rule before PR #447 lands vectors that rely on it. Mutation-proved: dropping the duplicate pre-scan, the empty-name disqualifier, the positional name comparison, or the map empty-key guard each kills exactly the test covering it, and forcing every collection positional kills the order-agnostic guard. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Swift implementation of the Prompty runtime. There was no Swift runtime in this repo before; nothing existing changes behaviour.
The Rust runtime is the behavioural reference throughout, and both are held to the same cross-runtime vectors in
spec/vectors.Important
Status: complete and green on the pinned
@typra/emitter@0.4.2. Merge is gated on an upstream emitter release.The runtime is finished and passes on Linux, macOS and Windows: 77 tests, 0 failures, including live provider E2E. Generated model files are never hand-edited; a scripted post-generation repair shim (
schema/scripts/patch-swift-emitter-defects.mjs) stands in for unfixed emitter defects and must be deleted before this is considered done.Six published emitter releases were probed and rejected, two of them prepared as commits and then deliberately reverted.
0.4.10is the closest yet — 23 of the 24 shim patch sites are fixed upstream — but one residual and one new defect remain, and its effect on C#/TypeScript is unmeasured.0.4.9fixes the last Swift defect (Connection.unknown) and the C# break, but a new named-collection feature emits struct-style accessors against polymorphic enums — 48 fresh compile errors.0.4.8leaves just one Swift defect (Connection.unknown) but still breaks C#.0.4.6fixes six Swift defects and regresses C# and TypeScript.0.4.5was announced upstream as carrying the generated-test fixes, but measured worse than0.4.9on Swift.0.4.3fixes four and was committed, until CI showed the same C# break — a regression the local run had masked. A shared emitter pin cannot be bumped for one language, so that belongs in its own repo-wide PR. See Validated against each upstream emitter release.Remaining merge acceptance: consume an emitter fixing the Swift defects without the regressions carried by 0.4.3 through 0.4.9 → delete the shim and its call site → restore Swift
test-dirinschema/tspconfig.yaml(still unusable: 1184 compile errors at 0.4.3, and 180 at both 0.4.8 and 0.4.9, across four test-generator defect classes) → regenerate cleanly → run the generated model tests, the 77 runtime tests, and the live E2E → push the final reviewed commit. See Upstream emitter defects.What's here
Two SwiftPM packages, and the split is the point:
prompty-modelPromptyModelpromptyPromptypromptyPromptyOpenAI135 generated model files, 26 hand-written sources, 14 test files.
The runtime defines no domain types of its own.
Prompty,Model,Message,ContentPart,Tooland the four pipeline protocols (Renderer,Parser,Executor,Processor) all come from the generated model, and the hand-written code conforms to them. Partway through this work I found that the generated model already emits the pipeline protocols, so I deleted the hand-written declarations I had started with and adopted the generated ones — a competing type layer is exactly what this port is meant to avoid.Covered: loading (including
${env:}/${file:}), rendering, parsing, preparation, protocols and registries, provider execution and processing, tools and turns, structured output, streaming, and harness durability/replay.Commits
99ab256b86eb126568f04e2ee5e4c6b7a64ed07156f638d83d60d232Propertysubtypes05f7040997c3a7a7PROMPTY_SKIP_SWIFT_SHIMemitter-validation escape hatchd13109af2a724c0366b60ec3@typra/emitter0.4.3, minimize and harden the Swift shim0a5a5e9915f7e620nameclassification in the shim header (retirement criterion)fa15d77e@typra/emitter0.4.8 probe: Swift near-clean, C# still blocking66de5c52@typra/emitter0.4.9 probe: both blockers fixed, new enum defect406fc882@typra/emitter0.4.5 probe: announced as fixed, measured worse75a38073dde5c617@typra/emitter0.4.10 probe: 23 of 24 sites fixed upstream, one residualdfb3aee6test-direrror counts: raw lines vs primary unique383058c2Connectionbase-field gap with measured load/save evidenceEach of the first four was verified to build and test standalone, by moving the not-yet-committed files aside — so no commit depends on a later one.
Continuous integration
Every other runtime has a
prompty-<lang>-check.yml; Swift had none, so nothing in CI compiled or ran this code..github/workflows/prompty-swift-check.ymlbuilds, tests and format-checks onubuntu-latestandmacos-latest.Three decisions worth flagging:
spec/**is in the trigger paths. The Swift tests are validated against the shared cross-runtime vectors, so a spec change has to re-run them. The other runtimes' check workflows watch only their own directory and have the same blind spot — fixing those is out of scope here.swift --version, not just the manifests, so a runner-image or toolchain change can never restore artifacts built by a different compiler.Package.resolvedis untracked — a library must not pin its consumers — so it can't contribute to the key.windows-latestis deliberately absent.swift-actions/setup-swiftmaps Windows releases only through 5.6.3, and theswift formatsubcommand needs 6.x. Windows is covered by local development, and the line-ending tests build their inputs from explicit\r\nliterals so they're platform-independent anyway.swift testis hermetic on CI because the live tests skip themselves without credentials.The formatting commit is not cosmetic drive-by work: the 135 generated
PromptyModelfiles already passswift-format lint --strictwith zero findings, which establishes swift-format defaults as the house Swift style here. The hand-written runtime didn't conform, so CI would have failed on its first run. Only the hand-written code is linted — the generated sources are the emitter's output, and an upstream formatting change must not break CI. Formatting was authored under Swift 6.3.3 and the lint gate runs under 6.0.3, so the first green run also confirms the two formatter versions agree.Evidence
CI, this branch — run 30904239140, both jobs green:
ubuntu-latestmacos-latestThe 7 skips are the live OpenAI tests declining to run without
OPENAI_API_KEY— the intended hermetic CI shape.Locally, Swift 6.3.3 on
x86_64-unknown-windows-msvc, where the credentials are present:Between them that's the runtime building and passing on Linux, macOS and Windows, and across two different Swift versions.
Live tests are real. Seven tests hit the actual OpenAI API — chat, token caps, streaming, streamed accumulation, structured output, tool calling, and a
.promptyfile with${env:}references — and they pass. They were run repeatedly with fresh random values to confirm they aren't flaky.I deliberately hardened them after review, because the first versions could pass for the wrong reasons:
A Swift-specific bug worth calling out
Swift treats a CRLF pair as a single
Character. For"a\r\nb", bothcomponents(separatedBy: "\n")andsplit(separator: "\n")return the whole string as one element, andtext.contains("\r")isfalse.So a prompt assembled in memory with Windows endings missed every role marker and collapsed into one lumped message; a CRLF journal replayed as zero records; the buffered SSE path yielded one unparseable line. Rust splits at the byte level and degrades gracefully, so this hazard has no counterpart in the reference and no LF-only spec vector can catch it.
Fixed with
Lines, a scalar-based splitter. ScanningunicodeScalarsrather thanCharacters is what makes it correct: normalising first and splitting on characters afterwards still fails, because rewriting"a\r\r\nb"yields"a\r\nb"whose CR and LF are now adjacent and cluster into one grapheme again.Character.isNewlineis deliberately not used — it also matches U+0085, U+2028 and U+2029, which are legal unescaped inside JSON strings, so splitting on them tore journal records in half and discarded them silently.A later pass found one more instance of the same hazard, and it was the worst of them.
Frontmatter.splitlocated its delimiters withfirstIndex(of: "\n"), which matches nothing in a CRLF document — so the opening---looked unterminated and the function returned empty frontmatter and an empty body, with no error at all.Loaderhappened to normalise before calling in, so files read from disk were unaffected and every existing test passed; butFrontmatter.splitis public API, and any direct caller silently lost the entire document. It is now line-based on the same scalar scanner. The regression tests cover it directly, and also add the two cases the suite had never exercised: a.promptyfile whose bytes on disk contain CRLF, and CRLF-delimited SSE.SSE.payloadwas trimming withCharacterSet.whitespaces, which is space and tab only, so a surviving CR defeated the[DONE]sentinel comparison.Fixing it surfaced two subtler faults that review caught before the commit landed. Once
splitnormalises,Loaderdoing so as well is not merely redundant but lossy: the first pass turnsa\r\r\nbintoa\r\nb, and the second reads that residual CR + LF as a terminator and deletes a lone CR that bothLinesand the Rust reference preserve.Loaderno longer normalises, leaving exactly one pass. And skipping leading blank lines withCharacterSet.whitespacesis wrong for the same reasonCharacter.isNewlineis wrong elsewhere — the set is horizontal-only, so a document opening with a vertical tab, form feed, NEL, or a Unicode separator would have been demoted to body-only, a genuine LF-input regression against the replaced algorithm. It now usesallSatisfy(isWhitespace).Because that rewrite sits under every vector test's load path, equivalence with the replaced algorithm was not argued but measured: a temporary differential harness reimplemented the original character-index algorithm and asserted the two agree across a 54-sample corpus, including four documents that must throw. A negative control — reverting the whitespace predicate — made it fail on exactly the vertical-tab and NEL samples, confirming the harness discriminates rather than passing vacuously.
Every fix in this area was mutation-verified: reverted deliberately, and confirmed to produce the exact predicted failure.
Upstream emitter defects
Twelve defects in the Typra Swift emitter were characterised and reported to the schema owner across five change requests. The highest-value one drops
extendsbase fields from threePropertysubtypes, all fiveToolsubtypes and theConnectionsubtypes — silent data corruption. Four of the twelve are fixed in0.4.3, that one in0.4.6, and the last Swift source defect in0.4.9— but every release from 0.4.3 on is rejected, the earlier ones for regressions in other runtimes and0.4.9for a new Swift defect of its own; see Validated against each upstream emitter release.schema/scripts/patch-swift-emitter-defects.mjsruns as part of generation and repairs the output. It is a scripted post-generation step, so generated files are still never hand-edited. It is pinned to the emitter version it was written against and fails loudly — rather than silently mis-patching — on an unrecognised version, a missing anchor, or a half-patched file.GeneratedModelRoundTripTestscovers every field it injects, so a silently regressed patch fails the suite rather than the runtime. It's designed to shrink to nothing and be deleted as upstream fixes land.Validated against each upstream emitter release
Setting
PROMPTY_SKIP_SWIFT_SHIM=1skips the patch so native emitter output can be evaluated. Every candidate release is regenerated and built this way, and the results reported upstream with line-level citations.The pin remains
0.4.2. Every later release was probed by generating unpatched Swift under it and diffing — so every verdict below is a per-file determination, not an inference from release notes — and all were rejected. Through 0.4.8 the reason was that the emitter is shared, so a bump regenerates every runtime and Swift gains arrived with C#/TypeScript regressions. At 0.4.9 those regressions are gone and the reason is Swift's own: a new defect that makes the shim larger, not smaller.Tool.customTool(CustomTool)wildcard fallback, with load/saveindirectonly on genuinely recursive polymorphic enumsrole: .assistant,.textPart(TextPart(…)))unknown?→Any?,Record<unknown>?→[String: Any]?,Message[]→[Message]extendsfields onProperty/Tool/ConnectionsubtypesConnectionfallback.unknown([String: Any])loadstill throwsbindingsloading (array-only today)*Yaml*generated tests= []on fields with an explicit= #[]0.4.3 is a real Swift gain — it would cut the shim from 419 to 345 lines and its generated Swift is byte-identical to what the current shim produces. It was adopted in
66b60ec3and reverted in0a5a5e99once CI showed the C# break. Swift gains cannot be bought with a C# regression, and a shared pin cannot be regenerated for one language.At 0.4.8 the Swift source generator is close to clean. Compiling native 0.4.8 output yields exactly 30 errors, every one
type 'Connection' has no member 'unknown', every one intool.swift. Probe-patching only that single enum case into the native output makes the model package compile — so a release addingConnection.unknownwould likely collapse this shim to nothing. Likely, not certainly: compiling is not the same as being correct. Dropped fields produce zero diagnostics, so each patch still has to be retired againstswift build --build-testsandswift test, never a clean build alone.0.4.8 also fixes the inherited-
extendsdefect includingname, which matters:nameis notextends-inherited (model Propertyandmodel Toolnever declare it; it arrives via theNamed<…>spread incore/core.tsp), so a fix scoped toextendsalone would have restored every other base field and silently dropped that one.0.4.9 clears both blockers that were reported upstream, and is still rejected.
Connection.unknownis emitted in exactly the requested shape, and the C# literal asymmetry is gone —Prompty.Core.Testsreports 48 passed, 0 failed, which beats even the pinned 0.4.2 baseline (it fails 16*Json*tests locally to a CRLF artifact). Go improves to 17FAILlines from 44; Rust stays green.What blocks it is new. 0.4.9 adds named-dict collections — the feature Prompty needs so
inputs:andtools:can be read as name-keyed maps. The generated helper assumes its element type is a struct, so for the polymorphic enumsPropertyandToolit emitsitem.name = nameandProperty.shorthandProperty, neither of which those enums declare (only the concreteArrayProperty/FunctionTool/ … structs do). That is 48 fresh compile errors —agent/prompty.swift30,core/property.swift10,tools/tool.swift8 — where 0.4.8 had 30. The shim would have to grow, not shrink. Routing the name through the dictionary beforeload, mirroring the save path that already callsremoveValue(forKey: "name"), would make the helper agnostic to struct-vs-enum elements; that has been reported upstream.The generated
test-diris no better either. After probe-patching those 48 source errors so the library would compile and the test target would actually build, it produced the same 180 errors as 0.4.8 — which is the whole reason the library must be made to compile first: if it fails, the test target is never built and the error count looks deceptively small.0.4.5is the one release probed out of order, and it is a cautionary case. It was announced upstream as carrying fixes for the four test-generator defect classes, so it was probed even though it is numerically behind the already-rejected 0.4.9. It is not a candidate: published before both 0.4.6 and 0.4.9, it predates the inherited-extendsfix and still omitsConnection.unknown, so native output is 30 errors intool.swift. Probe-patching only that case makes the library compile and then yields 228 test-build error lines, 57 unique, across six files — against 0.4.9's 180 and 45 across five. The extra file istools/ToolTests.swift, whose eight errors are allFunctionToolmissingnameanddescription;ReferenceConnectiondropsauthenticationModeandusageDescriptiontoo, but silently, which is precisely why restoringtest-dirmatters. The four columns marked ✅ above are inherited from 0.4.3 and corroborated by the library compiling onceConnectionalone was patched. Rejected on Swift, so the other runtimes were not measured.Because 0.4.5 is the first probed release to clear a majority of this shim, it was also classified patch by patch — mechanically, against native output, using the shim's own anchors. 13 of the 24 patch sites are fixed upstream and 11 are residual: the three
Connection.unknownpatches and all eight base-field injections. Applying only those 11 compiles the library, passes all 76 runtime tests including live E2E, and clearsToolTestsentirely — the first direct evidence that the base-field injections are load-bearing rather than defensive, since defect 10 otherwise fails silently. 49 generated-test errors remain across five files, sotest-dirstays out and the shim cannot be retired. Two measurement traps found here are now documented next to the code they affect: theparser.swiftpatch false-negatives against a fixed emitter because its replacement text is pre-wrapped to the formatter's width, andbaseFieldInjectionsdeliberately excludesConnectionsubtypes even though the generated tests assert them.0.4.10is the closest any release has come to making this shim deletable, and it is still rejected. Published after 0.4.9 and nowlatest, it fixes 23 of the 24 patch sites — every base-field injection, and the inherited fields land in the memberwise initializer too, which would settle the construction limitation recorded at the top of the shim.Connectiongains itsunknowncase andsavearm. The single residual isConnection.load'sdefault:, which still throwsunknownDiscriminatorinstead of returning.unknown, leaving that case declared but unreachable by the loader.The 0.4.9 named-collection defect persists, though it is much reduced:
item.name = nameandProperty.shorthandPropertyare still emitted against the polymorphic enums, now costing 10 errors confined toagent/prompty.swiftwhere 0.4.9 cost 48 across three files. Hand-synthesising the enum-level forwarders plus that oneConnectionline takesswift buildto exit 0. That measures compilation only — the forwarders are themselves a workaround, and the generatedtest-diris still unusable at 45 errors across five files — so deleting the shim needs both fixes upstream and a green generated-test run. C# and TypeScript were not re-measured, and every Swift-clean release so far has failed on exactly that gate.One contract wrinkle worth recording:
Toolnow emits bothcustomTool(CustomTool)andunknown([String: Any]), butTool.load'sdefault:still routes the wildcard to.customTool, so Prompty's*semantics survive and unknown tool kinds still arrive typed..unknownis therefore unreachable throughloadwhile still forcing any exhaustive consumerswitchwithout a catch-all to grow an arm — it broke this runtime's ownboundParameterNames. Both fixes and that observation are specified upstream.Compiling is not the bar for a type layer whose failure mode is silent, so the 0.4.10 probe was extended to load/save round-trips: a dropped field emits no diagnostic at all. Over three
Propertykinds, fiveToolkinds including the wildcard,ReferenceConnection, and theConnectionunknown fallback, 0.4.10-generated output scores 42/42 where this pinned configuration scores 40/42. The two-check delta is not a regression — it is theConnectionscope gap the shim has always declined to patch, and the probe covered onlyReferenceConnectionof the six connection subtypes, so the 0.4.10 result is scoped to that type.That gap was documented but never measured, which left nothing separating "deliberately unpatched" from "quietly broken". It is now pinned by a test walking all six
Connectionsubtypes: every one silently losesauthenticationModeandusageDescriptionbetween load and save. The scoping decision stands — injecting them restores data the runtime never reads and no vector covers, at the price of two more patches to delete on adoption — but it is recorded as an accepted limitation of the published model package rather than a free win. The test fails if either field starts surviving, which is the signal to re-audit and assert preservation instead. One further difference must be settled before adopting 0.4.10, not after: this shim writesnameonly when non-empty while 0.4.10 writes it unconditionally, andPrompty.savemapsinputs/outputs/toolsstraight through without strippingname, so an unnamed composite can reach vector output as"name": "".The 0.4.10 probe also invalidated the mechanical classifier used at 0.4.5: keyed on the shim's own anchors, it reported all eight base-field sites as still residual when the generated source proves them fixed, because the structural anchors had moved. Every verdict above was re-confirmed by reading generated source. That is now the documented rule, and the second known false-negative mechanism after the pre-wrapped
replacetext.Adoption of 0.4.8 is blocked by C#, and the cause is now pinned down:
schema/model/agent/agent.tsp:166genuinely ends with a trailing space; the emitter writes 56 escaped expected-value literals that preserve it and 24 verbatim input-YAML literals that lose it. At 0.4.2 both forms dropped it consistently and the suite passed. The escaped form is arguably the faithful one — the defect is the asymmetry, not the escaping.Three defects therefore still need shimming at the pinned 0.4.2:
Connection.unknownis never declared, yet the emitter writes.unknown([:])as a stored-property default at six sites in its owntool.swift. The generated model package does not compile standalone.Propertyalready emits exactly the right shape, so the fix is one enum case, asavearm and aloadfallback. Rust tolerates unknown kinds via_ => ConnectionKind::default(), so accepting them is parity-correct.extendsfields are dropped from derived structs, so threePropertysubtypes and all fiveToolsubtypes silently lose their base fields on both load and save.bindingsonly loads array form. The shared spec (spec/fixtures/tools_function.prompty) and the Rust reference (load_bindings) both accept map form, keyed by parameter name.Worth recording: those defect classes fail very differently. Missing
Connection.unknownis a hard compile failure. Missing inheritedPropertyfields produce zero compile errors, because the runtime reads those through the raw dictionary — the data simply disappears. That asymmetry is whyInheritedPropertyFieldTestsexists and why the shim's retirement criterion isswift build --build-testsandswift test, never a clean build alone.Cross-runtime effect of a bump
Every suite was measured against HEAD rather than assumed:
default:clauses removed, so unknown discriminators fall through to the map load instead of returning an empty value*Yaml*tests breakThe C# result is why compare-the-counts is not good enough, and this PR got it wrong once before CI caught it. On this Windows checkout,
PromptyConversionTestsalready fails 16 tests at HEAD — but asLoadJsonInput*/RoundtripJson*, a CRLF artifact of the local checkout. Under 0.4.3 it also fails 16 — but asLoadYamlInput*/RoundtripYaml*. Identical totals, completely disjoint identities. A count comparison reports "unchanged"; the truth is that a pre-existing local failure masked a real regression. The 0.4.3 failures are content-based ("some \npersonal"vs"some\npersonal"), so they reproduce on Linux CI. Both releases and this trap are now documented next toPINNED_EMITTER_VERSION.The shim, the omitted
test-dir, and this section all come out once an emitter carries the Swift fixes without the C#/TypeScript regressions.Scope
This port is not parity-complete, and the README says so explicitly rather than implying coverage that isn't there. Six of the ten shared vector files are exercised — load (25), render (23), parse (15), wire (22 of 27), process (17 of 21) and harness replay (5). The nine skipped wire and process cases are Anthropic ones: this package ships the OpenAI provider only, so both suites filter on
input.providerand an Anthropic package would pick them up unchanged. Four files are not run at all:engine/turn_vectors.jsonagent/agent_vectors.jsondiscovery/discovery_vectors.jsondiscovery/enrichment_vectors.jsonThese are a deliberate scoping decision for an initial port. The engine row is the nuanced one, and worth stating precisely rather than overclaiming the gap:
ReferenceTurnRunneralready implements the iteration loop, permission mediation, host tool execution and checkpointing, so three of the five engine vectors (final_output,ordered_tool_round,permission_denial_is_model_visible) describe behaviour that exists but is not yet asserted against the shared file. Onlydelegated_provider_stateandcancel_before_contextneed capabilities this port lacks. Tool calling is covered at the wire and processing layers — request serialization and response extraction. Wiring the engine vectors and closing those two capabilities is follow-up work tracked separately from this PR.Notes
.envis gitignored and was never committed — verified against the whole branch history. Outsideruntime/swift, this branch touches.gitignore, the new workflow,schema/tspconfig.yaml, the generation scripts, and the Swift emitter shim. The emitter pin stays at0.4.2and no other runtime's generated output is modified — see Cross-runtime effect of a bump.