Complete C# Prompty runtime parity - #442
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep canonical Anthropic content arrays and replay raw assistant blocks so tool-result turns retain correlation and signed thinking metadata. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use generated model discovery contracts across providers, embed shared fill-only capability data, preserve raw provider payloads, map Foundry shapes, and paginate Anthropic models. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
| { | ||
| } |
| { | ||
| } |
| { | ||
| } |
| { | ||
| null => string.Empty, | ||
| string text => text, | ||
| var value => JsonSerializer.Serialize(value), |
| { | ||
| null => string.Empty, | ||
| string text => text, | ||
| var value => SerializeCanonicalJson(value), |
| foreach (var childKey in dictionary.Keys.Cast<object>().ToList()) | ||
| { | ||
| if (childKey is string childName) | ||
| { | ||
| dictionary[childKey] = ResolveValue(dictionary[childKey], childName, parentDir, allowedRoots); | ||
| } | ||
| } |
| catch (Exception error) | ||
| { | ||
| failures.InvokerError = error; | ||
| throw new PortError(error.Message); | ||
| } |
| catch (Exception error) | ||
| { | ||
| failures.InvokerError = error; | ||
| throw new PortError(error.Message); | ||
| } |
| catch (Exception error) | ||
| { | ||
| throw PortError.Configuration(error.Message); | ||
| } |
There was a problem hiding this comment.
Pull request overview
This PR brings the C# Prompty runtime to feature and behavioral parity with the Rust reference and shared spec vectors, including wire formats, model discovery enrichment, and a new canonical durable turn engine that the public TurnAsync API routes through.
Changes:
- Extend OpenAI wire-format support (audio/file content parts, additional options passthrough) and expand spec-vector coverage.
- Introduce the canonical C# turn engine surface area (request/ports/exceptions/live pipeline adapter) and route public turn execution through it.
- Add provider-neutral model discovery mapping + enrichment backed by an embedded capability dataset, with vector-driven tests.
Show a summary per file
| File | Description |
|---|---|
| runtime/csharp/Prompty.OpenAI/WireFormat.cs | Adds audio/file content wiring and ModelOptions passthrough via SDK Patch. |
| runtime/csharp/Prompty.OpenAI/Prompty.OpenAI.csproj | Suppresses an additional analyzer warning during build. |
| runtime/csharp/Prompty.OpenAI/Models.cs | Implements generated IModelLister and maps/enriches raw model payloads. |
| runtime/csharp/Prompty.OpenAI.Tests/WireFormatTests.cs | Adds unit tests for AdditionalProperties passthrough behavior and safety checks. |
| runtime/csharp/Prompty.OpenAI.Tests/SpecVectorWireTests.cs | Expands wire-vector assertions to include passthrough options and removes previous skips. |
| runtime/csharp/Prompty.OpenAI.Tests/SpecVectorDiscoveryTests.cs | Adds vector-driven tests for discovery mapping + capability enrichment and embedded dataset drift. |
| runtime/csharp/Prompty.OpenAI.Tests/SpecVectorAgentTests.cs | Aligns agent vector assertions with canonical sequential tool-call constraints. |
| runtime/csharp/Prompty.Foundry/Models.cs | Adds IModelLister and improves Foundry deployment/catalog mapping + enrichment. |
| runtime/csharp/Prompty.Core/TurnRunner.cs | Records a tool-result event when permissions deny tool execution. |
| runtime/csharp/Prompty.Core/TurnEngineRequest.cs | Introduces the canonical durable turn-engine request contract and resume factories. |
| runtime/csharp/Prompty.Core/TurnEnginePorts.cs | Adds runtime-local engine ports and default/no-op implementations. |
| runtime/csharp/Prompty.Core/TurnEngineModelExtensions.cs | Adds runtime helpers on generated types needed by the engine (tool result messages, canonical JSON). |
| runtime/csharp/Prompty.Core/TurnEngineExceptions.cs | Adds canonical engine/port exception types for durability and recovery scenarios. |
| runtime/csharp/Prompty.Core/ReferenceResolver.cs | Makes ${env:...} / ${file:...} reference resolution recursive across nested objects/arrays. |
| runtime/csharp/Prompty.Core/PromptyLoader.cs | Resolves references over the full untyped tree prior to generated model loading. |
| runtime/csharp/Prompty.Core/PromptyChatParser.cs | Tightens canonical role marker parsing and nonce-based injection protection behavior. |
| runtime/csharp/Prompty.Core/Prompty.Core.csproj | Embeds the capability dataset as a resource for model discovery enrichment. |
| runtime/csharp/Prompty.Core/Pipeline.cs | Routes TurnAsync through the canonical engine and exposes TurnWithEngineRequestAsync. |
| runtime/csharp/Prompty.Core/ModelDiscovery.cs | Adds enrichment + raw-preservation utilities backed by embedded capability data. |
| runtime/csharp/Prompty.Core/LiveTurn.cs | Adds the live pipeline adapter wiring engine ports to existing registry/render/execute/process behaviors. |
| runtime/csharp/Prompty.Core/Jinja2Renderer.cs | Adds whitespace-protection workaround to improve Jinja2.NET rendering parity with vectors. |
| runtime/csharp/Prompty.Core/FrontmatterParser.cs | Reworks frontmatter parsing to treat missing delimiters as “instructions-only” and error on unclosed delimiters. |
| runtime/csharp/Prompty.Core/Data/model_capabilities.json | Adds the embedded provider capability dataset used for discovery enrichment. |
| runtime/csharp/Prompty.Core.Tests/TurnRunnerTests.cs | Extends tests to assert tool-result recording on permission denial. |
| runtime/csharp/Prompty.Core.Tests/TurnEngineVectorTests.cs | Adds shared-vector conformance tests for the canonical engine. |
| runtime/csharp/Prompty.Core.Tests/TurnEngineTestDoubles.cs | Adds deterministic ports (clock/ids/model/tools/durability) for engine tests. |
| runtime/csharp/Prompty.Core.Tests/TurnEngineResumeTests.cs | Adds tests validating resume behavior without duplicating effects/invocations. |
| runtime/csharp/Prompty.Core.Tests/TurnEngineReconciliationTests.cs | Adds reconciliation, cancellation, sequential tool execution, and post-commit behavior tests. |
| runtime/csharp/Prompty.Core.Tests/TurnEngineFailureTests.cs | Adds tests for retry, context snapshot validation, stream best-effort, and durability recovery contracts. |
| runtime/csharp/Prompty.Core.Tests/SpecVectorTests.cs | Removes prior Jinja2 known-skip now that renderer behavior is aligned. |
| runtime/csharp/Prompty.Core.Tests/ResilienceTests.cs | Updates expectations: parallel tool calls are rejected for deterministic durability ordering. |
| runtime/csharp/Prompty.Core.Tests/ParserTests.cs | Updates marker semantics (developer/tool treated as plain content) and adds new canonical-variant tests. |
| runtime/csharp/Prompty.Core.Tests/LoaderTests.cs | Adds tests for instructions-only prompts, missing closing delimiter, and recursive env resolution. |
| runtime/csharp/Prompty.Core.Tests/LiveTurnIntegrationTests.cs | Adds integration tests for LiveTurn engine wiring, durability/recovery, streaming token projection, and cancellation. |
| runtime/csharp/Prompty.Anthropic/Models.cs | Adds Anthropic model listing + mapping to generated ModelInfo with raw preservation/enrichment. |
| runtime/csharp/Prompty.Anthropic/AnthropicExecutor.cs | Improves tool-result / raw-content round-tripping and reconstructs streamed content blocks for correlation. |
| runtime/csharp/Prompty.Anthropic.Tests/AnthropicModelDiscoveryTests.cs | Adds pagination + raw payload preservation tests for Anthropic model listing. |
| runtime/csharp/Prompty.Anthropic.Tests/AnthropicExecutorTests.cs | Expands tests for correlated tool/thinking blocks and streaming reconstruction behavior. |
Review details
- Files reviewed: 39/39 changed files
- Comments generated: 2
- Review effort level: Lite
| private static ChatMessageContentPart BuildAudioContentPart(AudioPart audio) | ||
| { | ||
| byte[] bytes; | ||
| try | ||
| { | ||
| bytes = Convert.FromBase64String(audio.Source); | ||
| } | ||
| catch (FormatException) | ||
| { | ||
| bytes = Encoding.UTF8.GetBytes(audio.Source); | ||
| } | ||
|
|
||
| var format = AudioFormat(audio.MediaType); | ||
| var part = ChatMessageContentPart.CreateInputAudioPart( | ||
| BinaryData.FromBytes(bytes), | ||
| format == "mp3" ? ChatInputAudioFormat.Mp3 : ChatInputAudioFormat.Wav); | ||
| part.Patch.Set("$.input_audio.data"u8, audio.Source); | ||
| part.Patch.Set("$.input_audio.format"u8, format); | ||
| return part; | ||
| } |
There was a problem hiding this comment.
Updated the SDK patch values to JSON-encoded strings, eliminating malformed JSON while preserving AudioPart.source verbatim as required by the shared wire spec and Rust/Python/TypeScript implementations. Added serialized JSON regressions for both data URI and URL sources. Rejecting URLs would require a shared contract/vector change rather than a C#-only divergence.
| var failures = new LiveFailureState(); | ||
| var authorization = new LivePermissionPort(options.Permission, options.Guardrails); | ||
| var durability = new LiveDurabilityPort( | ||
| options.Durability ?? new NoopDurabilityPort(), | ||
| options.OnEvent, | ||
| agent, | ||
| agentMode, | ||
| request.MaxIterations); | ||
| var engine = new TurnEngine(new TurnEngineEffects | ||
| { | ||
| Model = new LiveModelPort(agent, executor, processor, options.Raw, agentMode, failures), | ||
| Tools = new LiveToolPort( | ||
| agent, | ||
| options.Tools, | ||
| request.Inputs as Dictionary<string, object?>, | ||
| authorization, | ||
| options.OnEvent), | ||
| Clock = new LiveClock(), | ||
| Ids = new LiveIds(), | ||
| Policy = new LivePolicyPort( | ||
| agent, | ||
| request.Inputs as Dictionary<string, object?>, | ||
| options.ContextBudget, |
There was a problem hiding this comment.
Fixed in 2d26f7c. LiveTurn now normalizes dictionary, read-only dictionary, and JSON-object inputs once for both tool and policy ports, and fails fast on unsupported shapes before any provider invocation. Added a regression test asserting the provider invocation count remains zero.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Review details
Suppressed comments (1)
runtime/csharp/Prompty.OpenAI/WireFormat.cs:98
- AudioPart wire formatting looks inconsistent: when
audio.Sourceis not base64 it is treated as UTF-8 bytes, but the patch still writes the original string intoinput_audio.data. Also,Patch.Sethere uses raw strings while other patch sites write JSON-encoded UTF-8 bytes, which risks producing invalid JSON in the serialized request body.
- Files reviewed: 39/39 changed files
- Comments generated: 0 new
- Review effort level: Lite
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Review details
Suppressed comments (4)
runtime/csharp/Prompty.OpenAI/WireFormat.cs:99
- BuildAudioContentPart mixes three different representations of the audio payload: it tries to decode base64 into bytes (for CreateInputAudioPart), but then patches
$.input_audio.datawith the originalaudio.Sourcestring (which may be non-base64 when the catch path runs). Also, Patch.Set elsewhere in this file uses JSON-encoded bytes; setting a raw string here risks inconsistent serialization and vector mismatches.
runtime/csharp/Prompty.Anthropic/AnthropicExecutor.cs:363 - FormatToolMessages batches tool_result blocks by iterating
Math.Min(toolCalls.Count, toolResults.Count), which will silently drop unmatched tool calls or results if a mismatch occurs. That would break tool-call/result correlation for the next request and make debugging difficult. It’s safer to fail fast when counts differ, since the caller contract is one result per tool call.
var toolResultBlocks = new List<Dictionary<string, object?>>();
for (var i = 0; i < Math.Min(toolCalls.Count, toolResults.Count); i++)
{
toolResultBlocks.Add(new()
{
["type"] = "tool_result",
["tool_use_id"] = toolCalls[i].Id,
["content"] = toolResults[i],
});
runtime/csharp/Prompty.OpenAI/WireFormat.cs:116
- AudioFormat() returns many formats (mp4/ogg/flac/webm/pcm/…) but BuildAudioContentPart only maps the SDK enum as Mp3 vs Wav. For non-mp3/wav media types this produces an inconsistent wire payload (e.g., CreateInputAudioPart uses Wav while the patched
formatfield saysflac). Limiting the mapping to the formats the SDK actually supports avoids emitting invalid requests.
runtime/csharp/Prompty.Core/Jinja2Renderer.cs:13 - ProtectedWhitespace is a long sentinel string, and the replacement repeats it once per whitespace character. This can blow up the template size by O(n * sentinelLength) around
{% endfor %}boundaries (e.g., 10 spaces become 10 * 37 characters), increasing memory use and potentially slowing rendering significantly. Using a single-character sentinel avoids this expansion while preserving exact whitespace length.
private const string ProtectedWhitespace = "__PROMPTY_JINJA_CONTROL_WHITESPACE__";
- Files reviewed: 39/39 changed files
- Comments generated: 0 new
- Review effort level: Lite
Encode OpenAI audio patch values as JSON while preserving the shared verbatim source contract, and reject unsupported canonical turn input shapes instead of silently dropping them. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Review details
Suppressed comments (4)
runtime/csharp/Prompty.OpenAI/WireFormat.cs:87
- BuildAudioContentPart computes/decodes
bytesbut then overwritesinput_audio.datavia Patch to preserve the originalaudio.Sourcestring. This means the decode/UTF8 allocation is unused work (and can be costly for large data URIs/URLs) and the base64 decode path may throw frequently.
runtime/csharp/Prompty.Core/Jinja2Renderer.cs:13 ProtectedWhitespaceis expandedmatch.Lengthtimes; with the current long marker string this can balloon the template size significantly for indented control blocks. Using a single-character sentinel avoids that intermediate string blow-up while keeping the same behavior.
private const string ProtectedWhitespace = "__PROMPTY_JINJA_CONTROL_WHITESPACE__";
runtime/csharp/Prompty.Core/ReferenceResolver.cs:62
ResolveValuepasses only the immediate key name into recursive calls (e.g.childName), so error messages for missing env vars / disallowed files can become ambiguous once references are nested. Carrying a dotted path (e.g.metadata.nested.values[0]) makes troubleshooting much easier.
if (childKey is string childName)
{
dictionary[childKey] = ResolveValue(dictionary[childKey], childName, parentDir, allowedRoots);
}
}
runtime/csharp/Prompty.Core/LiveTurn.cs:160
JsonElement.Deserialize<Dictionary<string, object?>>()is nullable; returning null here will later behave like "no inputs" even though the caller supplied a JSON object. Consider normalizing null to an empty dictionary so engine ports get a consistent shape.
JsonElement { ValueKind: JsonValueKind.Object } value =>
value.Deserialize<Dictionary<string, object?>>(),
- Files reviewed: 39/39 changed files
- Comments generated: 0 new
- Review effort level: Lite
Align handwritten C# Record<unknown> consumers with explicit-null schema semantics and add regressions across loading, providers, turns, durability, replay, and discovery. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Review details
Suppressed comments (1)
runtime/csharp/Prompty.OpenAI/WireFormat.cs:100
- BuildAudioContentPart eagerly tries to base64-decode AudioPart.Source (and falls back to UTF-8 bytes), but the serialized request body ultimately uses Patch.Set to emit the original source string verbatim as
input_audio.data. This means large base64 inputs will be decoded into a potentially large byte[] that never affects the wire payload, adding avoidable CPU/memory overhead.
- Files reviewed: 56/56 changed files
- Comments generated: 0 new
- Review effort level: Lite
Align remaining live-turn, engine, discovery, and test metadata consumers with explicit-null Record<unknown> values exposed by the accepted Typra candidate gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Review details
Suppressed comments (2)
runtime/csharp/Prompty.OpenAI/WireFormat.cs:87
BuildAudioContentPartattempts to base64-decodeaudio.Source(and falls back to UTF-8 bytes) but then patches$.input_audio.datato the originalaudio.Sourcestring. That means the decode work and extra byte-array allocation do not affect the serialized request (and can be significant for large data URIs). Consider using a minimal placeholder payload for the SDK object and relying on the patch for the actual wire value.
runtime/csharp/Prompty.Core/PromptyChatParser.cs:129PreRender()stores the marker nonce in anAsyncLocal, andCreateMessage()enforces it, but the nonce is never cleared after parsing completes. That can cause unrelated subsequent parses in the same async flow to fail with a nonce-mismatch even whenPreRender()was not used (and keeps per-call state alive longer than needed). Clearing the nonce afterParse()finishes makes the strict-mode guard self-contained per call.
// Flush last message
if (currentRole is not null)
{
messages.Add(CreateMessage(currentRole, currentContent, currentAttrs, validateNonce: true));
}
- Files reviewed: 56/56 changed files
- Comments generated: 0 new
- Review effort level: Lite
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Review details
Suppressed comments (2)
runtime/csharp/Prompty.OpenAI/WireFormat.cs:116
AudioFormat()can return values like "mp4", "ogg", "flac", etc., butCreateInputAudioPart(...)only selects betweenChatInputAudioFormat.Mp3andChatInputAudioFormat.Wav. The code then patches$.input_audio.formatwith the (potentially non-wav/mp3) string, which makes the serialized payload internally inconsistent and likely invalid for the OpenAI API/SDK.
Consider restricting the normalized format to only the SDK-supported values ("wav"/"mp3") so the patched JSON and the strongly-typed SDK format cannot disagree.
runtime/csharp/Prompty.OpenAI/WireFormat.cs:226
AdditionalPropertiesoption names are validated to contain only ASCII letters/digits/underscore, but names starting with a digit (e.g. "1") would still pass. Those cannot be addressed safely via the JSON path$.{name}and may patch the wrong location or throw in the SDK.
Tighten validation to require a JSON-property-like identifier (leading letter or underscore).
- Files reviewed: 56/56 changed files
- Comments generated: 0 new
- Review effort level: Lite
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
| catch (Exception error) | ||
| { | ||
| AgentEvents.EmitEvent(onEvent, AgentEventType.Error, new Dictionary<string, object?> | ||
| { | ||
| ["tool"] = request.Name, | ||
| ["error"] = error.Message, | ||
| }); | ||
| output = $"Error: Tool '{request.Name}' failed: {error.Message}"; | ||
| } |
There was a problem hiding this comment.
Review details
Suppressed comments (2)
runtime/csharp/Prompty.OpenAI/WireFormat.cs:98
AudioFormat()supports additionalaudio/*media types (e.g., flac/ogg/webm/strip-prefix), but the current test suite only asserts wav and mp3. Adding a couple of unit/spec-vector assertions for non-wav/mp3 formats would help ensure the spec’s broader mapping table stays correct and doesn’t regress silently.
runtime/csharp/Prompty.Core/Jinja2Renderer.cs:32- Replacing each matched whitespace character with the full
ProtectedWhitespacetoken multiplies the template size (and allocations) by ~O(token_length * whitespace_length). In pathological cases (large whitespace runs), this can materially increase memory/time spent before rendering. Consider encoding the run length once per match and expanding back to spaces after rendering, so the protected template growth stays O(number_of_matches).
var protectedTemplate = LoopBoundaryWhitespaceRegex().Replace(
template,
match => string.Concat(Enumerable.Repeat(ProtectedWhitespace, match.Length)));
var jinja = new Jinja2.NET.Template(protectedTemplate);
- Files reviewed: 56/56 changed files
- Comments generated: 0 new
- Review effort level: Lite
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Review details
Suppressed comments (1)
runtime/csharp/Prompty.OpenAI/WireFormat.cs:87
- BuildAudioContentPart decodes
audio.Sourceto bytes, but the serialized request is forced viapart.Patchto preserveaudio.Sourceverbatim (per spec). This means the base64/UTF8 decode work is unused and can be expensive for large sources (e.g., data URIs / URLs). Consider avoiding the decode and passing an empty payload toCreateInputAudioPart, relying on the patch fields for the actual wire representation.
- Files reviewed: 56/56 changed files
- Comments generated: 1
- Review effort level: Lite
| foreach (var (name, value) in opts.AdditionalProperties) | ||
| { | ||
| if (CanonicalOptionIsSet(name, agent, opts)) | ||
| continue; | ||
| if (name.Any(character => !char.IsAsciiLetterOrDigit(character) && character != '_')) | ||
| throw new ArgumentException( | ||
| $"OpenAI additional option name '{name}' cannot be represented safely by the SDK.", | ||
| nameof(agent)); | ||
|
|
||
| var path = Encoding.UTF8.GetBytes($"$.{name}"); | ||
| var json = BinaryData.FromObjectAsJson(value).ToMemory().Span; | ||
| options.Patch.Set(path, json); | ||
| } |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Summary
Validation
Typra follow-up
No files under
runtime/csharp/Prompty.Core/Modelwere edited. Exact consumer regeneration was revalidated with@typra/emitter0.4.6: schema generation succeeded, but the complete pre-checkpoint Core baseline still reported the same 16 generatedPromptyConversionTestsYAML multiline trailing-whitespace failures (1,110 passed / 1,126 total). The failed regeneration was not committed and the repository remains pinned to 0.4.2. A clean solution build also continues to expose generated nullable annotations without an explicit nullable context (CS8669); both defects were reported upstream with exact generated examples.Cancellation limitation
Cancellation is cooperative at canonical engine boundaries and while draining streams. Generated provider/tool protocols do not yet expose language-native cancellation, so an in-flight non-streaming provider or tool call may finish before cancellation is observed. Durable writes are intentionally non-cancellable once persistence begins.
Warning classification
A clean rebuild classified all smaller warnings as handwritten and this PR clears them: 7 CS8602 test dereferences, 6 CS8620 payload projections, and 2 CS0114 private exception members. No generated file emitted those codes. Remaining build warnings are exclusively the separately routed Typra CS8669 nullable-context defect.
Canonical engine port dependency
A separate TypeSpec/emitter follow-up will generate
IEnginePermissionPort,IEngineToolPort,IEngineDurabilityPort, andIEnginePostCommitPortwith the canonical signatures and native cancellation semantics. After that schema/emitter PR lands, this branch must remove only those four handwritten interface declarations fromTurnEnginePorts.csand consume the generated interfaces.PortErrorand the remaining richer runtime-local ports stay native. This dependency is documentation-only for now: no generated files or handwritten port declarations are changed in this PR until the upstream schema/emitter work lands.