Redesign experimental Engine turn API

[bmehta001]

Summary

• redesign the experimental Engine API around Engine -> Request -> Turn, with Requests created and owned by an Engine
• replace polling and unseen-token state with typed caller-buffered events from one canonical bulk OgaEngineRun
• add request-local Turn IDs, named cancellation, per-Turn generation limits, usage and finish metadata, and automatic reclamation when the final Request handle is released
• update the C ABI, C++ RAII wrapper, Python binding, examples, benchmarks, tests, and Engine architecture documentation

Addresses #2448.

API contract

OgaResult* OgaEngineRun(
    OgaEngine* engine,
    void* event_buffer,
    size_t event_capacity,
    size_t event_stride,
    size_t* out_event_count);

• each positive-capacity call drains retained events or executes at most one model transaction, never both
• capacity one preserves one-event-at-a-time pacing; larger buffers can receive one event per affected Request from the committed transaction
• events identify a borrowed OgaRequest* and Request-local turn_id; flags are a bitmask, so the final token may also carry TurnFinished
• OgaRequestOptions.max_session_tokens == 0 uses the Request's snapshotted OgaGeneratorParams.search.max_length
• opaque reusable OgaTurnParams provide per-Turn overrides; max_generated_tokens is implemented and unsupported setters return explicit errors
• stop sequences use token-ID sequences through OgaSequences; generation-time stop handling remains deferred
• OgaRequestClose releases Engine state while OgaDestroyRequest releases the caller-owned handle
• releasing the final public Request handle marks it abandoned; the Engine reclaims scheduler membership, retained events, and dynamic cache state at the next owner-thread boundary
• Engine, Request, and Turn operations use one host owner thread; model execution may still batch Requests in parallel

Runtime behavior

• dynamic batching supports resident continuation and prompt chunking
• static batching preserves logical close immediately, but a closed resident row remains physically allocated until the shared batch is recycled
• capacity, retryable, request-terminal, contract, and fatal failures are represented by typed events rather than diagnostic-string parsing
• Engine-level events have no Request; examples and benchmarks classify them before request lookup and use bounded retry handling where appropriate
• OgaCreateEngine retains the Model, so the caller may release its Model handle after successful Engine creation

Breaking changes

This intentionally replaces the previous experimental Engine API. The old create/add/continue/step/remove and unseen-token entry points, Request opaque-data accessors, and compatibility aliases are removed. Generator APIs are unchanged.

Validation

• Linux/WSL Debug native, Python, and Engine benchmark builds
• 77 focused Engine lifecycle/runtime tests
• focused Engine C API tests
• 29 Python Engine tests
• Python examples and benchmark syntax checks
• C++17 public-wrapper compile smoke
• full material review of runtime, ABI, wrapper, example, benchmark, and documentation changes

Windows-native build validation was unavailable because this environment has no Visual Studio 2022 installation.

Follow-up

• Consider adding Engine event-buffer generation tracking (for example, OgaEngineEventBufferGetGeneration) so C++ and managed wrappers can reject event/usage views retained across a later validated OgaEngineRun, instead of relying only on the documented borrowed-view lifetime.

[Copilot]

Pull request overview

This PR redesigns the experimental Engine low-level “turn” API across the native Engine implementation, C/C++ surface, Python binding, tests, examples, benchmarks, and documentation. It shifts the model to Engine-created, Engine-bound Requests with a BeginTurn / synchronous Run loop and an idempotent Close, while keeping generator APIs unchanged.

Changes:

• Replace the prior experimental create/add/continue/step/remove workflow with EngineCreateRequest, RequestBeginTurn, synchronous EngineRun, and RequestClose, including “borrowed alias” semantics for Run().
• Add per-turn generated-token budgeting (optional max_generated_tokens) and snapshot request-level generation parameters at request creation.
• Migrate/expand coverage across C API tests, C++ engine unit/contract tests, Python unit/integration tests, examples, benchmarks, and engine architecture docs to match the new contract.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
test/python/test_onnxruntime_genai_engine.py	Updates Python engine unit tests to the new create/begin_turn/run/close API and adds new contract tests (per-turn budget, snapshotting, close idempotence, opaque data).
test/python/integration/test_integration_engine.py	Updates Python integration coverage for the new Engine/Request contract and close semantics.
test/engine/scheduler_contract_tests.cpp	Adjusts scheduler contract tests to use Engine-owned request creation helpers and parameter snapshot flows.
test/engine/request_lifecycle_tests.cpp	Reworks lifecycle/state-machine tests to cover BeginTurn/Run/Close behavior, snapshotting, and transactional guarantees.
test/engine/paged_key_value_cache_tests.cpp	Updates cache tests to new request minting/begin-turn helpers.
test/engine/engine_test_helpers.h	Introduces helpers for Engine-owned request creation and prompting (CreateEngineRequest, CreateRequestWithPrompt).
test/engine/engine_test_doubles.h	Renames/updates doubles and trace comments from Step to Run semantics.
test/engine/engine_run_tests.cpp	Replaces Engine::Step contract tests with Engine::Run tests and expands failure-mode coverage.
test/c_api_tests.cpp	Adds/updates C API tests for TurnOptions, borrowed Run handle semantics, opaque data, and Close idempotence.
src/python/python.cpp	Updates Python bindings to expose Request.begin_turn, Request.close, Engine.create_request, Engine.run, and Python-owned opaque data.
src/ort_genai.h	Updates C++ RAII wrapper to match new C API (Engine creates requests; Run returns borrowed pointer; Request gains BeginTurn/Close).
src/ort_genai_c.h	Updates the C header surface: introduces OgaTurnOptions, replaces Step with Run, replaces add/remove/continue with create/begin_turn/close, and updates docstrings.
src/ort_genai_c.cpp	Implements new C ABI entry points (OgaEngineRun, OgaEngineCreateRequest, OgaRequestBeginTurn, OgaRequestClose) and borrowed-handle return plumbing.
src/engine/request.h	Replaces AddTokens/Continue/Remove with BeginTurn/Close; adds per-turn budgeting fields and opaque-data storage.
src/engine/request.cpp	Moves “turn admission” logic into Engine, adds per-turn budget enforcement, and updates close/opaque-data implementation.
src/engine/request_status.h	Updates request state semantics to “Engine-bound Unassigned” and Close behavior notes.
src/engine/paged_key_value_cache.cpp	Documentation tweak for Step → Run terminology.
src/engine/model_executor.h	Documentation tweak for Step → Run terminology.
src/engine/engine.h	Replaces AddRequest/RemoveRequest/Step with CreateRequest/Run and internal BeginTurn/CloseRequest hooks; adds Engine destructor.
src/engine/engine.cpp	Implements parameter snapshot cloning, Engine-owned request creation, transactional BeginTurn admission, CloseRequest, RunDynamic/RunStatic, and teardown close behavior.
src/engine/cache_manager.cpp	Updates static cache allocation logic to use request SearchOptions accessors.
examples/python/engine/tool-calling.py	Migrates example to create_request/begin_turn/run/close with improved runtime checks.
examples/python/engine/model-qa.py	Migrates interactive example to new API and ensures a single request is reused across turns via begin_turn.
examples/python/engine/continuous-batching.py	Refactors example to single-owner loop using create_request/begin_turn/run/close and removes prior threaded admission model.
docs/paged_attention_engine.md	Updates the Engine architecture doc to the new experimental API shape, ownership model, and single-owner-thread contract.
docs/paged_attention_batching.md	Updates batching doc references to the new Engine-owned request lifecycle contract.
benchmark/python/benchmark_engine_guidance.py	Migrates benchmark driver to create_request/begin_turn/run/close and updates error messages/flow.
benchmark/engine/scenarios/long_prefill.cpp	Migrates native benchmark scenario to Engine-created requests and Run/Close flow.
benchmark/engine/scenarios/decode_baseline.cpp	Migrates native benchmark scenario to Engine-created requests and Run/Close flow; updates token collection logic.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.