perf(sync): defer mutex wake-state allocation

[Coldwings]

Description

Make the non-token sync::mutex::lock() uncontended path allocation-free while preserving shared wake-state lifetime, FIFO ownership transfer, and popped-but-not-resumed recovery whenever a waiter actually parks.

Only a waiter that remains contended after the suspension-entry CAS directly constructs shared wake state in mutex-private manual storage, before taking the waiter-queue mutex. Token-aware locks retain eager arbitration state and the existing cancellation-versus-grant protocol.

This is an intentionally asymmetric performance tradeoff. Across two independent pinned Release comparisons, uncontended lock/unlock latency decreased by approximately 45.7% to 46.9%, while the permanently contended local-trampoline forced-handoff benchmark increased by approximately 2.7% to 3.3%. Using adverse paired-confidence-interval endpoints and approximately 26 ns ready / 60 ns handoff baselines, the conservative break-even is approximately 21% ready acquisitions. This favors workloads with a meaningful uncontended fraction; it does not claim every contention pattern becomes faster.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Performance improvement (optimization that improves speed/memory usage)
• Documentation (changes to documentation, comments, or examples)
• Refactoring (code changes that neither fix bugs nor add features)
• Tests (adding or modifying tests)
• Build/CI (changes to build system, CI configuration, or dependencies)

Related Issues

Closes #1038
Related to #1039

Changes Made

Core Changes

• Defer no-token wake-state construction until the initial ready check and suspension-entry CAS both observe contention.
• Directly construct the slow-path std::shared_ptr<wake_state> in mutex-private manual storage while preserving the existing 56-byte waiter layout.
• Preserve the locked recheck, FIFO queue publication, dequeue-to-schedule shared lifetime, ownership transfer, and popped-waiter grant recovery.
• Keep token-aware locks eagerly allocated and on the unchanged noexcept cancellation path.
• Add deterministic allocation/failure/race coverage, Release benchmarks, changelog, and wiki updates.

API Changes (if applicable)

Normal co_await mutex.lock() source and ownership semantics are unchanged. The low-level exception boundary of the non-token awaiter changes.

Before:

auto lock = mutex.lock();  // Wake-state allocation may throw here.
static_assert(noexcept(
    lock.await_suspend(std::noop_coroutine())));

After:

auto lock = mutex.lock();  // Construction is allocation-free.
static_assert(!noexcept(
    lock.await_suspend(std::noop_coroutine())));
// A genuinely contended await_suspend() may propagate std::bad_alloc.

await_ready() remains noexcept. Allocation occurs before taking internal_mutex_ or mutating the waiter queue, so std::bad_alloc leaves lock ownership and queue state unchanged. The token-aware awaiter retains eager construction and a noexcept await_suspend().

Migration Guide (if breaking change)

No migration is required for normal co_await mutex.lock() use. Low-level integrations that require the exact no-token await_suspend member to be noexcept, or call it from their own noexcept function, must allow or handle std::bad_alloc on the contended path. The token-aware overload remains non-throwing at suspension.

Testing

Unit Tests

• Added new tests for the changes
• Updated existing tests if needed
• All tests pass locally

Integration Tests

• Tested with existing examples
• Tested in real-world scenarios (if applicable)

Sanitizer Testing

• Tested with ASAN (AddressSanitizer)
• Tested with TSAN (ThreadSanitizer)
• No new warnings or errors

Test Results

Focused Normal: 388 assertions / 8 mutex test cases passed
Full Normal:    12,300 assertions / 816 test cases passed
Full ASAN:      12,299 assertions / 816 test cases passed; no diagnostics
Full TSAN:      12,294 assertions; 815 passed / 1 existing skip; no diagnostics
Post-rebase focused Normal, ASAN, TSAN: 388 assertions / 8 cases each

Deterministic tests verify zero allocations for ready locks and the ready/suspend unlock race; exactly one allocation for a parked lock; eager token arbitration; strong bad_alloc safety for non-token suspension and token construction; and existing destroy-after-dequeue and popped-handoff behavior.

Pinned Release comparisons used long-lived coroutine frames, fixed CPU affinity, and interleaved baseline/candidate order:

Uncontended lock/unlock:
  Candidate reduction across two independent 30-pair comparisons: about 45.7% to 46.9%

Forced two-task local-trampoline handoff:
  Candidate increase across those comparisons: about 2.7% to 3.3%

Final 10-pair direction check:
  Uncontended: -45.73%
  Forced handoff: +3.58%

Conservative mixed-workload break-even:
  About 21% ready acquisitions using adverse paired-CI endpoints

The forced-handoff benchmark is deliberately permanently contended and removes worker-scheduling noise through the local trampoline. It exposes the slow-path tradeoff rather than representing every production pattern. No timing threshold is enforced in CI.

Checklist

Code Quality

• My code follows the project's code style
• I have added/updated comments for complex logic
• I have removed any debug code, TODOs, or commented-out code
• My changes generate no new warnings

Documentation

• I have updated documentation
• I have added benchmark coverage
• I have updated API documentation

Testing

• I have added tests that prove the optimization is effective
• New and existing unit tests pass locally
• I have tested with ASAN and TSAN

Compatibility

• Normal co_await use remains compatible; the low-level exception-specification change is documented above
• I have considered the impact on existing users
• I have updated CHANGELOG.md

Performance (if applicable)

• I have considered ready and contended-path performance
• I have added benchmarks for performance-critical changes

Screenshots / Diagrams

Not applicable.

Additional Notes

The private manual storage does not embed wake_state in the coroutine frame. It only defers construction of the existing std::shared_ptr<wake_state>. Once a waiter parks, unlock and waiter destruction retain independent shared leases across the dequeue-to-schedule window.

Empty-shared_ptr assignment, swap, generic bit-packed storage, and allocation-before-recheck variants were benchmarked and rejected because they produced larger forced-handoff regressions.

Reviewer Guidance

Areas requiring special attention:

• Manual shared_ptr lifetime: engagement only after successful construction and exactly-once destruction.
• Strong allocation-failure safety before queue publication or ownership transfer.
• Preservation of the second CAS, FIFO handoff, popped-grant recovery, and dequeue-to-schedule shared lifetime.
• Separation of the throwing no-token suspend path from the unchanged token-aware cancellation path.
• Transparent interpretation of the ready gain, forced-handoff cost, and conservative break-even.

Questions for reviewers:

• Does every engaged manual-storage path destroy the shared_ptr exactly once?
• Can allocation failure leave mutex or waiter-queue state mutated?
• Are the tradeoff and low-level exception-specification change sufficiently clear?

[Copilot]

Pull request overview

Optimizes non-cancellable mutex acquisition by deferring wake-state allocation until contention requires parking.

Changes:

• Adds lazy wake-state construction while preserving cancellation and handoff behavior.
• Adds allocation-failure tests and mutex benchmarks.
• Documents performance and exception-boundary changes.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
include/elio/sync/mutex.hpp	Implements deferred wake-state construction.
include/elio/sync/detail/wake_state.hpp	Adds allocation-failure test hook.
tests/unit/test_sync_cancellation.cpp	Tests allocation and failure paths.
examples/microbench.cpp	Benchmarks mutex fast path and handoffs.
wiki/API-Reference.md	Documents allocation and exception behavior.
wiki/Performance-Tuning.md	Explains mutex performance tradeoffs.
CHANGELOG.md	Records the optimization.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Coldwings]

@copilot Please review the updated pull request head only. Verify that the revised performance guidance now distinguishes wake-state allocation from actual FIFO queue publication. Please review; do not make code changes.

[Copilot]

@copilot Please review the updated pull request head only. Verify that the revised performance guidance now distinguishes wake-state allocation from actual FIFO queue publication. Please review; do no...

The revised guidance at lines 440–443 of wiki/Performance-Tuning.md now separates the two outcomes correctly:

• Wake-state allocation happens when the waiter is still contended at the suspension-entry CAS.
• FIFO queue publication is a further step that only occurs if the final locked recheck (taken under the internal mutex) also fails.
• A waiter that succeeds at the locked recheck acquires without parking and releases the unused wake state.

The text no longer implies every slow-path allocation is queued. The separation is clear and accurate.

[Coldwings]

@copilot Please review the new head only. A read-only audit found that the public low-level lock_waiter(m, false/true) helper paths could double-construct the deferred shared state after the helper split. The new head preserves the old public dispatcher, gives normal awaitables direct hot-path helpers, and adds explicit true/false regression coverage. Please review for compatibility, object lifetime, and cancellation semantics; do not make code changes.

[Copilot]

@copilot Please review the new head only. A read-only audit found that the public low-level lock_waiter(m, false/true) helper paths could double-construct the deferred shared state after the helper ...

Reviewed the new head for compatibility, object lifetime, and cancellation semantics.

Double-construction risk: not present.

• lock_awaitable::await_suspend calls await_suspend_non_cancellable_impl directly, bypassing the dispatcher entirely.
• cancellable_lock_awaitable::await_suspend delegates through cancellable_lock_waiter::await_suspend_impl, which overrides the base to call await_suspend_cancellable_impl directly.
• Neither public awaitable reaches the shared lock_waiter::await_suspend_impl dispatcher, so there is no path that would route a non-cancellable waiter through the cancellable branch or vice versa.
• waiter_state::emplace() asserts !engaged before placement-new, providing a hard guard against double-construction on any path that might reach it twice.

Object lifetime: correct.

• Non-cancellable waiters start with engaged = false. emplace() sets engaged = true only after successful construction. emplace() is called exactly once, before taking the queue lock, before suspended is set.
• Cancellable waiters set engaged = true in the waiter_state constructor, eagerly. They never call emplace().
• waiter_state destructor destroys the wake_state_ptr only when engaged = true, so both paths see exactly one construction and one destruction.
• recover_cancelled_handoff_locked calls waiter_state_.wake() (which asserts engaged) only for queued waiters. Non-cancellable waiters are only queued after emplace() succeeds, so engaged is always true at that point.

Cancellation semantics: unchanged.

• Token-aware locks continue to allocate eagerly and use noexcept await_suspend through cancellable_lock_waiter.
• Non-token locks defer allocation to the contended await_suspend_non_cancellable_impl path; std::bad_alloc from emplace() escapes before any queue mutation or ownership transfer.
• The true/false regression coverage in the new tests exercises both branches of the lock_waiter(m, bool) constructor directly.