perf(runtime): sample worker progress for block detection

[Coldwings]

Description

Move autoscaler worker-block detection off the coroutine-resume hot path. The on_block trigger now samples each worker's published completed-resume counter once per autoscaler tick and measures how long a non-idle worker has made no sampled progress.

Ordinary resumes no longer call steady_clock::now() or publish an atomic time point. The concrete worker_thread::last_task_time() diagnostic remains source-compatible: its first call opts that worker into exact timestamp tracking for subsequent resumes.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Performance improvement (optimization that improves speed/memory usage)
• Documentation (changes to documentation, comments, or examples)
• Refactoring (code changes that neither fix bugs nor add features)
• Tests (adding or modifying tests)
• Build/CI (changes to build system, CI configuration, or dependencies)

Related Issues

Closes #1019
Related to #1017

Changes Made

Core Changes

• Track per-worker execution-counter snapshots and last observed progress in the autoscaler thread.
• Exclude idle/stopped workers and reset observations on progress, idle transitions, worker replacement, stop, and autoscaler restart.
• Remove unconditional exact timestamp updates from worker_thread::run_task().
• Preserve last_task_time() through an opt-in diagnostic flag.
• Add direct state-transition coverage and a real blocked-worker integration test with deterministic cleanup.
• Extend the microbenchmark with exact-timestamp and disabled-diagnostic costs.

API Changes (if applicable)

No signature changes. on_block is now explicitly documented as best-effort sampled detection:

• a worker must be non-idle;
• its completed-resume counter must make no sampled progress for longer than block_threshold;
• notification can trail the threshold by roughly one tick_interval.

Calling worker_thread::last_task_time() enables exact per-resume timestamps for that worker from then on. Periodic monitoring should prefer scheduler execution counters.

Migration Guide (if breaking change)

Not applicable. Existing call sites remain source-compatible.

Testing

Unit Tests

• Added new tests for the changes
• Updated existing tests if needed
• All focused tests pass locally

Integration Tests

• Tested with existing scheduler tests
• Tested the real autoscaler against a deliberately blocked worker

Sanitizer Testing

• Tested with ASAN (AddressSanitizer)
• Tested with TSAN (ThreadSanitizer)
• No new warnings or errors

Test Results

Release [autoscaler]: 8 test cases, 38 assertions passed
ASAN   [autoscaler]: 8 test cases, 38 assertions passed; leak detection enabled
TSAN   [autoscaler]: 8 test cases, 38 assertions passed; no race reports

Release [scheduler]: 74 test cases, 541 assertions passed
ASAN   [scheduler]: 74 test cases, 541 assertions passed; leak detection enabled
TSAN   [scheduler]: 74 test cases, 541 assertions passed; no race reports

GCC 12 Release build: developer warnings and warnings-as-errors enabled
scheduler_service_benchmark --smoke: passed

Checklist

Code Quality

• My code follows the project's code style
• I have added or updated comments for concurrency invariants
• I have removed any debug code, TODOs, or commented-out code
• My changes generate no new warnings

Documentation

• I have updated README, wiki, API reference, and code comments
• I have added examples for new features (if applicable)
• I have updated API documentation

Testing

• I have added tests that prove the optimization preserves behavior
• New and existing focused unit tests pass locally
• I have tested with ASAN and TSAN

Compatibility

• My changes are backward compatible
• I have considered the impact on existing users
• I have updated CHANGELOG.md

Performance (if applicable)

• I have considered the performance impact
• I have added benchmarks for performance-critical changes

Performance Results

Pinned to CPUs 0-4 on the same GCC 12 Release environment:

quick_benchmark yield:
  main baseline: approximately 32.1 ns/yield
  candidate:     approximately 12.2 ns/yield
  change:        approximately -62%

microbench:
  exact task timestamp publish: 17.70 ns/update
  disabled timestamp check:      0.27 ns/update

Task-spawn and general context-switch results remained within run-to-run noise. These figures are local directional evidence; CI compiles and smoke-tests benchmarks but does not enforce timing thresholds on shared runners.

Screenshots / Diagrams

Not applicable.

Additional Notes

The mixed service-latency smoke still reports roughly the configured 5 ms local backlog before I/O and remote-inbox service. That is expected: this PR removes per-resume metric overhead but intentionally does not implement the separate scheduler service-quantum change.

Reviewer Guidance

Areas requiring special attention:

• Verify that counter progress, idle transitions, worker replacement, stop, and autoscaler restart all reset the stall interval correctly.
• Check the source-compatible opt-in behavior of last_task_time().
• Confirm the relaxed atomic ordering is sufficient for best-effort diagnostics and TSAN-visible synchronization remains in the integration test.

Questions for reviewers:

• Is the documented sampled on_block contract precise enough about the possible one-tick detection lag?

[Copilot]

Pull request overview

Moves worker-block detection off the coroutine-resume hot path by sampling progress counters in the autoscaler.

Changes:

• Adds sampled worker-progress tracking and opt-in timestamp diagnostics.
• Adds unit/integration coverage and microbenchmarks.
• Documents the sampled on_block contract and performance impact.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
CHANGELOG.md	Records the performance change.
README.md	Summarizes sampled block detection.
examples/microbench.cpp	Benchmarks timestamp diagnostic costs.
include/elio/runtime/autoscaler.hpp	Implements progress-counter sampling.
include/elio/runtime/scheduler.hpp	Removes unconditional timestamp updates.
include/elio/runtime/worker_thread.hpp	Makes timestamp tracking opt-in.
tests/unit/test_autoscaler.cpp	Tests sampling and blocked-worker behavior.
wiki/API-Reference.md	Documents last_task_time().
wiki/Examples.md	Clarifies on_block semantics.
wiki/Performance-Tuning.md	Documents behavior and performance guidance.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Coldwings]

@copilot please re-review the current head only; do not make direct code modifications or push commits.

[Copilot]

@copilot please re-review the current head only; do not make direct code modifications or push commits.

Review of current head (3533758):

include/elio/runtime/worker_thread.hpp

1. last_task_time() is const but has a visible side effect (L263–266). The method is marked [[nodiscard]] … const noexcept and called on a const worker_thread*, but the first call flips track_task_time_ from false to true. That store is never reversed. Any code that calls last_task_time() for observation (e.g. the legacy fallback path in worker_progress_observation::sample) silently enables exact-timestamp collection for that worker permanently. This is documented as intentional opt-in, but the side effect is invisible at the call site and contradicts const. Consider either removing const, using a separate enable_task_time_tracking() mutator, or documenting it as "const by convention but mutating" with a clear // NOLINT or explicit mutable comment on the atomic.

2. track_task_time_ and last_task_time_ share a cache line (L364–365). The alignas(64) on track_task_time_ starts a new line, but last_task_time_ immediately follows without its own alignment. The owner thread writes last_task_time_ (during the unlikely path), while the autoscaler thread reads it. They will land on the same 64-byte line, so the diagnostic path still causes cross-core invalidation on every update, negating part of the isolation benefit. A separate alignas(64) or explicit padding before last_task_time_ would keep them on distinct lines if the isolation is intentional.

include/elio/runtime/autoscaler.hpp

3. block_observations_.resize(num_workers) called every tick (L264). resize is no-op when size is unchanged, but it is called on every autoscaler tick regardless. This is cheap, but if num_workers decreases (scale-down), previously accumulated observations for indices ≥ new_size are destroyed and their stall timers are reset silently; if num_workers then increases again, those slots start fresh. This is probably the intended behavior, but there is no comment explaining it. Worth a brief note that shrinking resets observations for removed slots.

4. worker_progress_observation::sample — idle branch updates worker_ and tasks_executed_ but not active_ (L39–45). When transitioning from idle to active on the next tick, the condition worker_ != identity || !active_ || tasks_executed_ != tasks_executed is satisfied because !active_ is true. That is correct. The idle branch also writes worker_ and tasks_executed_ even though those fields are not checked while active_ == false. This is benign, but slightly surprising — a comment clarifying that the idle branch pre-populates the fields for fast identity/counter detection on the next non-idle transition would help.

5. Legacy fallback path in sample reads worker->last_task_time() (L64). As noted in item 1 above, that call permanently enables exact-timestamp tracking on any custom worker type that exposes last_task_time(). For custom types where the method is genuinely cheap (pre-existing contract), this is fine, but for worker_thread itself the fallback is not reached (it has tasks_executed()), so it only matters for third-party scheduler types. Worth a code comment clarifying that this path intentionally opts the worker into tracking if last_task_time() has that side effect.

tests/unit/test_autoscaler.cpp

6. block_worker_until_released busy-spins on std::this_thread::yield() (L29–32). This is fine for a test coroutine that deliberately blocks its worker, but the spin loop holds the scheduler thread indefinitely in user space. On a shared/low-core CI runner this can starve other threads. A short std::this_thread::sleep_for(std::chrono::microseconds(100)) inside the loop would reduce CPU pressure while still blocking the worker.

7. block_capture_action static members are never reset between test cases (L78–86). reset() is called explicitly in the on_block samples worker execution progress test, but if a future test case also uses block_capture_action without calling reset(), it may see stale counts from earlier runs. Calling reset() in a fixture or at the start of every test that reads calls/worker_id/blocked_ms would make the isolation explicit.

Overall

The core design — sampling counters in the autoscaler tick rather than publishing timestamps on every resume — is sound. The worker_progress_observation state machine correctly distinguishes idle, first-seen-active, stalled, and progress cases. The integration test exercises the full path deterministically. The items above are correctness or documentation hazards rather than fundamental design problems.

[Coldwings]

Addressed the current-head review points in a2332d3:

• added explicit enable_task_time_tracking() while retaining the documented implicit last_task_time() opt-in for backward compatibility;
• isolated the timestamp storage from the per-resume tracking flag cache line;
• documented observation reset and legacy custom-worker behavior;
• reduced the blocked-worker test CPU pressure and made action-state reset explicit.

Local Release, ASAN, and TSAN validation remains green for both [autoscaler] (8 cases / 38 assertions) and [scheduler] (74 cases / 541 assertions).

@copilot please re-review the current head only; do not make direct code modifications or push commits.

[Copilot]

Addressed the current-head review points in a2332d3:

• added explicit enable_task_time_tracking() while retaining the documented implicit last_task_time() opt-in for backward compatibility;...

Review of current head (a2332d3):

All seven points from the previous review have been addressed. Here is the verification pass:

include/elio/runtime/worker_thread.hpp

1. ✅ enable_task_time_tracking() is now a separate, explicit non-const mutator (L261–264). last_task_time() retains the side-effect for backward compatibility and documents it in both the doc-comment and the inline comment. The mutable annotation on track_task_time_ is present and explained (L373–374).

2. ✅ last_task_time_ now has its own alignas(64) (L375–377), isolating it from track_task_time_ on a separate cache line. The comment explains the intent.

include/elio/runtime/autoscaler.hpp

3. ✅ block_observations_.resize(num_workers) is preceded by a comment explaining that shrinking discards observations for removed workers and that a later worker at the same index starts with a fresh baseline (L267–268).

4. ✅ The idle branch comment explains that the pre-populated fields accelerate the next active-sample transition (L40–41).

5. ✅ The legacy fallback comment documents that last_task_time() may have the opt-in side effect for custom worker types (L64–67).

tests/unit/test_autoscaler.cpp

6. ✅ block_worker_until_released now sleeps 100 µs per loop iteration (L82) instead of busy-spinning.

7. ✅ block_capture_action::reset() is called at the start of the integration test case (L239) and the existing autoscaler custom trigger pack test now calls block_capture_action::reset() in place of the direct calls.store(0) (L160).

No new issues found in this revision. The design, state-machine logic, and test coverage look correct.