perf(runtime): reduce activation working set entry overhead

[ReubenBond]

Summary

• Store the CLOCK idle and registration bits on each activation and pack them with the existing activation lifecycle status.
• Use the ActivationWorkingSet dictionary only for membership, eliminating the per-entry state object and dictionary updates during active/idle transitions.
• Serialize scans and lifecycle transitions using the member lock which ActivationData already requires, while treating weakly-consistent enumeration as advisory CLOCK behavior.
• Prevent completed keep-alive requests from re-registering activations after deactivation begins.
• Cover the state machine with generated reference-model traces, focused deterministic race schedules, and real ActivationData integration tests.

Impact

• Removes one 24-byte state object per working-set member, saving about 22.9 MiB per million activations and reducing managed object count.
• Packing the working-set bits does not increase activation object size.
• BenchmarkDotNet measured a full active-to-idle-to-active CLOCK cycle at 22.6 ns versus 21.5 ns for the original object-state implementation (approximately 5% slower and 3× faster than the generation-tagged dictionary design).

[Copilot]

Pull request overview

This pull request reduces activation working-set entry overhead by storing idle state directly in dictionary values.

Changes:

• Replaces per-entry state objects with Boolean values.
• Adds atomic state transitions and conditional eviction handling.
• Preserves existing public interfaces.

Show a summary per file

File	Summary	Review status
src/Orleans.Runtime/Catalog/ActivationWorkingSet.cs	Optimizes working-set storage and transition logic.	Critical eviction/re-add race requires an entry identity or generation check before OnIdle.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Suppressed comments (1)

src/Orleans.Runtime/Catalog/ActivationWorkingSet.cs:80

• The correctness-sensitive part of this optimization is the CAS-based active/idle transition, but the existing tests only add members and exercise collector buckets; they do not drive OnActive, VisitMember, or conditional eviction. Please add focused working-set tests covering active→idle→active transitions and a stale scan entry racing with reactivation before relying on this change.

                if (!isIdle || _members.TryUpdate(member, false, comparisonValue: true))
                {

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite

[github-actions]

Code coverage

77.50% line coverage - 97,377 / 125,646 lines

Coverage details

• Source files: 1,659
• Scope: every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS
• Sources: maintained repository code under src/ loaded by the complete CI test scope
• Commit: 4818a539b1
• View the test run and merged coverage artifact
• View the coverage reporting run

[Copilot]

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

src/Orleans.Runtime/Catalog/ActivationWorkingSet.cs:289

• IActivationWorkingSetMember is a public interface in Orleans.Runtime, so adding this required property is a source/binary compatibility break for external implementations. Unlike the observer methods, this member has no default implementation, so existing consumers implementing the interface will fail when upgrading. Please preserve compatibility with an internal state carrier/adapter, or otherwise handle this as an intentional major-version API change before exposing the property.

public interface IActivationWorkingSetMember
{
    /// <summary>
    /// Gets or sets whether this member was idle during the previous working-set scan.
    /// </summary>
    bool IsIdle { get; set; }

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

test/Orleans.Core.Tests/Runtime/ActivationCollectorTests.cs:654

• This synchronization does not make the test deterministic. After resumeScan.Set(), the first VisitMember releases the member lock, and the monitor can immediately begin the next scan while reactivationTask is competing for the same lock. If reactivation wins, the second scan observes IsIdle == false and produces another idle/active transition instead of the expected eviction, causing the assertions below to fail intermittently. Pause the scan on the wouldRemove == true visit (or otherwise gate the timer) before starting the reactivation.

                resumeScan.Set();
                await reactivationTask;
                await lifecycle.OnStop();

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (2)

src/Orleans.Runtime/Catalog/ActivationData.cs:1545

• OnDeactivating now clears IsInWorkingSet, so a request which was already running when deactivation starts (including DeactivateOnIdle, or a deactivation selected just before the request began) reaches this block with the bit cleared and re-registers the deactivating activation via OnActive. That temporarily makes it visible to the working set and can keep a stuck/deactivating activation counted until its final cleanup; the previous explicit-deactivation path deliberately left this bit set. Only re-add after a completion when the activation is still Valid (or otherwise exclude the deactivating state).

                if (!IsInWorkingSet)
                {
                    IsInWorkingSet = true;
                    _shared.InternalRuntime.ActivationWorkingSet.OnActive(this);
                }

src/Orleans.Runtime/Catalog/ActivationWorkingSet.cs:70

• The member is published to _members before _activeCount is incremented. If OnEvicted runs after this lock is released but before line 70 (the member lock is the only synchronization used by these methods), it can remove the new entry and decrement the count from zero, making Count temporarily negative and allowing an invalid working-set count to be published. Increment the count while still holding the member lock, as OnActive does.

        Interlocked.Increment(ref _activeCount);

• Files reviewed: 5/5 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (2)

src/Orleans.Runtime/Catalog/ActivationWorkingSet.cs:166

• The member lock only serializes VisitMember with mutations after the visit starts; it does not tie the enumerated key to the membership that was present when it was yielded. If enumeration yields a member, another thread removes and re-adds it, and then this visit runs, IsInWorkingSet is true for the new registration and a candidate result can mark that new registration idle (and emit OnIdle). Preserve an entry identity/generation, or otherwise prevent a pre-removal enumeration from mutating a re-added registration. The added race tests block re-addition while VisitMember holds the lock, so they do not cover this ordering.

        // Enumeration can retain a member across removal and re-addition. CLOCK state is advisory, so visit the
        // member's current state while holding its lock instead of adding a dictionary validation to every scan.
        lock (member)
        {
            if (!member.IsInWorkingSet)

src/Orleans.Runtime/Catalog/ActivationWorkingSet.cs:50

• A ConcurrentDictionary enumerator can retain an entry after OnEvicted removes it. Since OnEvicted now clears IsIdle, this filter can yield the removed member as active from Members (the IncomingRequestMonitor then analyzes it even though it is no longer registered). Include the member-registration bit in this filter so the advisory weak enumeration does not expose evicted activations.

            if (!pair.Key.IsIdle)
            {

• Files reviewed: 5/5 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Copilot review overview

Review tier: Lite
Findings: 1

New issues introduced by this change (1)

Severity	Finding
	src/​Orleans.Runtime/​Catalog/​ActivationWorkingSet.cs — EnumerateActiveMembers() filters IActivationWorkingSetMemberStatus entries only by IsIdle. If…

[Copilot]

Copilot review overview

Review tier: Lite
Findings: 1

Pre-existing issues (1)

Severity	Finding
	src/​Orleans.Runtime/​Catalog/​ActivationWorkingSet.cs — EnumerateActiveMembers() filters IActivationWorkingSetMemberStatus entries only by IsIdle. If… View comment

[Copilot]

Copilot review overview

Review tier: Lite
Findings: 1

Pre-existing issues (1)

Severity	Finding
	src/​Orleans.Runtime/​Catalog/​ActivationWorkingSet.cs — EnumerateActiveMembers() filters IActivationWorkingSetMemberStatus entries only by IsIdle. If… View comment

Suppressed comments (1)

src/Orleans.Runtime/Catalog/ActivationWorkingSet.cs:52

• Members enumeration can yield an evicted status-backed member if ConcurrentDictionary enumeration includes a stale entry: OnEvicted clears IsIdle, so the current predicate (!status.IsIdle) can return true even when status.IsInWorkingSet == false. Filtering on IsInWorkingSet avoids reporting members which have been removed from the working set (as indicated by the member’s own status bit).

            if (pair.Key is IActivationWorkingSetMemberStatus status
                ? !status.IsIdle
                : (pair.Value & IsIdleMask) == 0)

[Copilot]

Copilot review overview

🔵 Needs a closer look

Review tier: Lite
Findings: None

Issues resolved since last review (1)

Severity	Finding
	src/​Orleans.Runtime/​Catalog/​ActivationWorkingSet.cs — EnumerateActiveMembers() filters IActivationWorkingSetMemberStatus entries only by IsIdle. If… View resolved comment

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/Orleans.Runtime/Catalog/ActivationData.cs:432

• State and the working-set flags are read using Volatile.Read(ref _status), but SetState writes _status using a plain store. For lock-free readers to reliably observe updates across threads, the write side should also use Volatile.Write (or make _status volatile).

This issue also appears on line 434 of the same file.

    public void SetState(ActivationState state)
    {
        Debug.Assert(Monitor.IsEntered(this));
        _status = (byte)((_status & ~ActivationStateMask) | (byte)state);
    }

src/Orleans.Runtime/Catalog/ActivationData.cs:438

• The _status field is read via Volatile.Read(ref _status) from multiple threads (e.g., IsInWorkingSet/IsIdleInWorkingSet getters), but SetStatusFlag updates it with a non-volatile write. Use Volatile.Write (or make _status volatile) so other threads reliably observe flag changes.

    private void SetStatusFlag(byte mask, bool value)
    {
        Debug.Assert(Monitor.IsEntered(this));
        _status = value ? (byte)(_status | mask) : (byte)(_status & ~mask);
    }