Bug: winformspumphost-tests-load-flaky-visible-window

[drmoisan]

• Work Mode: full-bug

Summary

Unit tests built on QuickFiler.Test/TestSupport/WinFormsPumpHost.cs start a real WinForms message pump (Application.Run) on a dedicated STA thread and construct real WinForms controls. They fail nondeterministically when the machine is under heavy CPU load, and they display a visible window during an otherwise headless test run.

Environment

• OS/version: Windows 11 Pro 10.0.26200
• Runtime: .NET Framework 4.8.1, MSTest via vstest.console.exe (VS18 test platform)
• Command/flags used: ./scripts/vscode/Invoke-MSTestWithCoverage.ps1 -SearchRoot . (full-suite run with coverage)
• Data source or fixture: QuickFiler.Test/TestSupport/WinFormsPumpHost.cs; affected suites include QfcItemController_InitializationTests (the *ThroughThePumpHost* cases) and WpfDispatcherYieldTests

Steps to Reproduce

1. Drive the machine to sustained high CPU utilization (observed at approximately 96%).
2. Run the full MSTest suite with coverage using the command above.
3. Repeat several times and observe both the pass/fail outcome and the desktop.

Expected Behavior

Unit tests are deterministic and headless. Per .claude/rules/general-unit-test.md, tests must produce the same result given the same inputs and must not depend on real wall-clock timing. No unit test should create a visible window on the developer's desktop.

Actual Behavior

• The affected tests failed nondeterministically under load. During baseline capture for issue Bug: quickfiler-search-keystroke-focus-steal #438 on 2026-08-08, six attempts were required to obtain a clean full-suite baseline; the failures were confined to these suites and did not reproduce once the machine was idle.
• A visible window appeared during the run, because the host constructs a real WinForms control and pumps a real message loop.

Logs / Screenshots

• Attached minimal logs or screenshot
• Snippet: no captured failure log is retained; the observation is recorded in the issue Bug: quickfiler-search-keystroke-focus-steal #438 execution report and in .claude/agent-memory/atomic-executor/project_winformspumphost_tests_load_flaky.md. A fresh capture under induced load should accompany the fix.

Impact / Severity

• Blocker
• High
• Medium
• Low

Nondeterministic tests are corrosive to an autonomous development workflow: they force repeated full-suite reruns (six were needed for one baseline), and they train reviewers and agents to retry a red suite rather than investigate it, which is exactly the habit that lets a real regression through. The visible window also makes the suite unsuitable for unattended or headless CI execution.

Source

From: docs/features/potential/2026-08-08-winformspumphost-tests-load-flaky-visible-window.md

[drmoisan]

Additional symptom from the same root cause, observed 2026-08-08 during remediation of #438.

Repo-wide coverage measurement is not reproducible across runs. Five full-suite coverage runs on the same commit produced line-rates clustered around 0.8585-0.8586 rather than a stable value. The variance was isolated to a small, rotating set of legacy classes — EfcHomeController.cs, PropertyStore.cs, SubjectMapSco.Orchestration.cs, and SegmentStopWatch.cs (a wall-clock timing helper).

Concrete impact: a coverage floor recorded in one session (line 0.858665) could not be reproduced by a later run of the unmodified tree, which measured 0.858512. That caused a false coverage-regression failure on a change that had actually increased coverage. Any fixed cross-session coverage constant used as a gate will keep producing false failures until the underlying nondeterminism is fixed.

Also seen in the same session: one hung testhost and an unrelated intermittent failure in WpfDispatcherYieldTests.YieldAsync_WithoutDispatcher_RemainsStrict, which is consistent with the timing-dependent-test hypothesis in this issue.

Two implications worth folding into this issue's scope:

1. Timing-dependent tests affect the coverage metric, not just pass/fail stability.
2. Coverage gates should compare against a baseline captured in the same session and run rather than a constant carried across sessions. Full analysis: docs/features/active/2026-08-07-quickfiler-search-keystroke-focus-steal-438/evidence/other/timestamp-and-coverage-floor-correction.2026-08-08T19-25Z.md on branch bug/quickfiler-search-keystroke-focus-steal-438.

[drmoisan]

Fresh failure capture under load, with a controlled attribution experiment

The issue body notes that "no captured failure log is retained; ... A fresh capture under induced load should accompany the fix." This comment supplies that capture. It was obtained incidentally while fixing #508 (a different order-dependent test), where these failures blocked the final toolchain gate and had to be attributed before that work could proceed.

The two failing tests and the exact exception

Both in QuickFiler.Test, class QuickFiler.Controllers.Tests.QfcItemController_InitializationTests:

• InitializeBool_ThroughThePumpHost_CompletesAndInitializesState
• InitializeNineArgOverload_ThroughThePumpHost_SavesParametersAndDelegates

System.InvalidOperationException: Invoke or BeginInvoke cannot be called on a control
until the window handle has been created.
   at System.Windows.Forms.Control.MarshaledInvoke(...)
   at QuickFiler.Controllers.QfcItemController.InvokeBeginInvoke(Boolean async, Action action)
        in QuickFiler\Controllers\QfcItemController.FocusAndTheme.cs:line 256
   at QuickFiler.Controllers.QfcItemController.ToggleTips(Boolean async, ToggleState desiredState)
        in QuickFiler\Controllers\QfcItemController.FocusAndTheme.cs:line 204
   at QuickFiler.Controllers.QfcItemController.Initialize(Boolean async)
        in QuickFiler\Controllers\QfcItemController.Initialization.cs:line 185
   at ... QuickFiler.Test\TestSupport\WinFormsPumpHost.cs:line 95

This localizes the race precisely: WinFormsPumpHost hands control to QfcItemController.Initialize before the hosted control's window handle exists, and InvokeBeginInvoke at QfcItemController.FocusAndTheme.cs:256 marshals against that not-yet-created handle. It is a handle-creation ordering defect, not a general timing sensitivity.

Controlled four-run experiment (isolates scope, not just symptom)

Run at merge-base 003c5715, varying only the presence of an unrelated UtilitiesCS change:

#	Configuration	Scope	EXIT	Total	Passed	Failed
A	change present	QfcItemController_InitializationTests only, /InIsolation	0	9	9	0
B	change present	full QuickFiler.Test assembly alone, /InIsolation	0	867	867	0
C	change present	full 9-assembly instrumented suite	1	6295	6293	2
C (repeat)	change present	full 9-assembly instrumented suite	1	6295	6293	2
D	change fully reverted	full 9-assembly instrumented suite	1	6293	6291	2

Two findings worth recording:

1. The tests pass in class isolation (9/9) and in their own assembly (867/867), and fail only in the combined dotnet-coverage-instrumented 9-assembly run. Coverage instrumentation overhead is sufficient on its own to trigger the race — induced CPU load is not required to reproduce it. That is a cheaper and more reliable repro than the "drive the machine to ~96% CPU" steps currently in the issue body.
2. Run D is the control: with the unrelated change reverted and git status --porcelain -- '*.cs' '*.csproj' '*.sln' empty, the same two tests fail with the same exception. The failure is intrinsic to merge-base.

6293 / 6291 / 2 from run D is a byte-for-byte match with the "Run 1" baseline recorded in issue #508. So the historical Failed: 2 observation there was these two tests, and #508's own test accounted for the separate Failed: 1 observation in Run 2.

Provenance

The two tests were introduced by commit 8f98264c ("feat(quickfiler-test): add WinForms message-pump test seam (#230)", merged as PR #479), so the defect entered with the pump-host seam itself rather than with a later change.

One scope correction to the issue body

The issue currently lists WpfDispatcherYieldTests among the affected suites. That suite's nondeterminism had a different root cause — an unarranged ambient WPF Dispatcher precondition, not the WinForms pump host — and is fixed under #508 by making the dispatcher lookup injectable. After that fix, WpfDispatcherYieldTests passed 4667/4667 on three consecutive full parallel runs. It should be removed from the affected-suite list here so the remaining scope is unambiguously WinFormsPumpHost and its *ThroughThePumpHost* consumers.

Note also that a WPF Dispatcher running on an owned STA thread does not create a visible window; the visible-window symptom in this issue is specific to WinFormsPumpHost constructing a real WinForms control and running a real Application.Run message loop.

Full artifact

The complete experiment record, including SHA-256 verification that the reverted change was restored byte-for-byte, is committed at:

docs/features/active/2026-08-08-wpf-dispatcher-yield-test-order-dependent-508/evidence/regression-testing/preexisting-failure-attribution.2026-08-08T16-52.md