Wait names lose their trailing space, and two Hyperscale timer waits are ignored

[erikdarlingdata]

What does this PR do?

Four SQL Server wait names come back from the server with a trailing space. The collectors now trim the name where they read it. RBIO_COMM_RETRY and SQP_STATS_REPORTING, two Azure SQL Database Hyperscale timer waits, join the default ignore list. Lite and Darling share these collectors, so both apps change.

Why

The four names are EXTERNAL_GOVERNANCE_ATTR_SYNC_BACKGROUND, EDC_DOPP_LOCK, EDC_DOPP_BACKGROUND and SQP_STATS_REPORTING. T-SQL ignores the trailing space, so the server never shows a problem. Every comparison in this product is exact. That includes the C# ignore set, the DuckDB filters and the PostgreSQL predicates. An ignore entry without the space never matched one of these names. A lookup by the clean name found nothing.

A 3-hour measurement on one Hyperscale database (93 buckets of 2 minutes) shows two background waits that are timers:

• RBIO_COMM_RETRY has one waiter and a steady 1,000 ms of wait per second (minimum 960, maximum 1,126). It had 735 waits that averaged exactly 15.0 seconds.
• SQP_STATS_REPORTING has one waiter and 37 waits that averaged exactly 300.0 seconds. The server reports this name with a trailing space.

One waiter on a fixed interval that does not follow the workload is a timer, not contention. Kept in the charts, it only adds a constant line.

REMOTE_BLOCK_IO has the same timer shape in that measurement, and it stays visible. Its name says real remote I/O. The test workload most likely never forced a page-server read. Microsoft Learn lists it as "Internal use only". A Hyperscale operator needs to see a real page-server read wait.

A store on a SQL Server target holds none of the four names. The wait_stats collector writes only types with wait time above zero. All four sit at zero on the SQL Server 2022 and 2025 instances checked. Only Azure targets see a key change.

What changes

• New helper WaitTypeName.Trim in the Collectors assembly. It removes trailing whitespace and passes null through.
• These readers trim the name as it leaves the result set. They are wait_stats, waiting_tasks, query snapshots, the stall probe, and the significant waits that the system_health parser reads. The wait_stats and waiting_tasks readers trim before the ignore check. The wait_stats reader also trims before the delta key and the stored value.
• Lite's Live Snapshot button runs the query snapshots query but reads the rows itself. That read moves into ServerTab.ReadLiveSnapshotRow, which trims the name the same way. Darling's live Active Queries fetch already reads through the collector.
• The trim is in C#, not in the T-SQL with RTRIM. The query text is a pinned parity contract. The system_health parser has no T-SQL of its own. A test can drive the real reader with the spaced name. RTRIM also removes only the space character, and TrimEnd removes all trailing whitespace.
• The Collectors and Common assemblies carry no reference to each other, and PerfmonCounterTypeTests pins that. So the system_health parser in Common repeats the one-line TrimEnd(), with a comment that points at the helper.
• The trimmed name is a new key for the delta calculator. The first sample under it is a baseline and has no delta. Tests pin this in both apps. Rows stored with the space stay and age out with retention. There is no migration.
• RBIO_COMM_RETRY and SQP_STATS_REPORTING join IgnoredWaitDefaults.All and Lite's config/ignored_wait_types.json. A test keeps the two lists in step. The measurement sits in a comment beside the entries. That comment also says why REMOTE_BLOCK_IO is not listed.
• The README count of benign wait types goes from 124 to 126.

Which component(s) does this affect?

• Lite
• Darling
• Lite Tests
• Darling Tests
• SQL collection scripts
• Documentation
• Full Dashboard (deprecated)
• CLI Installer (deprecated)

Test plan

• Lite.Tests and Darling.Tests build with 0 warnings and 0 errors. DARLING_CONFIG was unset for every build and test run.
• Full Lite.Tests suite on the head, 6aaba7e: Total 6453, Failed 0.
• Full Darling.Tests suite at 6e095b4: Total 19065, Failed 0, Skipped 1197, Not Run 1. The live PostgreSQL classes skip without a rig. The Not Run test is marked explicit. After that commit, 70ebbb2 edits one block comment in IgnoredWaitDefaults.cs, and 6aaba7e changes only Lite and Lite.Tests. Neither commit changes code that Darling runs.
• Every class in each touched test file ran: WaitNameTrimTests, WaitNameTrimBaselineLiteTests, StallWaitProbePolicyTests, SharedCollectorDefaultsPinTests and WaitNameTrimDarlingTests.
• A SELECT-only query on the local SQL Server 2022 and 2025 instances shows the four names with a trailing space. Each has wait_time_ms of 0 and no clean twin.
• A live collector pass against SQL Server and Azure SQL Database. Not run. The reader pins use a fake reader that returns the spaced names.
• The live PostgreSQL classes. Not run, because DARLING_TEST_PG is not set here. This change adds no PostgreSQL query or migration.

The baseline pins model an upgrade. A store holds a row with the spaced name, the delta calculator seeds from it, and the first pass reads the same name. The Lite pin uses a real DuckDB file and the real DeltaCalculator. The Darling pin uses the real DarlingDeltaCalculator with the spaced baseline in its cache, as the store seed leaves it. A clean-named wait seeded the same way is the control. It gets a real delta on the same pass.

Red on the base code

The new pins ran with the read sites, the two ignore entries and the system_health trim reverted. The helper stayed so the tests compile. Each line below is red on that code and green with this change.

Lite.Tests, 12 of the 49 tests in the four classes failed:

SharedCollectorDefaultsPinTests.IgnoredWaitDefaults_NameTheHyperscaleTimers_AndLeaveRemoteBlockIoVisible [FAIL]
SharedCollectorDefaultsPinTests.IgnoredWaitDefaults_MatchLiteBundledJson [FAIL]
StallWaitProbePolicyTests.ReadAsync_TrimsTheTrailingSpaceAWaitNameCarries [FAIL]
WaitNameTrimTests.WaitStats_ReadAsync_StoresTheTrimmedName_AndKeysTheDeltaOnIt [FAIL]
WaitNameTrimTests.WaitStats_ReadAsync_ASpacedNameMatchesItsIgnoreEntry [FAIL]
WaitNameTrimTests.WaitStats_ReadAsync_WithTheShippedList_DropsTheHyperscaleTimers_AndStillCollectsRemoteBlockIo(useBundledJson: False) [FAIL]
WaitNameTrimTests.WaitStats_ReadAsync_WithTheShippedList_DropsTheHyperscaleTimers_AndStillCollectsRemoteBlockIo(useBundledJson: True) [FAIL]
WaitNameTrimTests.WaitingTasks_ReadAsync_StoresTheTrimmedName [FAIL]
WaitNameTrimTests.WaitingTasks_ReadAsync_WithTheShippedList_DropsTheHyperscaleTimers_AndStillCollectsRemoteBlockIo(useBundledJson: False) [FAIL]
WaitNameTrimTests.WaitingTasks_ReadAsync_WithTheShippedList_DropsTheHyperscaleTimers_AndStillCollectsRemoteBlockIo(useBundledJson: True) [FAIL]
WaitNameTrimTests.QuerySnapshots_ReadAsync_StoresTheTrimmedWaitType [FAIL]
WaitNameTrimBaselineLiteTests.FirstSampleUnderTheTrimmedKey_IsABaseline_NotTheRunningTotal [FAIL]

Darling.Tests, 6 of the 7 tests in the class failed:

WaitNameTrimDarlingTests.TheSharedIgnoreList_NamesTheHyperscaleTimers_AndLeavesRemoteBlockIoVisible [FAIL]
WaitNameTrimDarlingTests.WaitStats_WithDarlingsIgnoreSet_DropsTheTimers_AndStillCollectsRemoteBlockIo [FAIL]
WaitNameTrimDarlingTests.WaitingTasks_WithDarlingsIgnoreSet_DropsTheTimers_AndStillCollectsRemoteBlockIo [FAIL]
WaitNameTrimDarlingTests.FirstSampleUnderTheTrimmedKey_IsABaseline_NotTheRunningTotal [FAIL]
WaitNameTrimDarlingTests.SignificantWait_TrimsATrailingSpaceInTheWaitTypeText [FAIL]
WaitNameTrimDarlingTests.SignificantWait_ASpacedNameStillMatchesTheGatesIgnoreList [FAIL]

Some pins pass on the base code by design. The four helper tests in Lite need the helper. The Darling runner pin holds behavior that already exists and that the new entries depend on. Each of them went red under a mutation.

The helper mutation and the guard mutation ran alone. The list entry and the runner binding ran together in one pass, and each failure set below belongs to one of them. Only the list entries can fail a list pin, and only the runner source can fail the runner pin. Git restored every mutated file after the runs:

• The first-sample baseline guard removed, so a new key returns its running total as the first delta. In Lite, WaitNameTrimBaselineLiteTests.FirstSampleUnderTheTrimmedKey_IsABaseline_NotTheRunningTotal [FAIL] was the only failure among the 15 tests in the two WaitNameTrim classes (Expected 0, Actual 101). The Darling test of the same name was also the only failure, with the same values.
• The helper returns its input. In Lite, 13 of those 15 tests failed, including the four helper tests, and so did StallWaitProbePolicyTests.ReadAsync_TrimsTheTrailingSpaceAWaitNameCarries. In Darling, 3 of 7 failed: the two ..._AndStillCollectsRemoteBlockIo tests and the baseline test.
• REMOTE_BLOCK_IO added to both lists. In Lite, the four shipped-list rows and both SharedCollectorDefaultsPinTests pins failed. In Darling, the two ..._AndStillCollectsRemoteBlockIo tests and TheSharedIgnoreList_NamesTheHyperscaleTimers_AndLeavesRemoteBlockIoVisible failed.
• One IgnoredWaitTypes binding in DarlingCollectorRunner replaced with an empty set. WaitNameTrimDarlingTests.TheServiceRunner_BindsTheSharedIgnoreList_AndNoOtherSource [FAIL].
• The trim removed from ServerTab.ReadLiveSnapshotRow, run alone. That method is new in this change, so its pin cannot run on the base code. WaitNameTrimTests.LiveSnapshot_ReadRow_ShowsTheTrimmedWaitType [FAIL]: Expected "EXTERNAL_GOVERNANCE_ATTR_SYNC_BACKGROUND", Actual "EXTERNAL_GOVERNANCE_ATTR_SYNC_BACKGROUND " (the name with its trailing space).

What to double-check

• Existing Lite installs keep their own per-user ignored_wait_types.json. The seeder never overwrites it. So the two new entries reach new Lite installs and Darling only. A merge of new defaults into an existing list adds back any entry that a user removed on purpose. That is a product decision, so this PR leaves it.
• Old rows stay. On an Azure target, the wait lists can show "EDC_DOPP_LOCK " and "EDC_DOPP_LOCK" side by side until retention removes the old rows. Lite's display filter compares exact names and skips an entry that has a space, so it is unchanged.
• The wait_stats clear detector sums every cached baseline. A stale spaced baseline adds its old value to that sum. The test that needs most types to read lower never sees that key. So it cannot cause a false clear.
• Left alone on purpose:
  • SystemHealthSignificance holds a verbatim copy of the sp_HealthParser ignore list. The parser now trims the name before the gate compares it.
  • The Darling Query Store clutter list derives its QDS_ names from the shared list. None of the names in this change start with QDS_.
  • SessionSummaryStatsCollector uses wait_type only in a server-side LIKE test that counts sessions. It stores no name.
  • DmvBlockingSnapshotCollector does store a value built from the name. Its lock_mode column is the wait type with the LCK_M_, PAGEIOLATCH_ and PAGELATCH_ prefixes removed. But its filter admits only LCK_, PAGELATCH_, PAGEIOLATCH_ and RESOURCE_SEMAPHORE waits, so none of the four names can reach it.
  • The delta seeders in both apps load the stored name as it is. A row stored before the upgrade seeds a spaced key, and the trimmed name is a new key.

CHANGELOG

SECTION: Fixed
ENTRY:

• Wait names lose a trailing space, and two Hyperscale timer waits are ignored ([Wait names lose their trailing space, and two Hyperscale timer waits are ignored #4884]): Four SQL Server wait names carry a trailing space. They are EXTERNAL_GOVERNANCE_ATTR_SYNC_BACKGROUND, EDC_DOPP_LOCK, EDC_DOPP_BACKGROUND and SQP_STATS_REPORTING. An ignore entry or a lookup by the clean name never matched them. Both apps now trim the name as they read it. The first sample under the trimmed name is a baseline, so no spike shows. Rows stored with the space stay until retention removes them. RBIO_COMM_RETRY and SQP_STATS_REPORTING join the default ignore list for new Lite installs and for Darling, because they are fixed-interval timers on Hyperscale. Existing Lite installs keep their own list. REMOTE_BLOCK_IO stays visible.
  REF:
  [Wait names lose their trailing space, and two Hyperscale timer waits are ignored #4884]: Wait names lose their trailing space, and two Hyperscale timer waits are ignored #4884