Lite FinOps tests: CPU samples sit inside the 24-hour utilization read at any time of day

[erikdarlingdata]

Why

Six Lite FinOpsTests fail every day between 03:45 and 04:00 UTC and pass the rest of the day. They failed CI on #4882, which ran in that quarter hour.

The FinOps CPU scenarios seeded their CPU samples from TestDataSeeder.TestPeriodStart. That time is anchored to 04:00 UTC, because the analysis scenarios need a fixed midnight (#2177, #4385). Before 04:00 the anchor is 04:00 the day before, so the 16 samples sat at 00:00 to 03:45 the day before.

GetUtilizationEfficiencyAsync reads the last 24 hours from now (LocalDataService.FinOps.Utilization.cs, cutoff = DateTime.UtcNow.AddHours(-24)). After 03:45 it found no sample. HasCpuSample was false, so the CPU and VM right-sizing rules gave no advice, and the tests that expect that advice failed.

What changes

Test-only. No product file changes.

• TestDataSeeder takes an optional clock (Func<DateTime>). When it is left out, the seeder uses the real UTC clock. The FinOps CPU samples are placed back from that clock: 15 minutes apart, the newest 5 minutes before now (FinOpsCpuSampleTimes).
• Seven scenarios use it: over-provisioned Enterprise, right-sizing, VM right-sizing target, Azure SQL Database vCore, clean FinOps server, stable CPU for reserved capacity, and bursty CPU. The last three seed 32 samples, the others 16.
• The clean FinOps server used to seed two CPU copies of 16 samples, one from the anchor and one from now. The second copy was added earlier to get around this same gap. It now seeds one copy of 32 samples, from now. SeedFinOpsCpuUtilizationAsync takes the sample count.
• The clean server needs the 32 samples. Rule 14 (reserved capacity) returns no row under 24 samples, so with 16 it never reaches its stddevCpu > 0 guard. With 32 it does. The CPU is a flat 50, a standard deviation of 0, so that guard is what keeps rule 14 quiet.
• The analysis scenarios keep the 04:00 anchor, and SeedCpuUtilizationAsync works as before. Its doc now says its samples are placed from that anchor. It sends a new FinOps scenario to SeedFinOpsCpuUtilizationAsync and to the window test's scenario list.
• New class FinOpsCpuSampleWindowTests:
  • A theory seeds each of the 7 scenarios with the clock fixed at 03:50 UTC and at 12:00 UTC. It checks that every CPU sample is inside the 24 hours before that clock, the window the read uses. The fixed day is 30 days before the run. A scenario that goes back to the anchored seed then fails on any day, not only between 03:45 and 04:00.
  • A source pin checks that GetUtilizationEfficiencyAsync still reads the last 24 hours. It covers the cutoff line, the parameter that carries it, and the cpu_stats filter collection_time >= $2.

The FinOps reads the seeder feeds

Read	Window	Result
GetUtilizationEfficiencyAsync, cpu_stats (rules 2 and 12 need its HasCpuSample)	last 24 hours from now	The gap. Fixed here.
GetUtilizationEfficiencyAsync, grants	last 24 hours from now	No FinOps scenario seeds memory_grant_stats. Only analysis scenarios do.
GetUtilizationEfficiencyAsync, mem_latest and server_info	newest row	No window.
Rule 3 memory, v_memory_stats	7 days	Covers the anchor.
Rule 6 GetIdleDatabasesAsync, v_query_stats	7 days	Covers the anchor.
GetDatabaseSizeLatestAsync	newest collection_time	No window.
Rule 11 jobs	7 days	Its seeds are already placed from now.
Rule 12 VM, CPU and memory	7 days	Covers the anchor.
Rule 13 storage, file I/O	7 days	Covers the anchor.
Rule 14 reserved capacity, CPU	7 days	Covers the anchor.
GetHighImpactQueriesAsync	hoursBack from now	The test already passes an hoursBack that reaches TestPeriodStart (#4558), with a pin at 03:59.

The anchored samples are never more than about 28 hours old, so every 7-day window covers them.

The other 24-hour FinOps reads get no data from TestDataSeeder. They are GetServerMetricsAsync, GetTempdbSummaryAsync, GetMemoryGrantEfficiencyAsync and the workload reads. Their tests (FinOpsFleetReadParityTests, FinOpsServerInventoryTests and the AzureSqlDatabase*Tests) seed from DateTime.UtcNow.

Darling

No change needed. The viewer's FinOps reads use the same rolling windows: ViewerDataService.FinOps.Utilization.cs reads the last 24 hours. But Darling.Tests has no anchored seeder, and its FinOps tests seed from now. ServerInventoryFleetMetricsLiveTests seeds CPU at 1, 2 and 3 hours ago. FinOpsMergedReadsLiveTests and ViewerFinOpsIntervalHonestLiveTests seed at 1 hour ago. ViewerFinOpsRecommendationsTests are source pins.

Test plan

• Fails before the fix. With the anchored sample times put back, the theory fails all 7 scenarios at 03:50 and passes all 7 at 12:00: Total: 15, Errors: 0, Failed: 7. The messages read like OverProvisionedEnterprise at 03:50 UTC: 0 of its 16 CPU samples are inside the 24-hour utilization read and StableCpuForReservedCapacity at 03:50 UTC: 16 of its 32 CPU samples are inside the 24-hour utilization read.
• With the fix: FinOpsCpuSampleWindowTests and FinOpsTests together Total: 37, Errors: 0, Failed: 0.
• Planted regressions. Each was run and then restored:
  • The read's window cut to 12 hours: the source pin fails.
  • The cpu_stats filter changed from >= to >: the source pin fails.
  • The VM right-sizing scenario back on the anchored seed: that scenario fails at 03:50 and at 12:00.
  • The anchored sample times put back: all 7 scenarios fail at 03:50 and pass at 12:00.
  • Rule 14's stddevCpu > 0 guard dropped: CleanServer_NoDuckDbRecommendations fails on a reserved-capacity row, Stable CPU utilization (avg 50.0%, CV 0.00).
  • The same plant with the clean server back at 16 samples: all 37 pass. Without the 32 samples, no test holds that guard.
• Targeted classes while working, each Failed: 0: FinOpsTests (22), FactCollectorTests (42), FactCollectorMiseryTests (23), AzureSqlDatabaseMemoryScopeTests (28), AzureSqlDatabaseHostMathTests (28), AzureSqlDatabaseHardwareTests (9), FinOpsVerdictSourcePinTests (8), FinOpsFleetReadParityTests (5).
• Lite.Tests and Darling.Tests build with 0 warnings and 0 errors.
• Full Lite.Tests suite at the head: Lite.Tests Total: 6571, Errors: 0, Failed: 0, Skipped: 0, Not Run: 0, Time: 290.839s.
• Full Darling.Tests suite at the head, with the live-target variables unset: Darling.Tests Total: 19186, Errors: 0, Failed: 0, Skipped: 1210, Not Run: 1, Time: 147.389s. The skips are the live tests that need a target.

CHANGELOG

SECTION: None