Skip to content

MCP top-queries/top-procedures disclose a shortened raw-tier window (#4231) - #4278

Merged
erikdarlingdata merged 6 commits into
devfrom
fix/4231-raw-window-disclosure
Sep 25, 2026
Merged

erikdarlingdata merged 6 commits into
devfrom
fix/4231-raw-window-disclosure

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Part of #4231.

Why

At "Last 7 days", the WPF Queries grids and two MCP tools read raw tables. Once the rollups are armed, TimescaleDB drops those tables' chunks at 4 days, and none of these reads said so. #2364 fixed this for get_query_store_top: it reads the window floor and reports effective_start and window_truncated.

get_top_queries_by_cpu and get_top_procedures_by_cpu, over query_stats and procedure_stats, never got that fix. A caller who asked for 7 days of CPU ranking got about 4 days back, and hours_back came back unchanged. The WPF grids showed the same short window with no note.

What changes

This is the Darling half of #4231 (stage 1 of its ruling). The Lite half is #4279. Stage 3, routing the top-N reads to the hourly rollups, waits on #4186.

  • RawWindowFloor (new, in PerformanceMonitor.Darling.Storage) is one floor probe for the three raw tables: query_stats, procedure_stats and query_store_stats. It generalizes get_query_store_top reports a window it cannot serve: raw query_store_stats is dropped at 4 days #2364's DarlingDataReader.QueryStoreWindowFloorSql. It lives in Storage so the WPF viewer calls the same probe without referencing the service assembly. IsTruncated reuses DurationTrendRouting.TruncationSlack, the 90-minute boundary that get_query_store_top and the duration trends already use.
  • DarlingDataReader: QueryStoreWindowFloorSql and GetQueryStoreWindowFloorAsync now call the shared probe, with the same SQL text and behavior. New GetQueryStatsWindowFloorAsync and GetProcedureStatsWindowFloorAsync serve the two tools.
  • get_top_queries_by_cpu and get_top_procedures_by_cpu now return effective_start, effective_hours_back and window_truncated, plus a truncation_note when the window was cut short. The note text is Lite's, word for word (Lite MCP tools disclose a truncated query-stats window (#4231) #4279). The names and meaning match get_query_store_top. Every existing field, default and argument is unchanged.
  • get_query_store_top calls RawWindowFloor.IsTruncated in place of its own AddMinutes(90). The value is the same.
  • WPF viewer: the Top Queries, Top Procedures and Query Store grids each read their table's floor through the shared probe. When the floor cuts the window short, a note above the grid says "Showing since" and the effective start. The time is in the viewer's display time, formatted like the trend chart titles. The note refreshes on a toolbar-window load and on each slicer drag. Every window it probes is UTC, the same window the grid read.
  • Web viewer: the three tools' truncation_note shows through the existing get_pg_index_bloat needs a coverage summary (trusted/suppressed counts) — its unmeasured-first sort has misled three sessions in one night #3278 noteKey option, on the built-in server tabs and in the starter dashboard template. The Top Queries panel on the Queries tab is built by hand, so it renders the note itself.
  • Tests and pins: rows for the two tools in McpPayloadContractCensusTests (WindowFloorBlocks, WindowFloorTools), and the tools/list byte pins (DarlingMcpDataTools.txt, McpToolsListBudgetTests).

The probe's cost

On a rig with 6 daily chunks per table for one server, EXPLAIN (ANALYZE, BUFFERS) showed an index-only scan under a Limit for each table:

Table Index Buffers Time
query_stats idx_query_stats_time 5 0.054 ms
procedure_stats idx_procedure_stats_time 2 0.024 ms
query_store_stats idx_query_store_stats_time 2 0.034 ms

Each read stops at the first row of the oldest chunk in the window, even with 6 chunks present.

The served descriptions

The two tools' served heads are unchanged from dev (474 and 406 bytes). Both tools are shared with Lite. McpToolGuideTests.EverySharedToolName_CarriesTheMarkerOnBothSkus_OrNeither_WithByteIdenticalHeads requires their heads to match Lite's byte for byte, so this PR cannot change them alone. The disclosure sits in each tool's get_tool_guide tail, through McpHelpers.WindowTruncatedDescription. It comes before McpToolGuideTopics.CpuTimeExtremesAndAttribution, so the tail still ends on the CPU topic that McpToolGuideHeadsDataTests.CpuTimeExtremesTopic_RidesOnBothTopByCpuTools requires.

For the same reason, McpPayloadContractCensusTests.EveryWindowFloorTool_CarriesTheSharedClause_AndNoOtherToolDoes reads these two tools' guardrail from the full description, not the head. The code states that reason.

TotalCeilingBytes is 174,373, measured on the merged tree after #4272 and #4273. This PR adds no head bytes.

Left for #4231

Test plan

  • dotnet build Darling/Darling.Tests/Darling.Tests.csproj -c Debug: 0 warnings, 0 errors.
  • Full Darling.Tests, once, on commit e67ff186 after a merge of dev: 13,974 total, 5 failed, 49 skipped. The lane fixed all five, and two more that the fixes caused:
    • RepoFileAdoptionTests.TheLfReadingPins_AreExactlyTheOnesDeclaredHere: RawWindowFloorViewerPortTests.cs joins the declared LF-read list, because its web-source pin counts noteKey: "truncation_note" exactly only under an LF read.
    • McpToolGuideHeadsDataTests.EveryConvertedHead_CarriesItsGuardrailFact_AndThePointer and McpToolGuideTests.EveryConvertedHead_StaysAtOrUnder620Characters: the two heads are back at dev's length.
    • McpToolGuideHeadsDataTests.CpuTimeExtremesTopic_RidesOnBothTopByCpuTools: the tail order is swapped, as above.
    • McpToolGuideTests.EverySharedToolName_CarriesTheMarkerOnBothSkus_OrNeither_WithByteIdenticalHeads: both heads match Lite's again.
    • McpPayloadContractCensusTests.EveryWindowFloorTool_CarriesTheSharedClause_AndNoOtherToolDoes: the carve-out above.
    • McpToolsListBudgetTests.EveryServedDescription_AndEveryParameterDescription_MatchesItsCeiling: DarlingMcpDataTools.txt holds the measured 406 and 474 bytes.
  • The PR tender merged dev with MCP budget: describe_custom_view_catalog groups measures by source, under the response budget (#4198) #4272 and MCP budget: get_query_store_top's default stays under 32 KB (#4198) #4273 (commit 40f58504). Then these classes passed with no failures: McpToolsListBudgetTests, McpPayloadContractCensusTests, McpToolGuideTests, McpToolGuideHeadsDataTests, RepoFileAdoptionTests, the QueryStoreTop and QueryStoreClutter classes, the RawWindowFloor classes, QueryStoreRegressionsWebDefaultTests and DarlingWebEndpointsTests.
  • The live classes (RawWindowFloorMcpLiveTests and the live DarlingMcpDataToolsTests) need PostgreSQL. They skipped locally after the last fixes, so GitHub Actions CI decides.

CHANGELOG entry

SECTION: Fixed
ENTRY:

🤖 Generated with Claude Code

https://claude.ai/code/session_01FVjn4PBJN71NQXdFo6ZxNQ

erikdarlingdata and others added 4 commits September 25, 2026 07:54
…s_by_cpu (#4231)

query_stats and procedure_stats are raw-only tables dropped at 4 days when the
rollups are armed, and get_top_queries_by_cpu / get_top_procedures_by_cpu never
reported how far back their window actually reached. Adds a shared RawWindowFloor
probe (generalized from #2364's single-table QueryStoreWindowFloorSql) and wires
both tools to publish effective_start / effective_hours_back / window_truncated,
the same names and meaning get_query_store_top already uses.

Part of #4231. WPF grid headers and the web viewer note are not in this commit;
see the PR body for the handoff.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
…er note (#4231)

get_top_queries_by_cpu / get_top_procedures_by_cpu now carry Lite's (#4279) exact
truncation_note sentence, including the "or this server has been monitored for
less time than that" clause Q1a's version was missing.

WPF: Top Queries / Top Procedures / Query Store grids each read their raw table's
floor through the shared RawWindowFloor probe (ViewerDataService.*WindowFloorAsync)
and show "Showing since <effective start>" in the grid header when truncated,
formatted the way Performance Trends already formats a truncated head. Wired into
both the toolbar-window load and each slicer-drag re-read.

Web: the same three MCP tools' truncation_note now renders through panels.js's
existing #3278 noteKey opt-in, on the built-in server tabs and the starter
dashboard template.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
…sclosure

# Conflicts:
#	Darling/Darling.Tests/McpToolsListBudgetTests.cs
…cription heads (#4231)

Q1a2 stopped at its turn limit before re-running the 5 suite failures; unverified.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
erikdarlingdata added a commit that referenced this pull request Sep 25, 2026
…dinator (#4231)

Darling.Tests' McpToolGuideTests.EverySharedToolName_CarriesTheMarkerOnBothSkus_OrNeither_WithByteIdenticalHeads
requires get_query_store_top's served head to be byte-identical on both
SKUs. Darling's own #4231 PR (#4278) is a separate, still-open lane that
keeps its head as on dev, so this reverts Lite's head to match dev's
Darling head exactly (including its now-stale "Lite: no such floor" line)
and moves the corrected #4231 explanation to the tail instead, where
get_tool_guide serves it -- the payload's own window_truncated /
effective_start / effective_hours_back fields (and their tests) were
never wrong, only this shared head sentence.

Also updates Darling.Tests/McpPayloadContractCensusTests.cs's cross-SKU
rosters (WindowFloorBlocks, WindowFloorTools, CutNoteKeys) for the three
Lite tools that legitimately gained McpHelpers.WindowTruncatedDescription
and truncation_note in this PR -- data-only roster additions, no Darling
product code touched.

Verified: Darling.Tests McpToolGuideTests, McpToolGuideHeadsDataTests,
McpPayloadContractCensusTests all green (81/81); Lite.Tests
McpToolsListBudgetTests, McpToolGuideTests, McpToolGuideHeadsDataTests,
McpToolGuideHeadsCollectionLogTests, QueryWindowTruncationTests,
ServerTabCapabilityPinTests all green (37/37).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
erikdarlingdata and others added 2 commits September 25, 2026 09:07
…he twin-pin census check

Three fixes to the checkpoint (fdda3d0) that shrank get_top_queries_by_cpu's
and get_top_procedures_by_cpu's served heads back to dev's byte-identical
values, and the one census assert that shrink left failing:

- DarlingMcpDataTools.cs: swap the tail concatenation order on both tools to
  McpHelpers.WindowTruncatedDescription + McpToolGuideTopics.CpuTimeExtremesAndAttribution,
  so the tail ends with the shared CPU-extremes topic again
  (McpToolGuideHeadsDataTests.CpuTimeExtremesTopic_RidesOnBothTopByCpuTools).
- DarlingMcpDataTools.txt: bank the two tool lines to the now-measured 406/474
  bytes (down from the stale 678/742 sized for the removed head sentence).
- McpToolsListBudgetTests.cs: rewrite the #4231 change-log comments; they
  claimed +550 head bytes that no longer exist now the head matches dev.
- McpPayloadContractCensusTests.cs: EveryWindowFloorTool_CarriesTheSharedClause_AndNoOtherToolDoes
  required window_truncated / "not a page cut" in these two tools' HEAD, which
  is what the checkpoint's shrink had removed to satisfy D6's twin pin
  (McpToolGuideTests.EverySharedToolName_CarriesTheMarkerOnBothSkus_OrNeither_WithByteIdenticalHeads,
  shared with Lite's #4279). The two requirements are unsatisfiable together
  for a tool pair split across two PRs targeting the same tree: a Darling-only
  head sentence fails the twin pin regardless of merge order, so D6 wins.
  Scoped the census check to read both facts from the tail for just these two
  tools, with the reason and a note that a follow-up can move one identical
  sentence into both SKUs' heads once #4279 merges.

Targeted classes (151 total, 0 failed): RepoFileAdoptionTests,
McpToolGuideHeadsDataTests, McpToolGuideTests, McpPayloadContractCensusTests,
McpToolsListBudgetTests, QueryStoreTopWindowTests, QueryStoreClutterTests,
RawWindowFloorSharedHelperSourcePinTests.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
…sclosure

McpToolsListBudgetTests: keep every change-log line. #4231 adds no head
bytes, so TotalCeilingBytes is dev's measured total after #4272 and #4273
(174,373), re-measured on the merged tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FVjn4PBJN71NQXdFo6ZxNQ
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 25, 2026 13:52
@erikdarlingdata
erikdarlingdata merged commit f0370dc into dev Sep 25, 2026
17 of 18 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/4231-raw-window-disclosure branch September 25, 2026 13:52
erikdarlingdata added a commit that referenced this pull request Sep 25, 2026
* Lite: window-floor disclosure for get_top_queries_by_cpu, get_top_procedures_by_cpu, get_query_store_top (#4231)

query_stats, procedure_stats and query_store_stats are raw-only in Lite (no rollup fallback), and the
default 30-day retention_days is per-collector and user-settable, so a window can silently serve less
than asked for. Adds one shared floor probe (LocalDataService.GetQueryWindowFloorAsync) reading the same
v_ view (hot table UNION archived parquet) each grid/tool reads, and wires effective_start /
effective_hours_back / window_truncated into the three MCP tools that read those tables, using
McpQueryTools.TruncationSlack (already defined for the trend tools) as the threshold.

Part 1 of #4231's Lite half (helper + MCP tools + tests). Grids (Lite/Controls/ServerTab.*) are next.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

* Lite grids disclose a truncated query-stats window (#4231)

The Top Queries, Top Procedures and Query Store grids (and their time-range
slicers, which re-read the same grid over a narrower window) now show
"Showing since <effective start>" in the header when the shared
GetQueryWindowFloorAsync probe finds the raw table's floor cutting the
requested window short -- the same words and probe the three MCP tools
added in the previous commit on this branch use, via one shared
RefreshWindowTruncatedBannerAsync helper so the grid and the tool can never
disagree.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

* Pin the shared window-floor probe and the grid banner text (#4231)

A source pin (QueriesTabGridReads_RouteThroughSharedWindowFloorHelper)
confirms every Queries-tab grid and slicer read of query_stats,
procedure_stats and query_store_stats routes through the ONE shared
GetQueryWindowFloorAsync probe and calls RefreshWindowTruncatedBannerAsync,
never a hand-rolled second copy. Two more tests pin
ServerTab.SetWindowTruncatedBanner's text and visibility for the truncated
and not-truncated cases. Verified the source pin catches a dropped call
site by temporarily removing one and reverting.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

* Fix Lite MCP tool-head budget regressions from the #4231 disclosure text (#4231)

The earlier commits on this branch put the new raw-tier/window_truncated
sentence in the SERVED HEAD (before <<GUIDE>>) for get_top_procedures_by_cpu
and get_query_store_top, pushing them over their tools/list budget pins and,
for get_top_procedures_by_cpu, over the 620-char hard target. Both already
carry the same fact via the shared McpHelpers.WindowTruncatedDescription
appended in the guide tail, so the head text was redundant as well as
oversized.

- get_top_procedures_by_cpu: dropped the redundant head sentence, matching
  get_top_queries_by_cpu's existing (correct) pattern of leaving the
  disclosure to the tail.
- get_query_store_top: trimmed the head sentence to the essential fact
  (raw-tier retention floor, no rollup) instead of restating what the tail
  already says.
- Banked the resulting saving in McpToolsListBudget/McpQueryTools.txt
  (get_query_store_top's ceiling: 422 -> 390).
- Lite.Tests/McpToolGuideHeads.CollectionLog.cs: the "Lite: no such floor"
  guardrail fact and its dedicated test were pinning the PRE-#4231 shape
  (Lite genuinely had no floor then); updated both to the current, true
  fact and renamed the test accordingly.
- Lite.Tests/McpToolGuideHeads.Data.cs: CpuTimeExtremesTopic_RidesOnBothTopByCpuTools
  asserted the CPU-extremes topic ends each tool's tail; it no longer does,
  since WindowTruncatedDescription now trails it on both tools. Switched
  the assertion from EndsWith to Contains.

Full Lite.Tests suite pending in the next commit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

* Keep get_query_store_top's head byte-identical to Darling's, per coordinator (#4231)

Darling.Tests' McpToolGuideTests.EverySharedToolName_CarriesTheMarkerOnBothSkus_OrNeither_WithByteIdenticalHeads
requires get_query_store_top's served head to be byte-identical on both
SKUs. Darling's own #4231 PR (#4278) is a separate, still-open lane that
keeps its head as on dev, so this reverts Lite's head to match dev's
Darling head exactly (including its now-stale "Lite: no such floor" line)
and moves the corrected #4231 explanation to the tail instead, where
get_tool_guide serves it -- the payload's own window_truncated /
effective_start / effective_hours_back fields (and their tests) were
never wrong, only this shared head sentence.

Also updates Darling.Tests/McpPayloadContractCensusTests.cs's cross-SKU
rosters (WindowFloorBlocks, WindowFloorTools, CutNoteKeys) for the three
Lite tools that legitimately gained McpHelpers.WindowTruncatedDescription
and truncation_note in this PR -- data-only roster additions, no Darling
product code touched.

Verified: Darling.Tests McpToolGuideTests, McpToolGuideHeadsDataTests,
McpPayloadContractCensusTests all green (81/81); Lite.Tests
McpToolsListBudgetTests, McpToolGuideTests, McpToolGuideHeadsDataTests,
McpToolGuideHeadsCollectionLogTests, QueryWindowTruncationTests,
ServerTabCapabilityPinTests all green (37/37).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

* Give the window-truncated banner the same UTC window the grid read (#4279)

The slicer handlers passed server-local fromServer/toServer (ServerTimeHelper.ToServerTime)
to RefreshWindowTruncatedBannerAsync, and the custom-range refresh path passed server-local
cStart/cEnd, while GetQueryWindowFloorAsync compares straight against UTC collection_time.
On any server not on UTC the banner probed a window shifted by the server's offset.

Slicers now pass e.StartUtc/e.EndUtc directly. Refresh.cs computes the banner's window through
a new internal LocalDataService.GetQueriesTabWindowUtc helper -- the same GetTimeRange call
GetTopQueriesByCpuAsync/GetTopProceduresByCpuAsync/GetQueryStoreTopQueriesAsync already use for
their own window -- so the grid and its banner can never drift onto two different ranges again.

The comparison calls on the same lines (RefreshQueryStatsComparisonAsync and its two twins) are
deliberately untouched: GetComparisonRange() (ServerTab.Comparison.cs:59) is UTC for the default
hoursBack window but server-local for a custom range, the same split as the bug just fixed, so
giving the "current" side a UTC window while the baseline stays server-local would desync them
instead of fixing them. That needs GetComparisonRange() itself rerouted, which is a separate
design decision.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

* One shared window-floor head, byte-identical in Lite and Darling (#4231)

get_query_store_top's head drops the "Darling: "/"Lite: no such floor"
split for one shared sentence: window_truncated is a window floor, not
a page cut, because stored history can be shorter than asked. Both
get_top_queries_by_cpu and get_top_procedures_by_cpu gain the same
sentence in their heads (both stay under the 620-char budget). Lite's
CPU-tool tails now order McpHelpers.WindowTruncatedDescription before
McpToolGuideTopics.CpuTimeExtremesAndAttribution, matching Darling, and
all four now join with exactly one space at each seam.

Updates the pin tests that expected the old split text: the census
carve-out for the two CPU tools, both McpToolGuideHeads.CollectionLog
twin tests (rewritten as one shared-head assertion), and new guardrail
rows in both McpToolGuideHeads.Data files.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

* Sync #4231 head budgets: query_store_top shorter, both CPU tools grow (#4279)

Lite's #4279 lands the shared window-floor head, so the twin exemption
in Darling's budget no longer applies. Re-measured tools/list for both
SKUs: get_query_store_top's head drops the old split for the shared
sentence (422 -> 351), get_top_queries_by_cpu and get_top_procedures_by_cpu
each gain that same sentence (474 -> 600, 406 -> 532). Net +181 bytes on
both sides. Updated the three per-tool lines and each project's
TotalCeilingBytes (174,373 -> 174,554 Darling; 92,041 -> 92,222 Lite),
with a change-log comment recording the deltas.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FVjn4PBJN71NQXdFo6ZxNQ

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant