Server-paged aggregation, cohort queries, and projection (#1) - #2
Merged
Conversation
Investigations were pulling hundreds of raw spans to compute percentiles
client-side, overflowing the tool output limit and — because a single spans
response is capped at 500 rows — producing wrong numbers with no warning.
- aggregate_spans: pages /call/{id}/spans to exhaustion and returns
count/min/mean/max/percentiles, a first-third-vs-last-third trend, and
optional grouping by node/phase/event_name/turn/turn_bucket:N. Reports
complete:false rather than quietly measuring a prefix, and refuses to
compute a trend across mixed node types.
- aggregate_calls: same statistic across a cohort selected with list_calls
filters, per call and pooled, so "is this one call or the campaign?" is
one tool call.
- get_call_spans: limit/offset paging, and a leading TRUNCATED notice when
the page is a prefix of the match set.
- get_call_details: sections= and transcript_range= projection, so a 100KB+
payload no longer has to be spilled to disk to read the metadata.
- search_spans: rejects metadata_filter without a node/event_name scope —
unscoped it silently returns 0 hits against spans that carry the key — and
exposes the event_name and phase filters that provide that scope.
- _get retries 502/503/504 and transport errors twice with backoff.
- get_schema appends the semantics that produce wrong numbers when assumed:
turn_number is bot_stopped_count (not a user-turn counter), metadata_filter
scoping, row caps, per-table retention, and where token telemetry does and
does not live.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up probing of the live ext API corrected two things in the previous
commit.
/search accepts event_name and phase and filters on neither: phase="ttfb" and
phase="complete" return byte-identical rows, and node=llm.openai&event_name=
llm.usage returns 20 llm.request spans. Exposing them invited exactly the
silent-wrong-answer this PR exists to remove, so search_spans no longer takes
them, and its metadata_filter guard now requires node alone. count_spans does
honour both and is unchanged. Verified as genuinely filtering on /search: node,
level, campaign_id, value_ms_min/max, node-scoped metadata_filter.
Per-turn token counts turned out to already be in spans — the llm.usage event
carries prompt_tokens, cache_read_input_tokens, completion_tokens and
total_tokens keyed by turn_number — but only on providers that emit it
(simplismart does, the OpenAI path does not). aggregate_spans and
aggregate_calls take metric="metadata:<key>" to measure a numeric metadata
field instead of value_ms, so prompt-context growth comes from spans (30-day
retention) rather than trace logs (4 days):
aggregate_spans(sid, node="llm", event_name="llm.usage",
metric="metadata:prompt_tokens", group_by="turn")
Stat keys lose the _ms suffix when the metric is not a duration, since tokens
are not milliseconds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… skew Reverses the workaround from the previous commit. squadrun/lens#55 makes /search apply both filters, so removing them from the client was fixing the wrong layer — a client workaround for a bug being fixed upstream in the same breath. They deploy separately though, and installs here are per-machine git pulls, so a client can outrun the backend. search_spans now inspects the rows it gets back and prefixes FILTER_IGNORED when no row matches a filter it asked for, which is proof the backend dropped it. The check is one-sided: absence of matches proves a drop, presence proves nothing, since an ignored filter looks correct on a uniform page. Verified against the current (un-deployed) backend, which trips it. metadata_filter keeps its local guard, now worded as what the server enforces rather than as a workaround for an empty result. Also addresses the invisibility that let four merged commits sit unshipped: version is reported in the get_schema client notes, bumped to 0.2.0, and the README documents the update command, the bump-on-tool-change rule, and the backend-first deploy order. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3 tasks
Concrete vendor/model references (openai, deepgram, gemma4, simplismart) in tool docstrings and the get_schema client notes replaced with neutral placeholders (llm.<provider>, "the full model id") and vendor-free phrasing. Real node/model values come from get_schema (the backend), so examples lose nothing and the docs stop rotting when a provider/model name changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Closes #1. A full latency root-cause investigation through the lens MCP was mostly working around the tool surface, not analysing: to answer "is latency growing across this call?" you pulled 60+ ttfb spans and computed percentiles in Python, two pulls overflowed the tool token limit and had to be spilled to disk, and several behaviours returned empty instead of erroring — indistinguishable from "no data", which led to a wrong conclusion caught in review. Every change here is grounded in a specific thing that cost time or produced a wrong number.
What changed
aggregate_spans(new)group_by=turn_bucket:Nfor the growth curve;metric=metadata:<key>for per-turn tokens.aggregate_calls(new)get_call_detailssections=andtranscript_range=projection — a 258KB payload that overflowed the limit becomes 1,139 chars.search_spansmetadata_filter(returned 0 hits silently against spans that carry the key); re-exposesevent_name/phase; prefixesFILTER_IGNOREDwhen the backend is older than #55 and drops them._getlist_calls504 seen in testing).get_schematurn_numberisbot_stopped_count(not a user-turn counter), row caps, per-table retention, where token telemetry lives.git pull, no auto-update) and reports the running version, so a stale install is diagnosable.How I observed this working
Percentiles cross-checked against the raw rows for the same call — identical both ways:
Payload reduction (real call): raw spans
16,960chars →aggregate_spans836;get_call_details18,149→1,139forsections="metadata".The metadata_filter root cause, reproduced live on spans whose preview is literally
{"model":"gpt-5.6-luna","transport":"sse"}:Now the unscoped form raises with the fix, instead of returning a misleading empty page.
tests/test_stats.pypasses (the only code here that produces a number rather than relaying one) — percentile/grouping/range/metric helpers.Rollout & gating
No flag — new tools + additive params, existing tools unchanged. Install is a per-machine
git pull ~/.cache/lens-mcp/repo(documented in the README); nothing auto-updates, so after merge, users must pull.FILTER_IGNOREDmeans the backend predates squadrun/lens#55 — deploy that first.Risks & deferred scope
event_name/phaseneed lens#55;FILTER_IGNOREDdetects and reports when they're missing rather than returning wrong rows.metric=metadata:per-turn tokens work only where the provider emitsllm.usage(simplismart yes, OpenAI path no) — documented; backend fix is lens#54.Tests
tests/test_stats.py— helper coverage, no new deps (uv run python tests/test_stats.py). Percentiles also cross-checked against raw rows live (above).Related
Followed by #3 (fleet aggregation + discovery tools). Backend counterpart: squadrun/lens#56. Filed from #1.
🤖 Generated with Claude Code