Skip to content

Commit acc1c86

Browse files
bai-uipathclaude
bai-uipath
and
claude
authored
feat(evalboard): make analytics surfaces harness-aware and stream tables (#45)
* feat(evalboard): scope task trends to one harness Trends aggregated the last N pipeline runs regardless of harness, but the nightly now rotates claude-code / codex / antigravity as separate runs — so a single per-task pass rate, status strip, and averages were blended across harnesses with incomparable pass rates and cost profiles, making reliably-passing tasks read as broken. Scope the page to one harness (default claude-code, the daily primary) with a switcher, and fold the tag-rail counts into the cached aggregate so the window loads once instead of twice per view. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(evalboard): make all pages harness-aware and stream tables Extend harness scoping beyond trends to the whole board and make the harness set data-driven so a new harness (e.g. delegate) appears on its own: - lib/harness.ts: shared leaf module (constants + normalize + a whitelist-free param parser) so client and server can import without bundle contamination. - listRecentHarnesses(): cached discovery of which harnesses have recent runs; the switcher lists exactly those (plus the active one), no hardcoded set. - Watchlist and path-to-ga: scoped to one harness (default claude-code) with a selector, like trends — their blended pass rates / streaks / charts were mixing incomparable harnesses. - Front page: analytics (success chart + tag rail) scoped to a harness; the run LIST stays all-harness (its job is to show every recent run, and the Harness column already disambiguates each row). - Tag rail preserves the active harness across chip clicks (fixes a reset on trends too). Also stream the trends and watchlist tables behind a Suspense boundary so the header + selector paint immediately instead of blocking on the cold data load. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(evalboard): match watchlist skeleton header to avoid layout shift The Suspense fallback rendered only the harness selector, while the loaded view adds a "last N … runs" pill in the same right-aligned group, so the selector jumped left when the pill appeared. Mirror the pill (pulsing count) in the skeleton so the header geometry is stable through streaming. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(evalboard): address review nits on harness plumbing - Use the DEFAULT_HARNESS constant for the default-harness value in trends.ts and overview.ts's never-empty fallback instead of re-hardcoding the "claude-code" literal, removing a silent drift trap. - Render harnessShortLabel in the trends provenance + empty-state prose so the view no longer shows both "Claude Code" and "claude-code". - Drop a no-op readonly cast in the harness selector. - Add parseHarnessParam tests (array/absent/whitespace/valid/reject/length). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 6df6e9b commit acc1c86

17 files changed

Lines changed: 656 additions & 84 deletions

File tree

‎evalboard/app/_components/harness-badge.tsx‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,35 @@
11
import Image from "next/image";
22

3-
// Vendor logo for a run's harness (RunConfig). Renders the recognizable
3+
// Canonical harness constants live in the leaf lib/harness module (shared by
4+
// the server data layer); re-exported here so existing badge importers are
5+
// unaffected.
6+
export { KNOWN_HARNESSES, type KnownHarness } from "@/lib/harness";
7+
8+
// Vendor logo + labels for a run's harness (RunConfig). Renders the recognizable
49
// vendor mark instead of raw "claude-code"/"codex"/"antigravity" text,
510
// mirroring the Slack rollup's vendor emoji. A missing harness defaults to
611
// claude-code (the nightly). This is an internal-only column — the caller
712
// gates it behind isInternal (see lib/edition.ts).
8-
const HARNESS_LOGO: Record<string, { src: string; label: string }> = {
13+
const HARNESS_LOGO: Record<string, { src: string; label: string; short: string }> = {
914
"claude-code": {
1015
src: "/harness/claude-code.png",
1116
label: "Claude Code · Anthropic",
17+
short: "Claude Code",
1218
},
13-
codex: { src: "/harness/codex.png", label: "Codex · OpenAI" },
19+
codex: { src: "/harness/codex.png", label: "Codex · OpenAI", short: "Codex" },
1420
antigravity: {
1521
src: "/harness/antigravity.png",
1622
label: "Antigravity · Google Gemini",
23+
short: "Antigravity",
1724
},
1825
};
1926

27+
// Short human label for a harness id ("Claude Code"), for selectors and prose.
28+
// Unknown ids fall through to the raw id.
29+
export function harnessShortLabel(harness: string): string {
30+
return HARNESS_LOGO[harness]?.short ?? harness;
31+
}
32+
2033
export function HarnessBadge({ harness }: { harness?: string | null }) {
2134
const key = harness ?? "claude-code";
2235
const logo = HARNESS_LOGO[key];

‎evalboard/app/_components/harness-selector.tsx‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
"use client";
2+
3+
import { usePathname, useRouter, useSearchParams } from "next/navigation";
4+
import { HarnessBadge, harnessShortLabel } from "./harness-badge";
5+
6+
// Segmented control for the trends page's harness scope. Sets `?h=<harness>`
7+
// while preserving the active q/tag params, mirroring WindowSelector. Each
8+
// segment shows the vendor logo + short label so it reads like the per-run
9+
// harness badge on the runs tables.
10+
export function HarnessSelector({
11+
current,
12+
harnesses,
13+
}: {
14+
current: string;
15+
harnesses: readonly string[];
16+
}) {
17+
const router = useRouter();
18+
const pathname = usePathname();
19+
const searchParams = useSearchParams();
20+
const set = (h: string) => {
21+
const p = new URLSearchParams(searchParams.toString());
22+
p.set("h", h);
23+
router.replace(`${pathname}?${p.toString()}`, { scroll: false });
24+
};
25+
// Always show the active harness, even if it has aged out of the recent
26+
// window (so a deep-linked `?h=` still reads as selected rather than absent).
27+
const opts = harnesses.includes(current)
28+
? harnesses
29+
: [current, ...harnesses];
30+
return (
31+
<div className="inline-flex border border-gray-200 rounded-md overflow-hidden text-sm">
32+
{opts.map((h) => {
33+
const active = h === current;
34+
return (
35+
<button
36+
key={h}
37+
type="button"
38+
onClick={() => set(h)}
39+
aria-pressed={active}
40+
className={`flex items-center gap-1.5 px-3 py-1 ${active ? "bg-studio-blue text-white" : "bg-white text-gray-700 hover:bg-gray-50"}`}
41+
>
42+
<HarnessBadge harness={h} />
43+
{harnessShortLabel(h)}
44+
</button>
45+
);
46+
})}
47+
</div>
48+
);
49+
}

‎evalboard/app/_overview/tag-rail.tsx‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import Link from "next/link";
22
import type { TagCount } from "@/lib/overview";
3+
import { DEFAULT_HARNESS } from "@/lib/harness";
34
import type { Window } from "@/lib/reviews-types";
45

56
type Variant = "neutral" | "rose" | "indigo";
@@ -35,11 +36,16 @@ function hrefForTag(
3536
tag: string | null,
3637
window: Window | null,
3738
q: string | null,
39+
harness: string | null,
3840
): string {
3941
const params = new URLSearchParams();
4042
if (window) params.set("window", window);
4143
if (tag) params.set("tag", tag);
4244
if (q) params.set("q", q);
45+
// Preserve the active harness scope across tag clicks (omit the default to
46+
// keep URLs clean). Without this, filtering by a tag would silently reset a
47+
// codex/antigravity view back to claude-code.
48+
if (harness && harness !== DEFAULT_HARNESS) params.set("h", harness);
4349
const qs = params.toString();
4450
return qs ? `${basePath}?${qs}` : basePath;
4551
}
@@ -52,6 +58,7 @@ function TagChip({
5258
basePath,
5359
window,
5460
q,
61+
harness,
5562
}: {
5663
tag: string;
5764
count: number;
@@ -60,11 +67,12 @@ function TagChip({
6067
basePath: string;
6168
window: Window | null;
6269
q: string | null;
70+
harness: string | null;
6371
}) {
6472
const s = STYLES[variant];
6573
return (
6674
<Link
67-
href={hrefForTag(basePath, active ? null : tag, window, q)}
75+
href={hrefForTag(basePath, active ? null : tag, window, q, harness)}
6876
scroll={false}
6977
className={`inline-flex items-center gap-1 text-[11px] leading-none px-2 py-1 rounded border transition-colors ${active ? s.chipActive : s.chip}`}
7078
>
@@ -111,6 +119,7 @@ export function MergedTagRail({
111119
basePath = "/",
112120
window = null,
113121
q = null,
122+
harness = null,
114123
limit = 24,
115124
}: {
116125
skills: TagCount[];
@@ -123,6 +132,8 @@ export function MergedTagRail({
123132
// Null on pages that don't expose a window selector (e.g. /trends).
124133
window?: Window | null;
125134
q?: string | null;
135+
// Active harness scope to preserve in chip links (null = not harness-scoped).
136+
harness?: string | null;
126137
limit?: number;
127138
}) {
128139
const s = pickShown(skills, limit, activeTag);
@@ -146,6 +157,7 @@ export function MergedTagRail({
146157
basePath={basePath}
147158
window={window}
148159
q={q}
160+
harness={harness}
149161
/>
150162
))}
151163
{r.shown.map((tc) => (
@@ -158,6 +170,7 @@ export function MergedTagRail({
158170
basePath={basePath}
159171
window={window}
160172
q={q}
173+
harness={harness}
161174
/>
162175
))}
163176
{t.shown.map((tc) => (
@@ -170,6 +183,7 @@ export function MergedTagRail({
170183
basePath={basePath}
171184
window={window}
172185
q={q}
186+
harness={harness}
173187
/>
174188
))}
175189
{totalRemaining > 0 && (

‎evalboard/app/page.tsx‎

Lines changed: 30 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,10 @@ import {
33
getAdhocRunListing,
44
getOverview,
55
getRunListing,
6+
listRecentHarnesses,
67
type TagCount,
78
} from "@/lib/overview";
9+
import { parseHarnessParam, DEFAULT_HARNESS } from "@/lib/harness";
810
import { fmtDuration, fmtRunTime, fmtTimestamp, passClass } from "@/lib/format";
911
import { WindowSelector } from "./_components/window-selector";
1012
import { WINDOWS, type Window } from "@/lib/reviews-types";
@@ -15,7 +17,8 @@ import { ChipLegend, MergedTagRail } from "./_overview/tag-rail";
1517
import { TableScroll } from "./_components/scroll-table";
1618
import { CollapsibleRail } from "./_components/collapsible-rail";
1719
import { isInternal } from "@/lib/edition";
18-
import { HarnessBadge } from "@/app/_components/harness-badge";
20+
import { HarnessBadge, harnessShortLabel } from "@/app/_components/harness-badge";
21+
import { HarnessSelector } from "@/app/_components/harness-selector";
1922

2023
export const dynamic = "force-dynamic";
2124

@@ -98,6 +101,7 @@ export default async function Page({
98101
window?: string;
99102
tag?: string;
100103
q?: string;
104+
h?: string;
101105
limit?: string;
102106
alimit?: string;
103107
}>;
@@ -106,14 +110,20 @@ export default async function Page({
106110
const window = parseWindow(params.window);
107111
const activeTag = parseTag(params.tag);
108112
const q = parseQ(params.q);
113+
const harness = parseHarnessParam(params.h);
109114
const limit = parseLimit(params.limit);
110115
const adhocLimit = parseAdhocLimit(params.alimit);
111116
const isFiltered = activeTag != null || q != null;
112117

113-
const [overview, listing, adhoc] = await Promise.all([
114-
getOverview(window, activeTag, q),
118+
// The analytics block (chart + rails) is scoped to one harness so the
119+
// success line stops zigzagging across incomparable harnesses. The run
120+
// LIST stays all-harness — seeing every recent run is the page's job, and
121+
// the Harness column already disambiguates each row.
122+
const [overview, listing, adhoc, harnesses] = await Promise.all([
123+
getOverview(window, activeTag, q, harness),
115124
getRunListing(window, activeTag, q, limit),
116125
getAdhocRunListing(adhocLimit),
126+
listRecentHarnesses(),
117127
]);
118128

119129
const skills = filterTagsByQuery(overview.skills, q);
@@ -133,10 +143,14 @@ export default async function Page({
133143
const rawAlimit = Array.isArray(params.alimit)
134144
? params.alimit[0]
135145
: params.alimit;
146+
// Omit the default harness from URLs to keep them clean; carry a non-default
147+
// scope through every self-link so it isn't reset by pagination/clear.
148+
const hParam = harness === DEFAULT_HARNESS ? undefined : harness;
136149
const base = {
137150
window,
138151
tag: activeTag,
139152
q,
153+
h: hParam,
140154
limit: rawLimit,
141155
alimit: rawAlimit,
142156
};
@@ -146,7 +160,7 @@ export default async function Page({
146160
limit: Math.min(tableTotalLabel, shownCount + DEFAULT_LIMIT),
147161
});
148162
const showAllHref = buildHref({ ...base, limit: "all" });
149-
const clearAllHref = buildHref({ window });
163+
const clearAllHref = buildHref({ window, h: hParam });
150164

151165
// Ad-hoc section disclosure: rows are filtered (by `q`) then capped to
152166
// adhocLimit; offer "Show all" while more match than are shown, and a
@@ -212,7 +226,7 @@ export default async function Page({
212226
{overview.runs.length === 1 ? "" : "s"}
213227
{" · "}
214228
<Link
215-
href={buildHref({ window })}
229+
href={buildHref({ window, h: hParam })}
216230
scroll={false}
217231
className="text-studio-blue hover:underline"
218232
>
@@ -221,14 +235,21 @@ export default async function Page({
221235
</>
222236
) : (
223237
<>
224-
Success rate per run across the last{" "}
225-
{window} · {overview.runs.length} run
238+
Success rate per{" "}
239+
{harnessShortLabel(harness)} run across the
240+
last {window} · {overview.runs.length} run
226241
{overview.runs.length === 1 ? "" : "s"}
227242
</>
228243
)}
229244
</p>
230245
</div>
231-
<WindowSelector current={window} />
246+
<div className="flex items-center gap-3">
247+
<HarnessSelector
248+
current={harness}
249+
harnesses={harnesses}
250+
/>
251+
<WindowSelector current={window} />
252+
</div>
232253
</div>
233254
<DailySuccessChart
234255
data={overview.runs}
@@ -261,6 +282,7 @@ export default async function Page({
261282
activeTag={activeTag}
262283
window={window}
263284
q={q}
285+
harness={harness}
264286
limit={24}
265287
/>
266288
</CollapsibleRail>

‎evalboard/app/path-to-ga/page.tsx‎

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,14 @@
11
import Link from "next/link";
2-
import { getOverview, getTagTaskBreakdown } from "@/lib/overview";
2+
import {
3+
getOverview,
4+
getTagTaskBreakdown,
5+
listRecentHarnesses,
6+
} from "@/lib/overview";
7+
import { parseHarnessParam } from "@/lib/harness";
38
import { humanizeTaskId } from "@/lib/format";
49
import { WindowSelector } from "../_components/window-selector";
10+
import { HarnessSelector } from "../_components/harness-selector";
11+
import { harnessShortLabel } from "../_components/harness-badge";
512
import { WINDOWS, type Window } from "@/lib/reviews-types";
613
import { DailySuccessChart } from "../_overview/daily-chart";
714
import { TableScroll } from "../_components/scroll-table";
@@ -26,14 +33,18 @@ function passClass(pct: number | null): string {
2633
export default async function PathToGaPage({
2734
searchParams,
2835
}: {
29-
searchParams: Promise<{ window?: string }>;
36+
searchParams: Promise<{ window?: string; h?: string }>;
3037
}) {
3138
const params = await searchParams;
3239
const window = parseWindow(params.window);
40+
const harness = parseHarnessParam(params.h);
3341

34-
const [overview, taskRows] = await Promise.all([
35-
getOverview(window, TAG, null),
36-
getTagTaskBreakdown(window, TAG),
42+
// Scope to one harness — readiness of a task is per-harness, and a blended
43+
// chart/pass-rate mixes incomparable regimes.
44+
const [overview, taskRows, harnesses] = await Promise.all([
45+
getOverview(window, TAG, null, harness),
46+
getTagTaskBreakdown(window, TAG, harness),
47+
listRecentHarnesses(),
3748
]);
3849

3950
const runsInWindow = overview.runs.length;
@@ -52,10 +63,14 @@ export default async function PathToGaPage({
5263
</h1>
5364
<p className="text-sm text-gray-500">
5465
Score for every task tagged{" "}
55-
<span className="font-mono text-gray-700">{TAG}</span>.
66+
<span className="font-mono text-gray-700">{TAG}</span> on{" "}
67+
{harnessShortLabel(harness)}.
5668
</p>
5769
</div>
58-
<WindowSelector current={window} />
70+
<div className="flex items-center gap-3">
71+
<HarnessSelector current={harness} harnesses={harnesses} />
72+
<WindowSelector current={window} />
73+
</div>
5974
</div>
6075

6176
<section className="border border-gray-200 rounded-lg bg-white p-4 space-y-4">

‎evalboard/app/trends/__tests__/trends-view.test.tsx‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,22 @@
11
import { describe, expect, test, vi } from "vitest";
22
import { render, screen } from "@testing-library/react";
33
import type { TaskTrend } from "@/lib/trends";
4+
import { KNOWN_HARNESSES } from "@/app/_components/harness-badge";
45

56
// The view imports the server action for the expandable history rows; stub it
67
// so rendering doesn't pull the blob-backed loader into a jsdom test.
78
vi.mock("../actions", () => ({
89
fetchTaskHistoryAction: vi.fn(async () => []),
910
}));
1011

12+
// The header's HarnessSelector reads router/params hooks; stub them so the view
13+
// renders in jsdom without a router provider.
14+
vi.mock("next/navigation", () => ({
15+
useRouter: () => ({ replace: vi.fn() }),
16+
usePathname: () => "/trends",
17+
useSearchParams: () => new URLSearchParams(),
18+
}));
19+
1120
const { TrendsView } = await import("../trends-view");
1221

1322
function trend(overrides: Partial<TaskTrend>): TaskTrend {
@@ -35,6 +44,8 @@ function renderView(tasks: TaskTrend[], runIds: string[]) {
3544
runIds={runIds}
3645
q={null}
3746
activeTag={null}
47+
activeHarness="claude-code"
48+
harnesses={KNOWN_HARNESSES}
3849
skills={[]}
3950
taskTags={[]}
4051
reviewTags={[]}

0 commit comments

Comments
 (0)