diff --git a/README.md b/README.md index affde00..738cce2 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,18 @@ -# OpenBot +

+ OpenBot +

-> Open-source, self-hosted GitHub maintenance automation for teams that want to own their bot, prompts, models, data, and cost controls. +

OpenBot

-[![status](https://img.shields.io/badge/status-pre--alpha-orange)](./docs/prd/openbot-prd.md) -[![license](https://img.shields.io/badge/license-Apache--2.0-blue)](./LICENSE) -[![python](https://img.shields.io/badge/python-3.12%2B-blue)](./pyproject.toml) +

+ Open-source, self-hosted GitHub maintenance automation for teams that want to own their bot, prompts, models, data, and cost controls. +

+ +

+ status + license + python +

OpenBot is a GitHub App backend for maintainer workflows: @@ -250,6 +258,17 @@ point yet. The alpha readiness criteria are documented in OpenBot uses evals as product verification, not as a separate demo agent. +![OpenBot eval architecture: an offline lane runs benchmark datasets through Inspect AI tasks, thin solvers and the openbot.evaluation facade into the production agent and sandbox, then scores and exports predictions; a planned online lane harvests traces and GitHub outcome signals from production runs into online eval and 30-day trend metrics. Both lanes write to LangSmith. A coverage strip lists the four product workflows: triage, review, fix and chat.](./docs/assets/openbot-eval-architecture.svg) + +Two lanes over one LangSmith substrate. The **offline** lane is what exists +today: a benchmark dataset is driven through the *production* agent and +sandbox via `openbot.evaluation`, so a suite measures the shipped code path +rather than an eval-only copy of it. The **online** lane is v0.3 in the +[eval PRD](./docs/prd/openbot-eval-prd.md) and is mostly unbuilt — the traces +are already there, the outcome-signal collector and the online judge are not. +Solid borders in the diagram are built; dashed borders are specified but not +implemented. + The target architecture is: ```text diff --git a/docs/assets/openbot-eval-architecture.svg b/docs/assets/openbot-eval-architecture.svg new file mode 100644 index 0000000..6a90eb2 --- /dev/null +++ b/docs/assets/openbot-eval-architecture.svg @@ -0,0 +1,175 @@ + + OpenBot eval architecture + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + OpenBot — eval architecture + two lanes over one LangSmith substrate: offline benchmarks driven through the production agent, online signal from production runs + v0.1 built · v0.3 planned + + OFFLINE · v0.1 · BUILT + + ONLINE · v0.3 · PLANNED + + COVERAGE — THE FOUR PRODUCT WORKFLOWS + + + + + + + + + + + + + + + + + + + + + + + + + + + 1 + benchmark datasets + LangSmith mirrors published by make -C evals data-* + Martian CRB · SWE-bench V · SWE-QA-Pro · SWT-Bench + + 2 + Inspect AI task — one per benchmark + evals/tasks/: review_martian · fix_swe_bench · + chat_swe_qa · test_swt_bench + dataset + solver + scorers wired per task + + 3 + thin solver → openbot.evaluation + evals/solvers/*.py call runner.run_*_sample and adapt + the domain result into scorer / prediction shape + forbidden here: eval-only agents, sandboxes, tools + + 4 + production agent + real sandbox + the facade opens the prod sandbox, clones at base_sha, + then calls DeepAgents review / fix / chat / repro — + the same code the worker runs in production + + 5 + scorers + prediction export + review: LLM judge + overlap · chat: SWE-QA-Pro judge + fix / test: JSONL → official harness (Modal / docker) + → grade writeback as pass@1 feedback + + 1 + production workflow runs + triage · review · fix · chat on real repositories + live traffic, no benchmark dataset + + 2 + auto-instrumented traces + LANGSMITH_TRACING on the web and worker processes; + every agent run already lands in LangSmith today + this half of the online lane exists now + + 3 + outcome signals from GitHub + was the review comment resolved or acted on; did the + chat answer need a follow-up; was a triage label fixed + collector not built yet + + 4 + online eval + annotation queue + sampled LLM judge over live runs, plus human labels + from the LangSmith annotation queue + suites: triage_online · review_online · chat_online + + 5 + 30-day trend metrics + action_rate · resolved_rate · correct_30d · + macro_f1_30d · followup_rate · pass@1_90d + trend observation only — never blocks a merge + LangSmith — one substrate for both lanes + datasets · experiment rows · per-sample feedback · traces · annotation queue + every row carries dataset_version · solver_family=openbot_agent · capability · openbot_git_sha · mode + triage + off · triage_gitbugs (v0.2) + on · macro_f1_30d (v0.3) + no solver in evals/ yet + review + off · review_martian — mean F1 + on · action / resolved rate + 10% F1 drop blocks merge + fix + off · fix_swe_bench — pass@1 + on · fix_swe_bench_live 90d + rolling benchmark, not prod + chat + off · chat_swe_qa — 5-dim + on · correct_30d · followup + judge must exist; warn only + sample + solve() + run_*_sample + domain result + experiment row + traced + run id + sampled + scored + trend rows + + + offline path — built + + online path — planned + + writes to LangSmith + solid = built today · dashed = specified in the eval PRD but not built · test_swt_bench is a fifth surface, not a workflow + + diff --git a/docs/assets/openbot-logo.png b/docs/assets/openbot-logo.png new file mode 100644 index 0000000..703b559 Binary files /dev/null and b/docs/assets/openbot-logo.png differ diff --git a/docs/assets/openbot-logo.svg b/docs/assets/openbot-logo.svg new file mode 100644 index 0000000..e6f1bce --- /dev/null +++ b/docs/assets/openbot-logo.svg @@ -0,0 +1,11 @@ + + OpenBot + + + + + + + + diff --git a/docs/assets/openbot-social-preview.png b/docs/assets/openbot-social-preview.png new file mode 100644 index 0000000..9c8ca43 Binary files /dev/null and b/docs/assets/openbot-social-preview.png differ diff --git a/docs/assets/openbot-social-preview.svg b/docs/assets/openbot-social-preview.svg new file mode 100644 index 0000000..0964ad6 --- /dev/null +++ b/docs/assets/openbot-social-preview.svg @@ -0,0 +1,20 @@ + + OpenBot + + + + + + + + + + + OpenBot + Self-hosted GitHub maintenance automation + triage · review · fix · chat — your repo, your keys, your data + + diff --git a/scripts/_diagram_svg.py b/scripts/_diagram_svg.py new file mode 100644 index 0000000..251445e --- /dev/null +++ b/scripts/_diagram_svg.py @@ -0,0 +1,245 @@ +"""Shared Style 2 ("Dark Terminal") SVG emitter for the diagrams under ``scripts/``. + +``build_architecture_diagram.py`` and ``build_eval_architecture_diagram.py`` +draw with the same palette, arrow markers and semantic attributes; only their +node/edge tables differ. Keeping the emitter in one place stops the two +diagrams drifting apart visually, and means both carry the same +``data-graph-role`` / ``data-node-id`` / ``data-edge-id`` contract that the +fireworks-tech-graph geometry and composition validators read. + +Not a general-purpose SVG library — it emits exactly the element shapes those +two diagrams need, in the paint order the validators expect (background → +containers → edges → node shapes → node text → edge labels → legend). +""" + +from __future__ import annotations + +import html + +# ── palette ──────────────────────────────────────────────────────────────── +BG_A, BG_B = "#0f0f1a", "#1a1a2e" +TXT, SUB, MUTE, LINE = "#e2e8f0", "#94a3b8", "#64748b", "#334155" + +# Semantic arrow colours. Each needs a marker pair emitted into . +FLOW, STATE, LLM, POLICY, EVAL = "#3b82f6", "#10b981", "#a855f7", "#f97316", "#64748b" +MARKER_OF = {FLOW: "m-flow", STATE: "m-state", LLM: "m-llm", POLICY: "m-policy", EVAL: "m-eval"} + +# (fill, stroke) per semantic node role. +THEMES = { + "external": ("#132a47", "#3b82f6"), + "compute": ("#1c1917", "#ea580c"), + "queue": ("#052e16", "#059669"), + "agent": ("#1e1b4b", "#7c3aed"), + "store": ("#052e16", "#059669"), + "sandbox": ("#292214", "#eab308"), + "config": ("#0f172a", "#334155"), + "eval": ("#0f172a", "#475569"), + "planned": ("#131a28", "#64748b"), +} + + +def esc(s: str) -> str: + return html.escape(s, quote=True) + + +class Canvas: + """Accumulates SVG source lines in validator-friendly paint order.""" + + def __init__( + self, + width: int, + height: int, + *, + title: str, + aria_label: str, + quality_profile: str = "showcase", + ) -> None: + self.width = width + self.height = height + self.lines: list[str] = [] + self._open(title, aria_label, quality_profile) + + # ── low-level ────────────────────────────────────────────────────────── + def add(self, line: str) -> None: + self.lines.append(line) + + def render(self) -> str: + return "\n".join(self.lines) + "\n" + + # ── document scaffold ────────────────────────────────────────────────── + def _open(self, title: str, aria_label: str, quality_profile: str) -> None: + add = self.add + add( + f'') + add(f" {esc(title)}") + add(" ") + add(" ") + add(' ') + add(f' ') + add(f' ') + add(" ") + for colour, mid in MARKER_OF.items(): + add( + f' ' + ) + add(f' ') + add(" ") + add( + f' ' + ) + add(f' ') + add(" ") + add(" ") + add( + f' ' + ) + + def title_block(self, heading: str, subtitle: str, tag: str) -> None: + add = self.add + add( + f' {esc(heading)}' + ) + add(f' {esc(subtitle)}') + add( + f' {esc(tag)}' + ) + + # ── graph elements ───────────────────────────────────────────────────── + def container( + self, + cid: str, + x: int, + y: int, + w: int, + h: int, + label: str, + lx: int, + ly: int, + *, + dashed: bool = True, + ) -> None: + """Draw a grouping boundary. + + ``dashed`` defaults to True because a dashed boundary is the plain + grouping convention. Pass ``dashed=False`` in a diagram that also + uses dash to mean "not built yet", so a built group is not drawn in + the unbuilt style. + """ + dash_attr = ' stroke-dasharray="6,5"' if dashed else "" + self.add( + f' " + ) + self.add( + f' {esc(label)}' + ) + + def edge( + self, + eid: str, + source: str, + target: str, + d: str, + colour: str, + *, + dash: str | None = None, + twoway: bool = False, + ) -> None: + mid = MARKER_OF[colour] + attrs = [ + 'data-graph-role="edge"', + f'data-edge-id="{eid}"', + f'data-source="{source}"', + f'data-target="{target}"', + f'd="{d}"', + 'fill="none"', + f'stroke="{colour}"', + 'stroke-width="1.8"', + f'marker-end="url(#{mid})"', + ] + if twoway: + attrs.append(f'marker-start="url(#{mid}-start)"') + if dash: + attrs.append(f'stroke-dasharray="{dash}"') + self.add(f" ") + + def node( + self, nid: str, x: int, y: int, w: int, h: int, theme: str, *, dashed: bool = False + ) -> None: + fill, stroke = THEMES[theme] + dash_attr = ' stroke-dasharray="6,4"' if dashed else "" + self.add( + f' ' + ) + + def badge(self, cx: int, cy: int, text: str, stroke: str) -> None: + self.add( + f' ' + ) + self.add( + f' {esc(text)}' + ) + + def text( + self, x: int, y: int, size: int, weight: int, fill: str, txt: str, *, anchor: str = "start" + ) -> None: + anchor_attr = f' text-anchor="{anchor}"' if anchor != "start" else "" + self.add( + f' {esc(txt)}' + ) + + def edge_label( + self, eid: str, x: int, y: int, size: int, colour: str, txt: str, *, anchor: str = "start" + ) -> None: + anchor_attr = f' text-anchor="{anchor}"' if anchor != "start" else "" + self.add( + f' {esc(txt)}' + ) + + def legend( + self, + items: list[tuple[str, str | None, bool, str]], + *, + y: int, + footer: str, + footer_y: int, + ) -> None: + """Emit the swatch row plus a footer note, both excluded from geometry.""" + self.add(' ') + lx = 36.0 + for colour, dash, twoway, text in items: + mid = MARKER_OF[colour] + extra = f' stroke-dasharray="{dash}"' if dash else "" + start = f' marker-start="url(#{mid}-start)"' if twoway else "" + self.add( + f' ' + ) + self.add( + f' {esc(text)}' + ) + lx += 22 + 30 + len(text) * 5.9 + self.add( + f' {esc(footer)}' + ) + self.add(" ") diff --git a/scripts/build_architecture_diagram.py b/scripts/build_architecture_diagram.py index e17d79d..6c1667f 100644 --- a/scripts/build_architecture_diagram.py +++ b/scripts/build_architecture_diagram.py @@ -17,27 +17,11 @@ from __future__ import annotations -import html from pathlib import Path -W, H = 1000, 960 - -BG_A, BG_B = "#0f0f1a", "#1a1a2e" -TXT, SUB, MUTE, LINE = "#e2e8f0", "#94a3b8", "#64748b", "#334155" - -THEMES = { - "external": ("#132a47", "#3b82f6"), - "compute": ("#1c1917", "#ea580c"), - "queue": ("#052e16", "#059669"), - "agent": ("#1e1b4b", "#7c3aed"), - "store": ("#052e16", "#059669"), - "sandbox": ("#292214", "#eab308"), - "config": ("#0f172a", "#334155"), - "eval": ("#0f172a", "#475569"), -} +from _diagram_svg import EVAL, FLOW, LLM, MUTE, POLICY, STATE, SUB, THEMES, TXT, Canvas -FLOW, STATE, LLM, POLICY, EVAL = "#3b82f6", "#10b981", "#a855f7", "#f97316", "#64748b" -MARKER_OF = {FLOW: "m-flow", STATE: "m-state", LLM: "m-llm", POLICY: "m-policy", EVAL: "m-eval"} +W, H = 1000, 960 # --------------------------------------------------------------- geometry grid SPINE_X, SPINE_W = 296, 410 # centre column, centre line x = 501 @@ -320,154 +304,56 @@ (EVAL, "4,3", False, "eval reuse (offline)"), ] -lines: list[str] = [] -A = lines.append - - -def esc(s: str) -> str: - return html.escape(s, quote=True) - - -# --------------------------------------------------------------- header + defs -A(f'') -A(" OpenBot runtime architecture") -A(" ") -A(" ") -A(' ') -A(f' ') -A(f' ') -A(" ") -for colour, mid in MARKER_OF.items(): - A(f' ') - A(f' ') - A(" ") - A( - f' ' - ) - A(f' ') - A(" ") -A(" ") -# --------------------------------------------------------------- 1. background -A(f' ') - -# --------------------------------------------------------------- title block -A( - f' ' - "OpenBot — runtime architecture" -) -A( - f' self-hosted GitHub App: webhook →' - " preflight gates → queued worker → sandboxed agent → write-back" +canvas.title_block( + "OpenBot — runtime architecture", + "self-hosted GitHub App: webhook → preflight gates → queued worker → sandboxed agent" + " → write-back", + "v0.1 · pre-alpha", ) -A(f' v0.1 · pre-alpha') -# --------------------------------------------------------------- 2. containers for cid, cx, cy, cw, ch, label, lx, ly in CONTAINERS: - A( - f' ' - ) - A( - f' {esc(label)}' - ) + canvas.container(cid, cx, cy, cw, ch, label, lx, ly) -# --------------------------------------------------------------- 3. edges for eid, src, tgt, d, colour, dash, twoway, *_ in EDGES: - mid = MARKER_OF[colour] - attrs = [ - 'data-graph-role="edge"', - f'data-edge-id="{eid}"', - f'data-source="{src}"', - f'data-target="{tgt}"', - f'd="{d}"', - 'fill="none"', - f'stroke="{colour}"', - 'stroke-width="1.8"', - f'marker-end="url(#{mid})"', - ] - if twoway: - attrs.append(f'marker-start="url(#{mid}-start)"') - if dash: - attrs.append(f'stroke-dasharray="{dash}"') - A(f" ") + canvas.edge(eid, src, tgt, d, colour, dash=dash, twoway=twoway) -# --------------------------------------------------------------- 4. node shapes for nid, (x, y, w, h, theme) in NODES.items(): - fill, stroke = THEMES[theme] - dash = ' stroke-dasharray="6,4"' if theme == "eval" else "" - A( - f' ' - ) + canvas.node(nid, x, y, w, h, theme, dashed=theme == "eval") -# --------------------------------------------------------------- 5. node text for nid, (badge, text_x, rows) in BODY.items(): x, y, w, h, theme = NODES[nid] _, stroke = THEMES[theme] if badge: - cy = y + rows[0][0] - 5 - A( - f' ' - ) - A( - f' {badge}' - ) + canvas.badge(x + 20, y + rows[0][0] - 5, badge, stroke) for dy, size, weight, fill_c, txt in rows: if text_x is None: - A( - f' {esc(txt)}' - ) + canvas.text(x + w // 2, y + dy, size, weight, fill_c, txt, anchor="middle") else: - A( - f' {esc(txt)}' - ) + canvas.text(text_x, y + dy, size, weight, fill_c, txt) -# --------------------------------------------------------------- 6. edge labels for eid, _src, _tgt, _d, colour, _dash, _twoway, label, lx, ly, anchor, size in EDGES: - anchor_attr = f' text-anchor="{anchor}"' if anchor != "start" else "" - A( - f' {esc(label)}' - ) - -# --------------------------------------------------------------- 7. legend -A(' ') -lx = 36.0 -for colour, dash, twoway, text in LEGEND: - mid = MARKER_OF[colour] - extra = f' stroke-dasharray="{dash}"' if dash else "" - start = f' marker-start="url(#{mid}-start)"' if twoway else "" - A( - f' ' - ) - A(f' {esc(text)}') - lx += 22 + 30 + len(text) * 5.9 -A( - f' cross-cutting — Sentry · LangSmith ·' - " Langfuse · Prometheus tracing on both processes; audit_log is the product-facing ledger" + canvas.edge_label(eid, lx, ly, size, colour, label, anchor=anchor) + +canvas.legend( + LEGEND, + y=916, + footer="cross-cutting — Sentry · LangSmith · Langfuse · Prometheus tracing on both" + " processes; audit_log is the product-facing ledger", + footer_y=942, ) -A(" ") -A("") +canvas.add("") OUT = Path(__file__).resolve().parent.parent / "docs" / "assets" / "openbot-architecture.svg" @@ -480,8 +366,8 @@ def main() -> int: produce an untracked copy the README cannot see. """ OUT.parent.mkdir(parents=True, exist_ok=True) - OUT.write_text("\n".join(lines) + "\n", encoding="utf-8") - print(f"wrote {OUT} ({len(lines)} lines)") + OUT.write_text(canvas.render(), encoding="utf-8") + print(f"wrote {OUT} ({len(canvas.lines)} lines)") return 0 diff --git a/scripts/build_eval_architecture_diagram.py b/scripts/build_eval_architecture_diagram.py new file mode 100644 index 0000000..4586a8a --- /dev/null +++ b/scripts/build_eval_architecture_diagram.py @@ -0,0 +1,322 @@ +#!/usr/bin/env python3 +"""Regenerate ``docs/assets/openbot-eval-architecture.svg`` (embedded in README.md). + +Shows the two eval lanes described by ``docs/prd/openbot-eval-prd.md`` and +``docs/prd/openbot-eval-suites.md``: + + * OFFLINE (v0.1, built) — benchmark datasets driven through the *production* + agent path via ``openbot.evaluation``, scored, then exported to LangSmith. + * ONLINE (v0.3, planned) — signal harvested from real production runs. + +plus a coverage strip for the four product workflows in +``openbot/domain/workflows.py`` (triage / review / fix / chat). + +Solid borders are built today; dashed borders are specified in the eval PRD +but not implemented. Edit the tables below and re-run: + + python3 scripts/build_eval_architecture_diagram.py + +Layout follows the fireworks-tech-graph architecture-diagram rules (Style 2, +"Dark Terminal") under the ``showcase`` composition profile. The shared +emitter lives in ``scripts/_diagram_svg.py``. +""" + +from __future__ import annotations + +from pathlib import Path + +from _diagram_svg import FLOW, LLM, MUTE, STATE, SUB, THEMES, TXT, Canvas + +W, H = 1060, 1012 + +# ── grid ─────────────────────────────────────────────────────────────────── +LANE_W = 432 +A_X, B_X = 58, 570 # lane node left edges +A_CX, B_CX = A_X + LANE_W // 2, B_X + LANE_W // 2 +ROWS = [(132, 66), (238, 74), (352, 74), (466, 80), (586, 74)] +LS_Y, LS_H = 722, 66 +CARD_Y, CARD_H, CARD_W = 858, 70, 207 +CARD_X = [56, 303, 550, 797] + +# id: (x, y, w, h, theme, dashed) +NODES: dict[str, tuple[int, int, int, int, str, bool]] = {} +for _i, (_y, _h) in enumerate(ROWS, start=1): + NODES[f"a{_i}"] = (A_X, _y, LANE_W, _h, "", False) + NODES[f"b{_i}"] = (B_X, _y, LANE_W, _h, "", False) + +_A_THEMES = ["store", "external", "external", "agent", "compute"] +_B_THEMES = [ + ("external", False), + ("store", False), + ("planned", True), + ("agent", True), + ("sandbox", True), +] +for _i, _theme in enumerate(_A_THEMES, start=1): + _x, _y, _w, _h, _, _ = NODES[f"a{_i}"] + NODES[f"a{_i}"] = (_x, _y, _w, _h, _theme, False) +for _i, (_theme, _dashed) in enumerate(_B_THEMES, start=1): + _x, _y, _w, _h, _, _ = NODES[f"b{_i}"] + NODES[f"b{_i}"] = (_x, _y, _w, _h, _theme, _dashed) + +NODES["ls"] = (36, LS_Y, 988, LS_H, "store", False) +for _i, _cx in enumerate(CARD_X, start=1): + NODES[f"c{_i}"] = (_cx, CARD_Y, CARD_W, CARD_H, "eval" if _i == 1 else "external", _i == 1) + +# id, x, y, w, h, label, label x, label y, dashed (dashed == not built yet) +CONTAINERS = [ + ("cA", 36, 96, 476, 586, "OFFLINE · v0.1 · BUILT", 50, 116, False), + ("cB", 548, 96, 476, 586, "ONLINE · v0.3 · PLANNED", 562, 116, True), + ("cC", 36, 822, 988, 126, "COVERAGE — THE FOUR PRODUCT WORKFLOWS", 50, 842, False), +] + +# id, source, target, d, colour, dash, label, label x, label y +EDGES = [ + ("ea1", "a1", "a2", f"M {A_CX},198 L {A_CX},238", FLOW, None, "sample", 286, 221), + ("ea2", "a2", "a3", f"M {A_CX},312 L {A_CX},352", FLOW, None, "solve()", 286, 335), + ("ea3", "a3", "a4", f"M {A_CX},426 L {A_CX},466", FLOW, None, "run_*_sample", 286, 449), + ("ea4", "a4", "a5", f"M {A_CX},546 L {A_CX},586", FLOW, None, "domain result", 286, 569), + ("ea5", "a5", "ls", f"M {A_CX},660 L {A_CX},{LS_Y}", STATE, None, "experiment row", 286, 697), + ("eb1", "b1", "b2", f"M {B_CX},198 L {B_CX},238", LLM, "5,4", "traced", 798, 221), + ("eb2", "b2", "b3", f"M {B_CX},312 L {B_CX},352", LLM, "5,4", "run id", 798, 335), + ("eb3", "b3", "b4", f"M {B_CX},426 L {B_CX},466", LLM, "5,4", "sampled", 798, 449), + ("eb4", "b4", "b5", f"M {B_CX},546 L {B_CX},586", LLM, "5,4", "scored", 798, 569), + ("eb5", "b5", "ls", f"M {B_CX},660 L {B_CX},{LS_Y}", STATE, None, "trend rows", 798, 697), +] + +# node id -> (badge, text x or None for centred, [(dy, size, weight, fill, text)]) +BODY = { + "a1": ( + "1", + A_X + 40, + [ + (22, 12, 700, TXT, "benchmark datasets"), + (40, 10, 400, SUB, "LangSmith mirrors published by make -C evals data-*"), + (56, 10, 400, SUB, "Martian CRB · SWE-bench V · SWE-QA-Pro · SWT-Bench"), + ], + ), + "a2": ( + "2", + A_X + 40, + [ + (22, 12, 700, TXT, "Inspect AI task — one per benchmark"), + (40, 10, 400, SUB, "evals/tasks/: review_martian · fix_swe_bench ·"), + (55, 10, 400, SUB, "chat_swe_qa · test_swt_bench"), + (70, 10, 400, MUTE, "dataset + solver + scorers wired per task"), + ], + ), + "a3": ( + "3", + A_X + 40, + [ + (22, 12, 700, TXT, "thin solver → openbot.evaluation"), + (40, 10, 400, SUB, "evals/solvers/*.py call runner.run_*_sample and adapt"), + (55, 10, 400, SUB, "the domain result into scorer / prediction shape"), + (70, 10, 400, MUTE, "forbidden here: eval-only agents, sandboxes, tools"), + ], + ), + "a4": ( + "4", + A_X + 40, + [ + (22, 12, 700, TXT, "production agent + real sandbox"), + (40, 10, 400, SUB, "the facade opens the prod sandbox, clones at base_sha,"), + (55, 10, 400, SUB, "then calls DeepAgents review / fix / chat / repro —"), + (70, 10, 400, SUB, "the same code the worker runs in production"), + ], + ), + "a5": ( + "5", + A_X + 40, + [ + (22, 12, 700, TXT, "scorers + prediction export"), + (40, 10, 400, SUB, "review: LLM judge + overlap · chat: SWE-QA-Pro judge"), + (55, 10, 400, SUB, "fix / test: JSONL → official harness (Modal / docker)"), + (70, 10, 400, MUTE, "→ grade writeback as pass@1 feedback"), + ], + ), + "b1": ( + "1", + B_X + 40, + [ + (22, 12, 700, TXT, "production workflow runs"), + (40, 10, 400, SUB, "triage · review · fix · chat on real repositories"), + (56, 10, 400, MUTE, "live traffic, no benchmark dataset"), + ], + ), + "b2": ( + "2", + B_X + 40, + [ + (22, 12, 700, TXT, "auto-instrumented traces"), + (40, 10, 400, SUB, "LANGSMITH_TRACING on the web and worker processes;"), + (55, 10, 400, SUB, "every agent run already lands in LangSmith today"), + (70, 10, 400, MUTE, "this half of the online lane exists now"), + ], + ), + "b3": ( + "3", + B_X + 40, + [ + (22, 12, 700, TXT, "outcome signals from GitHub"), + (40, 10, 400, SUB, "was the review comment resolved or acted on; did the"), + (55, 10, 400, SUB, "chat answer need a follow-up; was a triage label fixed"), + (70, 10, 400, MUTE, "collector not built yet"), + ], + ), + "b4": ( + "4", + B_X + 40, + [ + (22, 12, 700, TXT, "online eval + annotation queue"), + (40, 10, 400, SUB, "sampled LLM judge over live runs, plus human labels"), + (55, 10, 400, SUB, "from the LangSmith annotation queue"), + (70, 10, 400, MUTE, "suites: triage_online · review_online · chat_online"), + ], + ), + "b5": ( + "5", + B_X + 40, + [ + (22, 12, 700, TXT, "30-day trend metrics"), + (40, 10, 400, SUB, "action_rate · resolved_rate · correct_30d ·"), + (55, 10, 400, SUB, "macro_f1_30d · followup_rate · pass@1_90d"), + (70, 10, 400, MUTE, "trend observation only — never blocks a merge"), + ], + ), + "ls": ( + None, + None, + [ + (24, 13, 700, TXT, "LangSmith — one substrate for both lanes"), + ( + 43, + 11, + 400, + SUB, + "datasets · experiment rows · per-sample feedback · traces · annotation queue", + ), + ( + 59, + 10, + 400, + MUTE, + "every row carries dataset_version · solver_family=openbot_agent · capability ·" + " openbot_git_sha · mode", + ), + ], + ), + "c1": ( + None, + CARD_X[0] + 14, + [ + (20, 12, 700, TXT, "triage"), + (36, 10, 400, SUB, "off · triage_gitbugs (v0.2)"), + (51, 10, 400, SUB, "on · macro_f1_30d (v0.3)"), + (66, 10, 400, MUTE, "no solver in evals/ yet"), + ], + ), + "c2": ( + None, + CARD_X[1] + 14, + [ + (20, 12, 700, TXT, "review"), + (36, 10, 400, SUB, "off · review_martian — mean F1"), + (51, 10, 400, SUB, "on · action / resolved rate"), + (66, 10, 400, MUTE, "10% F1 drop blocks merge"), + ], + ), + "c3": ( + None, + CARD_X[2] + 14, + [ + (20, 12, 700, TXT, "fix"), + (36, 10, 400, SUB, "off · fix_swe_bench — pass@1"), + (51, 10, 400, SUB, "on · fix_swe_bench_live 90d"), + (66, 10, 400, MUTE, "rolling benchmark, not prod"), + ], + ), + "c4": ( + None, + CARD_X[3] + 14, + [ + (20, 12, 700, TXT, "chat"), + (36, 10, 400, SUB, "off · chat_swe_qa — 5-dim"), + (51, 10, 400, SUB, "on · correct_30d · followup"), + (66, 10, 400, MUTE, "judge must exist; warn only"), + ], + ), +} + +LEGEND = [ + (FLOW, None, False, "offline path — built"), + (LLM, "5,4", False, "online path — planned"), + (STATE, None, False, "writes to LangSmith"), +] + +canvas = Canvas( + W, + H, + title="OpenBot eval architecture", + aria_label=( + "OpenBot eval architecture: an offline lane runs benchmark datasets through Inspect AI" + " tasks, thin solvers and the openbot.evaluation facade into the production agent and" + " sandbox, then scores and exports predictions; a planned online lane harvests traces and" + " GitHub outcome signals from production runs into online eval and 30-day trend metrics." + " Both lanes write to LangSmith. A coverage strip lists the four product workflows:" + " triage, review, fix and chat." + ), +) + +canvas.title_block( + "OpenBot — eval architecture", + "two lanes over one LangSmith substrate: offline benchmarks driven through the production" + " agent, online signal from production runs", + "v0.1 built · v0.3 planned", +) + +for cid, cx, cy, cw, ch, label, lx, ly, dashed in CONTAINERS: + canvas.container(cid, cx, cy, cw, ch, label, lx, ly, dashed=dashed) + +for eid, src, tgt, d, colour, dash, *_ in EDGES: + canvas.edge(eid, src, tgt, d, colour, dash=dash) + +for nid, (x, y, w, h, theme, dashed) in NODES.items(): + canvas.node(nid, x, y, w, h, theme, dashed=dashed) + +for nid, (badge, text_x, rows) in BODY.items(): + x, y, w, h, theme, _ = NODES[nid] + _, stroke = THEMES[theme] + if badge: + canvas.badge(x + 20, y + rows[0][0] - 5, badge, stroke) + for dy, size, weight, fill_c, txt in rows: + if text_x is None: + canvas.text(x + w // 2, y + dy, size, weight, fill_c, txt, anchor="middle") + else: + canvas.text(text_x, y + dy, size, weight, fill_c, txt) + +for eid, _src, _tgt, _d, colour, _dash, label, lx, ly in EDGES: + canvas.edge_label(eid, lx, ly, 10, colour, label) + +canvas.legend( + LEGEND, + y=974, + footer="solid = built today · dashed = specified in the eval PRD but not built ·" + " test_swt_bench is a fifth surface, not a workflow", + footer_y=998, +) +canvas.add("") + +OUT = Path(__file__).resolve().parent.parent / "docs" / "assets" / "openbot-eval-architecture.svg" + + +def main() -> int: + """Write the diagram to its one tracked location (see the sibling script).""" + OUT.parent.mkdir(parents=True, exist_ok=True) + OUT.write_text(canvas.render(), encoding="utf-8") + print(f"wrote {OUT} ({len(canvas.lines)} lines)") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())