From faf5a493f1f808363d8836b6e9da21d9a9aedd94 Mon Sep 17 00:00:00 2001 From: Michael Mullarkey Date: Sun, 13 Sep 2026 16:34:05 -0600 Subject: [PATCH 1/4] feat: blendtutor-course skill for authoring courses, lessons, evals, and Quarto/site output A project skill under .claude/skills/ walks an author from source material to a published course: scaffold with init/new, write lessons whose checks, solution, hints, and success_criteria work in webR/Pyodide, write a four-case minimal eval (canonical, alternative, near-miss, check-caught), verify solution and eval cases against the checks locally for R and Python without paid calls, then export a Quarto snippet/page or build a static site. .gitignore keeps local .claude state ignored while sharing .claude/skills/. --- .claude/skills/blendtutor-course/SKILL.md | 230 ++++++++++++++++++++++ .gitignore | 4 +- 2 files changed, 233 insertions(+), 1 deletion(-) create mode 100644 .claude/skills/blendtutor-course/SKILL.md diff --git a/.claude/skills/blendtutor-course/SKILL.md b/.claude/skills/blendtutor-course/SKILL.md new file mode 100644 index 0000000..eeb8db8 --- /dev/null +++ b/.claude/skills/blendtutor-course/SKILL.md @@ -0,0 +1,230 @@ +--- +name: blendtutor-course +description: Author a blendtutor course end to end — scaffold the course, write R or Python lessons with checks, solutions, hints, and success criteria, write a minimal eval suite, verify everything locally, then produce a Quarto snippet/page or a static browser site. Use when someone wants to create or extend a blendtutor course, turn a chapter or tutorial into interactive exercises, write eval cases for a lesson, or publish lessons to Quarto or GitHub Pages. +argument-hint: [r|python] [quarto|site] +--- + +# blendtutor course authoring + +Turn source material into blendtutor lessons that grade correctly and render well. +One lesson = one exercise = one `lessons/.yaml` plus its `lessons/eval_.yaml`. + +## 0. Preconditions + +- `blendtutor --version` is **0.2.0 or newer** (older versions drop `packages`, + `gotchas`, and `success_criteria` from Quarto exports). Install from + . +- R lessons: `Rscript` on PATH. Python lessons: `uv` on PATH. +- Quarto output: Quarto >= 1.4. +- `run` and `eval` make **paid LLM calls** and need `FIREWORKS_API_KEY` (or + `ANTHROPIC_API_KEY`) in the environment. Never write a key into a file you commit. + +## 1. Gather + +Ask only for what you cannot infer: + +- **Source material**: read the chapter or tutorial; pick the one skill each exercise practices. +- **Language**: `r` or `python`, per lesson. +- **Output**: Quarto (snippet into an existing book/page, or a standalone page) and/or a static site. +- **Where the course lives**: an existing directory holding `blendtutor.toml`, or a new one. + +## 2. Scaffold + +```bash +blendtutor init # only if no blendtutor.toml exists yet +cd +blendtutor new lesson --lang # writes lessons/.yaml + lessons/eval_.yaml +``` + +`new` registers the lesson in `blendtutor.toml`. Paths are one argument: +`lessons/.yaml`, never `lessons .yaml`. + +## 3. Write the lesson + +Replace the scaffold's hello-world content. Keep every field short. + +```yaml +lesson_name: "" +language: Python # or R +description: "" +textbook_reference: "" # optional + +exercise: + type: "function_writing" + prompt: | + <2-3 sentences. Name every variable or function the checks rely on.> + code_template: | + + solution: | + + hints: | + - + success_criteria: | + - + llm_evaluation_prompt: | + You are grading a beginner exercise on . + + The student submitted this code: + {student_code} + + + Call respond_with_feedback with two or three encouraging sentences. + +checks: # top level, not inside exercise + - "assert " # R: "stopifnot()" + +packages: # only if needed + - pandas +``` + +Rules that keep lessons working in the browser: + +- **Inline the data.** Build a small data frame in `code_template` instead of + loading a dataset package; webR and Pyodide may not have it. Choose values + whose correct answer is easy to assert (for example, averages like 48.0 and 39.0). +- **Checks assert results, not source text.** They run after the learner's code + in the same session and fail if they raise. Any lesson with at least one check + gets a Check button. +- **No checks for prose-like exercises** (pseudocode, explanations): there is + nothing to assert. The grader, driven by `success_criteria`, does the work. +- **`success_criteria` reaches the grader** (0.2.0+). Put the lesson's real + point there, especially what a check cannot verify (structure, style, naming). +- **`hints` and `gotchas` are bullet lists**, or `validate` rejects them. +- **`packages`** names cannot contain quotes, commas, or spaces. + +## 4. Write the minimal eval + +`lessons/eval_.yaml`, next to the lesson. Four cases, each with a comment: + +```yaml +# Eval suite for .yaml: 2 correct, 2 incorrect. +cases: + # Correct: the canonical solution + - submission: |- + + expected: correct + # Correct: different names or methods, same intent + - submission: |- + + expected: correct + # Incorrect: near-miss that passes the checks but misses the lesson's point + - submission: |- + + expected: incorrect + # Incorrect: a realistic mistake the checks catch + - submission: |- + + expected: incorrect +``` + +The near-miss case is what measures the grading prompt; do not skip it. For +lessons without checks, make both incorrect cases content mistakes (a missing +step, the wrong operation). + +## 5. Verify locally (no paid calls) + +```bash +blendtutor validate lessons/.yaml +blendtutor list . +``` + +Then run the solution and every eval case against the checks. Python: + +```bash +uv run --no-project --with pyyaml --with python - <<'PY' +import contextlib, io, yaml +lesson = yaml.safe_load(open("lessons/.yaml")) +cases = yaml.safe_load(open("lessons/eval_.yaml"))["cases"] +def run(label, code): + env = {} + with contextlib.redirect_stdout(io.StringIO()): + exec(code, env) + results = [] + for check in lesson.get("checks", []): + try: + exec(check, env); results.append("pass") + except Exception as e: + results.append(f"fail({type(e).__name__})") + print(label, results) +run("solution", lesson["exercise"]["solution"]) +for i, case in enumerate(cases, 1): + run(f"case {i} ({case['expected']})", case["submission"]) +PY +``` + +R (pass the lesson path; the eval file is found next to it): + +```bash +uv run --no-project --with pyyaml python - lessons/.yaml <<'PY' +import pathlib, subprocess, sys, tempfile, yaml +lesson_path = pathlib.Path(sys.argv[1]) +lesson = yaml.safe_load(lesson_path.read_text()) +cases = yaml.safe_load(lesson_path.with_name("eval_" + lesson_path.name).read_text())["cases"] +def run(label, code): + checks = "\n".join( + f'r <- c(r, tryCatch({{ {c} ; "pass" }}, error = function(e) "fail"))' for c in lesson.get("checks", []) + ) + script = f"r <- character()\ninvisible(capture.output({{\n{code}\n}}))\n{checks}\ncat(r)\n" + with tempfile.NamedTemporaryFile("w", suffix=".R", delete=False) as f: + f.write(script) + out = subprocess.run(["Rscript", f.name], capture_output=True, text=True) + print(label, out.stdout.strip() or f"submission error: {out.stderr.strip().splitlines()[-1:]}") +run("solution", lesson["exercise"]["solution"]) +for i, case in enumerate(cases, 1): + run(f"case {i} ({case['expected']})", case["submission"]) +PY +``` + +Expect: solution passes; correct cases pass; the near-miss passes (only the +grader can reject it); the checks-catch case fails. Fix the lesson if not. + +Only then, **after confirming with the user** (paid calls): + +```bash +blendtutor eval lessons/.yaml # grader accuracy against expected verdicts +blendtutor eval lessons/.yaml --write-report # also saves eval-report.json for the site +``` + +If the grader misjudges a case, tighten `success_criteria` and the +`llm_evaluation_prompt`, then re-run `eval`. + +## 6. Publish + +### Quarto + +```bash +blendtutor export-quarto lessons/.yaml # snippet: paste into an existing .qmd +blendtutor export-quarto --document lessons/.yaml # standalone page with front matter +blendtutor export-quarto --key-page > api-key.qmd # optional API key page +``` + +Project setup, once: + +1. Run `quarto add mcmullarkey/blendtutor` **from the folder that contains + `_quarto.yml`** (or the standalone `.qmd`). Installed anywhere else, the + filter never loads and exercises render as plain text. +2. Enable the filter with `filters: [mcmullarkey/blendtutor]` in `_quarto.yml` + **or** in the page's front matter, not both. `--document` pages already declare it. +3. Commit `_extensions/` so CI renders (for example, a GitHub Pages workflow). +4. Preview with `quarto preview`. Opening HTML via `file://` blocks the widget's JavaScript. + +R runs in `type: book` projects without cross-origin isolation, on webR's +slower channel; `coi: true` speeds up standalone pages only. + +### Static site + +```bash +blendtutor build . --target pyodide -o site # Python course +blendtutor build . --target webr -o site # R course +``` + +One target per build, so keep R and Python lessons in separate courses. The +site deploys to GitHub Pages as-is. `--password` encrypts it; `--embed-key` +(requires `--password`) puts a real API key inside the encrypted site, where +anyone with the password can use it. Mention that risk before suggesting it. + +## 7. Report + +Tell the user which files you created, the local check results per case, whether +paid evals ran (and their accuracy), and the exact command or snippet for their +chosen output. diff --git a/.gitignore b/.gitignore index cdf1aba..c0e26f4 100644 --- a/.gitignore +++ b/.gitignore @@ -5,7 +5,9 @@ # output under docs/evals/ (deliberately committed) nor Rust eval fixtures # under crates/*/tests/fixtures/evals/. **/.smevals/ -.claude/ +# Local Claude state stays ignored; project skills under .claude/skills/ are shared. +.claude/* +!.claude/skills/ rv/library/ # Rust From 7d952001cca617b6720a23c429c4e61f428c0574 Mon Sep 17 00:00:00 2001 From: Michael Mullarkey Date: Sun, 13 Sep 2026 16:35:58 -0600 Subject: [PATCH 2/4] fix(review): R check script cleans up temp files; template shows optional gotchas --- .claude/skills/blendtutor-course/SKILL.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.claude/skills/blendtutor-course/SKILL.md b/.claude/skills/blendtutor-course/SKILL.md index eeb8db8..b56277e 100644 --- a/.claude/skills/blendtutor-course/SKILL.md +++ b/.claude/skills/blendtutor-course/SKILL.md @@ -59,6 +59,8 @@ exercise: hints: | - + gotchas: | # optional: common mistakes, also bullets + - success_criteria: | - llm_evaluation_prompt: | @@ -168,6 +170,7 @@ def run(label, code): with tempfile.NamedTemporaryFile("w", suffix=".R", delete=False) as f: f.write(script) out = subprocess.run(["Rscript", f.name], capture_output=True, text=True) + pathlib.Path(f.name).unlink(missing_ok=True) print(label, out.stdout.strip() or f"submission error: {out.stderr.strip().splitlines()[-1:]}") run("solution", lesson["exercise"]["solution"]) for i, case in enumerate(cases, 1): From bb02dc8a6b5cdb347718f2011f54167d6b9cc63b Mon Sep 17 00:00:00 2001 From: Michael Mullarkey Date: Sun, 13 Sep 2026 16:48:15 -0600 Subject: [PATCH 3/4] docs: move Quarto extension docs into the whole-game chapter, drop Creating Lessons, point README at the skill README keeps install, API key, the command overview, and Pages deploy, adds a Claude Code skill quick reference, and links each Quarto topic to its whole-game section (88 lines). The whole-game chapter now carries the Quarto extension quick start, export, auto-bootstrap opt-out, COI, demo book, and BYOK prose plus the eval-report path scrub note. Creating Lessons duplicated the whole game and is removed from the book. The docs-contract scripts read the moved prose from whole-game.md; README keeps its line ceiling and a pin on the demo-book link. --- .github/workflows/docs.yml | 2 +- README.md | 96 +++---------- docs/book/src/SUMMARY.md | 1 - docs/book/src/creating-lessons.md | 164 ---------------------- docs/book/src/whole-game.md | 126 ++++++++++++++--- scripts/check-docs.sh | 8 +- scripts/tests/test_demo_docs.sh | 68 +++++---- scripts/tests/test_quarto_distribution.sh | 64 +++++---- 8 files changed, 196 insertions(+), 333 deletions(-) delete mode 100644 docs/book/src/creating-lessons.md diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 92fdeac..b64e67d 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -112,7 +112,7 @@ jobs: run: | if [ -d docs/evals ]; then if rg -l '/Users/' docs/evals/ | grep -q .; then - echo "docs.yml: /Users/ leak in docs/evals/ (scrub per creating-lessons.md Step 9)" >&2 + echo "docs.yml: /Users/ leak in docs/evals/ (scrub per whole-game.md Eval report)" >&2 exit 1 fi rm -rf docs/book/book/evals diff --git a/README.md b/README.md index 0ccd3b3..2b2bd5d 100644 --- a/README.md +++ b/README.md @@ -57,93 +57,31 @@ don't need it). Two example courses are deployed alongside the docs: - **[R example site (webR)](https://mcmullarkey.github.io/blendtutor/examples/r/)** - **[Python example site (Pyodide)](https://mcmullarkey.github.io/blendtutor/examples/python/)** +## Author a course with Claude Code + +This repo ships a Claude Code skill, +[`blendtutor-course`](.claude/skills/blendtutor-course/SKILL.md). In a clone, +run `/blendtutor-course ` (or ask Claude to build a +course): it scaffolds lessons with checks and solutions, writes a minimal eval +suite, verifies the checks locally, and exports a Quarto snippet or builds a site. + ## Quarto extension -blendtutor also ships as a [Quarto](https://quarto.org) extension for -interactive coding exercises in `.qmd` documents — in-browser editor, instant -checks, solution reveal, AI hints, all static HTML. Requires **Quarto >= 1.4**: +Interactive exercises in `.qmd` documents (Quarto >= 1.4, extension version +0.2.0). Run this from the folder that contains `_quarto.yml`: ```bash quarto add mcmullarkey/blendtutor ``` -Installs to `_extensions/mcmullarkey/blendtutor/` (version 0.2.0). **Run it from the -folder that contains `_quarto.yml`** (or the `.qmd`): Quarto only discovers `_extensions/` -there, so one installed a directory up never loads; assets are install-path-independent. - -#### Quick start (zero hand-written bootstrap) - -A complete copy-paste document — zero hand-written bootstrap. Filter by name, -`.blendtutor` div, render: - -````markdown ---- -title: "My exercises" -filters: [mcmullarkey/blendtutor] ---- - -::: {.blendtutor language="r"} -Write a function `add(a, b)` that returns the sum. - -```r -add <- function(a, b) { ___ } -``` -::: -```` - -Render, open in a browser — interactive immediately. Grade submissions with a -`{.r .checks}` block (`stopifnot(add(1, 2) == 3)`); Python: same div, `language="python"`. - -Optional blocks inside the div add a `{.r .solution}`, `::: {.hints}` / `::: {.gotchas}` -bullets, and a `::: {.success-criteria}` rubric for AI feedback; `packages="dplyr"` on the -div preloads packages. `blendtutor export-quarto lesson.yaml` writes the div from a lesson -(`--document` for a full page, `--key-page` for the API key page). - -#### Auto-bootstrap opt-out - -The filter auto-bootstraps by default; to wire up the runtime yourself, set -`bt-auto-bootstrap: false` in the YAML header. To keep it but disable the -auto-mounted AI feedback, set `bt-feedback: false` — see -[BYOK](#byok-bring-your-own-key). - -### Cross-origin isolation (COI) - -webR runs faster with `SharedArrayBuffer`, which needs cross-origin isolation -(COOP/COEP). Opt in with `coi: true` (page YAML header) or `coi="true"` (any div); -the filter injects a service-worker shim. Pyodide-only pages do not need COI. - -> **Book-mode limitation:** COI does not function in Quarto `type: book` -> projects — the shim's scope cannot cover the book's `_output/` pages, so webR -> uses its slower non-isolated channel ([ADR-0015](docs/adr/0015-opt-in-coi-cross-origin.md)). - -### Demo book - -A complete demo book with R and Python exercises lives in -[`demo-book/`](demo-book/), rendered live at - (rebuild locally with -`cd demo-book && quarto render`). It is a Quarto `type: book` project, so -COI does not take effect (limitation above). Python exercises are fully interactive -and every page ships a static fallback. R exercises run in the book too, on webR's slower -fallback channel; the CLI-built [example sites](#deploy-to-github-pages) add isolation, -R exercises run interactively via webR there. Over `file://` you get static exercise content only; serve over HTTP: - -```bash -cd demo-book/_output && python3 -m http.server 8000 -``` +[The whole game](https://mcmullarkey.github.io/blendtutor/whole-game.html#quarto-extension) covers the rest: -## BYOK (Bring Your Own Key) - -Browser feedback uses the learner's own API key — no server-side key. Feedback -is **auto-mounted**: the injected bootstrap imports `exercise-feedback.js` and -calls `mountAllFeedback(registry)` after the runtime starts. The key is entered -once on the API key page (the demo book ships one) and shared via `localStorage` -— readable by any JavaScript on the page's origin, so never reuse a critical -key; it is sent only to `api.fireworks.ai`. BYOK is Fireworks-only (pinned model -`accounts/fireworks/models/deepseek-v4-flash-0731`); the CLI supports other -providers (see [API key](#api-key)). Serve over HTTP — `file://` breaks -`localStorage` sharing and blocks ES modules, so feedback never mounts. -Self-hosted CSP: add `connect-src https://api.fireworks.ai` (Pages cannot set -CSP headers; the shim covers only COOP/COEP). +- [Quick start](https://mcmullarkey.github.io/blendtutor/whole-game.html#quick-start-zero-hand-written-bootstrap) +- [Export a lesson](https://mcmullarkey.github.io/blendtutor/whole-game.html#export-a-lesson) with `blendtutor export-quarto` +- [Auto-bootstrap opt-out](https://mcmullarkey.github.io/blendtutor/whole-game.html#auto-bootstrap-opt-out) +- [Cross-origin isolation](https://mcmullarkey.github.io/blendtutor/whole-game.html#cross-origin-isolation-coi) and R in book projects +- [Demo book](https://mcmullarkey.github.io/blendtutor/whole-game.html#demo-book) +- [BYOK feedback](https://mcmullarkey.github.io/blendtutor/whole-game.html#byok-bring-your-own-key) ## License diff --git a/docs/book/src/SUMMARY.md b/docs/book/src/SUMMARY.md index f8d8364..c350764 100644 --- a/docs/book/src/SUMMARY.md +++ b/docs/book/src/SUMMARY.md @@ -3,7 +3,6 @@ [Introduction](./introduction.md) - [The whole game](./whole-game.md) -- [Creating Lessons](./creating-lessons.md) - [Architecture](./architecture.md) - [Example sites](./examples.md) - [API reference](./api-reference.md) diff --git a/docs/book/src/creating-lessons.md b/docs/book/src/creating-lessons.md deleted file mode 100644 index 1f18db7..0000000 --- a/docs/book/src/creating-lessons.md +++ /dev/null @@ -1,164 +0,0 @@ -# Creating Lessons - -The command-first guide to authoring a lesson: scaffold a course, author R and Python lessons, validate, grade, and build a deployable site. For a complete end-to-end walkthrough, see [The whole game](./whole-game.md); the shipped reference courses (`examples/write-less-code-r/`, five R lessons, webR; and `examples/write-less-code-python/`, five Python lessons, Pyodide) are the canonical worked examples ([Example sites](./examples.md)). - -## Prerequisites - -```bash -git clone https://github.com/mcmullarkey/blendtutor.git -cd blendtutor -cargo install --path crates/cli -export FIREWORKS_API_KEY=fw_... -``` - -`init`, `new`, `validate`, and `build` need no key; only `run` and `eval` call the LLM provider. - -## Step 1 — Scaffold a course - -```bash -blendtutor init my-stats-course -``` - -Creates `my-stats-course/` with a `blendtutor.toml` manifest, a starter lesson, a matching eval suite, and a `.gitignore`. Each `[[lessons]]` entry maps a stable slug (`id`) to a lesson `path` relative to the manifest, in the order lessons appear in the built site; `new` appends entries automatically (`Manifest`/`ManifestEntry` in the [API reference](./api/blendtutor_core/index.html) define the fields). - -## Step 2 — Add an R lesson - -```bash -blendtutor new lesson --lang r seed-data -``` - -Writes `lessons/seed-data.yaml`; `--lang` selects the runtime target (`r` or `python`). Abridged from `examples/write-less-code-r/01_seed_data.yaml`: - -```yaml -lesson_name: "Seed Data" # shown in the browser site -language: R # or Python; selects the runtime -exercise: - type: "function_writing" - prompt: | # instructions the learner sees - Create a data frame called `survey_data` with 5 respondents and 6 stress items (stress_1 through stress_6) measured on a 1-5 scale. - code_template: | # starter code pre-filled in the editor - survey_data <- data.frame(respondent_id = 1:5, ...) - solution: | # reference solution (eval + site self-check) - survey_data <- data.frame(respondent_id = 1:5, stress_1 = c(3, ...), ...) - llm_evaluation_prompt: | # grading prompt; must contain {student_code} - ...Student submitted this code: {student_code}... -checks: # must evaluate without error, before the LLM verdict - - "stopifnot(identical(dim(survey_data), c(5L, 7L)))" - - "stopifnot(identical(survey_data$stress_6, c(1, 2, 3, 4, 5)))" -``` - -`description`, `example_usage`, and `success_criteria` round out the file; the [`Lesson` and `Exercise` structs](./api/blendtutor_core/index.html) define every field. - -## Step 3 — Add a Python lesson - -```bash -blendtutor new lesson --lang python tally -``` - -Same shape with `language: Python`; an optional `packages` list (e.g. `packages: [pandas]`) is loaded by the Pyodide runtime in the browser. Full example: `examples/write-less-code-python/01_seed_data.yaml`. - -## Step 4 — Register lessons in the manifest - -`new` appends manifest entries automatically; hand-added lesson files need a matching `[[lessons]]` block. `examples/write-less-code-r/blendtutor.toml` shows a complete five-lesson manifest. - -## Step 5 — Validate - -```bash -blendtutor validate lessons/seed-data.yaml -blendtutor validate lessons/seed-data.yaml --format json -``` - -Nonzero exit when the lesson is invalid, so it drops cleanly into CI. - -## Step 6 — Run a submission - -```bash -blendtutor run lessons/seed-data.yaml --code submission.R -echo 'survey_data <- data.frame(respondent_id = 1:5)' | blendtutor run lessons/seed-data.yaml # --code omitted: reads stdin -``` - -Executes the `checks`, then asks the LLM for a verdict; the exit code reflects the verdict. `--format json` for a structured report. - -## Step 7 — Write an eval suite - -Each lesson pairs with a sibling `eval_.yaml` — `new` scaffolds a minimal one-case starter suite next to the lesson (edit it in place; `eval` discovers the suite by the sibling convention), and the starter course's `eval_lesson_hello.yaml` shows the fuller two-case shape — containing sample submissions and expected verdicts (abridged from `examples/write-less-code-r/eval_01_seed_data.yaml`): - -```yaml -cases: - - submission: |- - survey_data <- data.frame(respondent_id = 1:5, stress_1 = c(3, 4, 5, 2, 1), ...) - expected: correct - - submission: |- - survey_data <- data.frame(respondent_id = 1:5, stress_1 = c(3, 4, 5, 2, 1)) - expected: incorrect # near-miss: runs cleanly, missing stress_6 -``` - -Include **near-miss** cases — submissions that run cleanly but are subtly wrong; they measure whether the grading prompt catches realistic mistakes, not just syntax errors. The [`EvalSuite` schema](./api/blendtutor_core/index.html) documents the fields. - -## Step 8 — Score the grading prompt - -```bash -blendtutor eval lessons/seed-data.yaml -``` - -Replays the eval cases through the run pipeline and reports how often the grader's verdict matches the expected label — regression-test grading accuracy before shipping. `--case N` for one case, `--format json` for structured output. Real paid calls: run against the provider your deployed site will use. - -To persist the score for `build` to fold into the site's eval-results page, add `--write-report`: it writes the full-shape `eval-report.json` next to the course's `blendtutor.toml` (found from the lesson's directory, wherever you run from), overwriting a previous report with a warning. `--case N` and `--write-report` are mutually exclusive — a single-case report would render as the course-level accuracy. - -## Step 9 — Generate the eval report - -```bash -blendtutor eval-report lessons/seed-data.yaml -``` - -Grades every case with polarity AND a real paid LLM-judge call (driving the pinned [`smevals`](https://pypi.org/project/smevals/) runner via [`uv`](https://docs.astral.sh/uv/), which must be on PATH), producing `.smevals/` (ephemeral, never commit) and `docs/evals//` (the committed static report, published at `/evals//`). Exits 0 as long as the run recorded its cases — a low score is evidence, not a failure. It fails only when a stage produced nothing usable. Same API key as `eval`. - -If you recorded from a git worktree, scrub the checkout prefix before committing — `scripts/check-docs.sh` fails any `/Users/` path under `docs/evals/`: - -```bash -# find -exec, not docs/evals/**: globstar is absent on macOS bash 3.2 — -# the glob would match nothing and the scrub would silently no-op -find docs/evals \( -name 'eval.json' -o -name 'run.yaml' \) -exec \ - perl -pi -e 's{/Users/[^/]*/portfolio/(?:worktree-|blendtutor-)[^/]*/}{}g' {} + -git add docs/evals -git commit -m "eval report: lessons/seed-data.yaml" -``` - -## Step 10 — Build a browser site - -```bash -blendtutor build my-stats-course --target webr -o site # R course -blendtutor build my-stats-course --target pyodide -o site # Python course -``` - -Emits a fully static site — `index.html`, per-lesson JSON, the in-browser runtime, and an embedded eval-results page when the course carries `eval-report.json` — deployable to GitHub Pages as-is. webR needs cross-origin isolation; the build ships a vendored [`coi-serviceworker`](https://github.com/gzuidhof/coi-serviceworker) shim, so no header configuration is required. See the README for deployment details. - -## Step 11 — Export a lesson to Quarto - -`export-quarto` converts a single lesson YAML into Quarto source on stdout — validating first and refusing invalid lessons. It has three shapes: - -```bash -# A fenced-div snippet to paste into an existing page -blendtutor export-quarto lessons/lesson_hello.yaml - -# A complete, renderable page (title, filter, and coi for R in the front matter) -blendtutor export-quarto --document lessons/lesson_hello.yaml > hello.qmd - -# The API key page learners use to store their Fireworks key -blendtutor export-quarto --key-page > api-key.qmd -``` - -Every optional field the widget understands is carried over: `code_template`, `checks`, `solution`, `hints`, `gotchas`, `success_criteria` (added to the in-browser feedback prompt), and `packages` (preloaded in webR/Pyodide). A lesson with no checks, solution, or hints still exports, with a warning on stderr: its widget offers only Run and LLM feedback. - -To render: - -1. Run `quarto add mcmullarkey/blendtutor` **from the folder that contains `_quarto.yml`** (or the `.qmd`, for a standalone page). Quarto only looks for `_extensions/` there; installing one level up leaves the filter undiscoverable and exercises render as plain text. -2. Enable the filter. `--document` and `--key-page` pages declare it themselves; for a snippet, add `filters: [mcmullarkey/blendtutor]` to the page or to `_quarto.yml` — not both, since Quarto merges the lists. -3. R exercises run everywhere. On standalone pages, `coi: true` lets webR use SharedArrayBuffer for faster execution; in `type: book` projects the COI service worker cannot control pages, so webR falls back to a slower channel that still runs R. Python exercises need no COI. -4. Serve over HTTP (`quarto preview`): `file://` blocks the ES modules and `localStorage` the widget needs. - -Requirements and the rendered snippet are covered by the [README §Quarto Extension](https://github.com/mcmullarkey/blendtutor#quarto-extension); for the end-to-end deploy see [whole-game §Quarto deploy](./whole-game.md#quarto-deploy). - -## Complete example courses - -The reference courses show the full workflow end to end: five lessons each, sibling eval suites, manifests, and committed `eval-report.json`. Browse `examples/write-less-code-{r,python}/`, copy a lesson, and adapt the prompt, checks, and eval cases. diff --git a/docs/book/src/whole-game.md b/docs/book/src/whole-game.md index 085b214..86b94b9 100644 --- a/docs/book/src/whole-game.md +++ b/docs/book/src/whole-game.md @@ -3,10 +3,12 @@ This chapter walks one course end to end: scaffold it, add a lesson, validate it, run it, score the grader's polarity, generate an eval report with the LLM judge, and deploy it two ways — as a static browser site and as a Quarto -document. Every command is copy-paste ready. For the field-by-field anatomy of -a lesson file, see -[Creating Lessons](./creating-lessons.md); for the Quarto extension and site -deployment details, see the [README](../../../README.md). +document. Every command is copy-paste ready. + +To author a course with Claude Code, use the +[`blendtutor-course` skill](https://github.com/mcmullarkey/blendtutor/blob/main/.claude/skills/blendtutor-course/SKILL.md) +that ships in the repo: it writes lessons with checks and solutions, a minimal +eval suite, and the Quarto or site output, and verifies the checks locally. ## Init — scaffold a course @@ -109,6 +111,14 @@ blendtutor eval-report lesson_hello.yaml This needs the same `FIREWORKS_API_KEY` as `eval`. +If you recorded from a git worktree, scrub the checkout prefix before +committing; `scripts/check-docs.sh` fails any `/Users/` path under `docs/evals/`: + +```bash +find docs/evals \( -name 'eval.json' -o -name 'run.yaml' \) -exec \ + perl -pi -e 's{/Users/[^/]*/portfolio/(?:worktree-|blendtutor-)[^/]*/}{}g' {} + +``` + ## Reading the eval report The report is static files under `docs/evals/lesson_hello/` — no server @@ -159,33 +169,111 @@ blendtutor build hello-course --target webr -o site `site/` output deploys to GitHub Pages as-is, and the build embeds an eval-results page when the course has a report. -## Quarto deploy +## Quarto extension -`export-quarto` prints the lesson as a Quarto fenced div, ready to paste into a -Quarto document: +blendtutor also ships as a [Quarto](https://quarto.org) extension for +interactive coding exercises in `.qmd` documents: in-browser editor, instant +checks, solution reveal, and AI feedback, all static HTML. Requires **Quarto >= 1.4**: ```bash -blendtutor export-quarto lesson_hello.yaml +quarto add mcmullarkey/blendtutor ``` +This installs to `_extensions/mcmullarkey/blendtutor/` (version 0.2.0). +**Run it from the folder that contains `_quarto.yml`** (or the `.qmd`): Quarto +only discovers `_extensions/` there, so an extension installed one directory up +never loads and exercises render as plain text. Assets deploy alongside the +rendered HTML, so asset resolution is install-path-independent. Commit +`_extensions/` so CI (for example a GitHub Pages workflow) renders the same filter. + +### Quick start (zero hand-written bootstrap) + +A complete copy-paste document with zero hand-written bootstrap. Filter by name, +`.blendtutor` div, render: + ````markdown +--- +title: "My exercises" +filters: [mcmullarkey/blendtutor] +--- + ::: {.blendtutor language="r"} - +Write a function `add(a, b)` that returns the sum. + +```r +add <- function(a, b) { ___ } +``` + +```{.r .checks} +stopifnot(add(1, 2) == 3) +``` ::: ```` -For a page that renders on its own, add `--document`; for the page learners use -to store their API key, run `blendtutor export-quarto --key-page > api-key.qmd`. +Preview it with `quarto preview` and the exercise is interactive immediately. +Python exercises use the same div with `language="python"`. Enable the filter in +the page front matter or in `_quarto.yml`, not both. -To render exercises, install the blendtutor Quarto extension — see the -[README](../../../README.md) for requirements (Quarto 1.4 or newer) — from the -folder that contains `_quarto.yml`, then render: +### Auto-bootstrap opt-out + +The filter auto-bootstraps by default; to wire up the runtime yourself, set +`bt-auto-bootstrap: false` in the YAML header. To keep it but disable the +auto-mounted AI feedback, set `bt-feedback: false`; see +[BYOK](#byok-bring-your-own-key). + +### Export a lesson + +`export-quarto` writes the div from a lesson file, carrying its prompt, code +template, checks, solution, hints, gotchas, success criteria, and packages: ```bash -quarto add mcmullarkey/blendtutor -quarto render +blendtutor export-quarto lesson_hello.yaml # snippet to paste into a page +blendtutor export-quarto --document lesson_hello.yaml > page.qmd # standalone page with front matter +blendtutor export-quarto --key-page > api-key.qmd # the API key page +``` + +Written by hand, the same pieces are optional blocks inside the div: a +`{.r .solution}` code block, `::: {.hints}` and `::: {.gotchas}` bullet divs, a +`::: {.success-criteria}` rubric for AI feedback, and a `packages="dplyr,purrr"` +attribute on the div. `export-quarto` warns on stderr when a lesson has no +checks, solution, or hints. + +### Cross-origin isolation (COI) + +webR runs faster with `SharedArrayBuffer`, which needs cross-origin isolation +(COOP/COEP). Opt in with `coi: true` (page YAML header) or `coi="true"` (any div); +the filter injects a service-worker shim. Pyodide-only pages do not need COI. + +> **Book-mode limitation:** COI does not function in Quarto `type: book` projects: the shim's scope cannot cover the book's `_output/` pages, so webR uses its slower non-isolated channel ([ADR-0015](https://github.com/mcmullarkey/blendtutor/blob/main/docs/adr/0015-opt-in-coi-cross-origin.md)). + +### Demo book + +A complete demo book with R and Python exercises lives in +[`demo-book/`](https://github.com/mcmullarkey/blendtutor/tree/main/demo-book), +rendered live at (rebuild +locally with `cd demo-book && quarto render`). It is a Quarto `type: book` project, so +COI does not take effect (limitation above). Python exercises are fully interactive +and every page ships a static fallback. R exercises run in the book too, on webR's slower +fallback channel; on the CLI-built [example sites](./examples.md), +R exercises run interactively via webR with isolation. +Over `file://` you get static exercise content only; serve over HTTP: + +```bash +cd demo-book/_output && python3 -m http.server 8000 ``` -R exercises run in books too, on webR's slower non-isolated channel; `coi: true` speeds them up on standalone pages. See -[Creating Lessons §Step 11](./creating-lessons.md#step-11--export-a-lesson-to-quarto). +## BYOK (Bring Your Own Key) + +Browser feedback uses the learner's own API key, with no server-side key. +Feedback is **auto-mounted**: the injected bootstrap imports +`exercise-feedback.js` and calls `mountAllFeedback(registry)` after the runtime +starts. Clicking Get feedback without a stored key shows the key form inline, +and a `--key-page` page manages the key too. The key is shared via +`localStorage`, readable by any JavaScript on the page's origin, so never reuse +a critical key; it is sent only to `api.fireworks.ai`. BYOK is Fireworks-only +(pinned model `accounts/fireworks/models/deepseek-v4-flash-0731`); the CLI +supports other providers (see the [README](https://github.com/mcmullarkey/blendtutor#api-key)). +Serve over HTTP: `file://` breaks `localStorage` sharing and blocks ES modules, +so feedback never mounts. Self-hosted CSP: add +`connect-src https://api.fireworks.ai` (Pages cannot set CSP headers; the shim +covers only COOP/COEP). diff --git a/scripts/check-docs.sh b/scripts/check-docs.sh index 0a08649..db93cf1 100755 --- a/scripts/check-docs.sh +++ b/scripts/check-docs.sh @@ -94,13 +94,13 @@ if [ -d docs/evals ]; then fi # AC-215 — committed smevals evidence must be portable: no worktree-specific -# absolute paths under docs/evals/. The creating-lessons.md Step 9 convention +# absolute paths under docs/evals/. The whole-game.md "Eval report" scrub convention # strips the /Users/.../portfolio// prefix (worktree-issue-N/, # worktree-*, blendtutor-*) from lesson/runner/checker fields before git add; # this pin fails closed on any /Users/ leak regardless of where it came from. if [ -d docs/evals ]; then if rg -l '/Users/' docs/evals/ >/dev/null; then - echo "docs: /Users/ absolute path leaked into docs/evals/ (scrub per creating-lessons.md Step 9)" >&2 + echo "docs: /Users/ absolute path leaked into docs/evals/ (scrub per whole-game.md "Eval report")" >&2 exit 1 fi fi @@ -225,7 +225,7 @@ test -f "$book_out/whole-game.html" \ || { echo "docs: built mdBook missing whole-game.html (page not in SUMMARY.md?)" >&2; exit 1; } grep -q 'evals/lesson_hello' "$book_out/whole-game.html" \ || { echo "docs: built whole-game.html missing evals/lesson_hello citation" >&2; exit 1; } -grep -q 'export-quarto' "$book_out/creating-lessons.html" \ - || { echo "docs: built creating-lessons.html missing export-quarto step" >&2; exit 1; } +grep -q 'export-quarto' "$book_out/whole-game.html" \ + || { echo "docs: built whole-game.html missing export-quarto step" >&2; exit 1; } echo "docs: OK — merged site at $book_out (book at /, API at /api, examples at /examples/{r,python}, demo book at /demo-book/, evals at /evals/)" diff --git a/scripts/tests/test_demo_docs.sh b/scripts/tests/test_demo_docs.sh index d1a662e..d505a36 100644 --- a/scripts/tests/test_demo_docs.sh +++ b/scripts/tests/test_demo_docs.sh @@ -22,11 +22,10 @@ # c6. Pyodide accuracy guard, whole README (pyodide needs no COI) # c7. No stale /examples/ conflation inside the demo section # c8. Extend-don't-duplicate: 'COI does not function in Quarto' == 1 AND -# 'Book-mode limitation' == 1 (whole README) -# c9. Region pin: live demo-book URL at line >= 60 and < 150 (whole -# README; issue #225 reslimmed the README 383→149 lines, so the old -# 288-342 region no longer exists) -# c10. ADR-0015 pointer in README + file exists +# 'Book-mode limitation' == 1 (whole-game chapter) +# c9. README links to the whole-game demo section (whole-game.html#demo-book); +# the Quarto/COI/demo prose moved out of README into that chapter +# c10. ADR-0015 pointer in the whole-game chapter + file exists # c11. Distribution-doc pins survive (test_quarto_distribution.sh README # group): python3 -m http.server 8000 present; 'COI configuration' # absent; PANDOC_SCRIPT_FILE absent; `type: book` present; demo-book/ @@ -47,12 +46,16 @@ ok() { echo " PASS: $1"; PASS=$((PASS + 1)); } ko() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); } README="README.md" +# The Quarto/COI/demo-book prose moved from README into the whole-game chapter +# (README keeps a short pointer section); content pins read DOC, README keeps +# the dead-URL guard, the pointer link, and the concision ceiling. +DOC="docs/book/src/whole-game.md" DEMO_BOOK_DIR="demo-book" ADR_FILE="docs/adr/0015-opt-in-coi-cross-origin.md" # Demo section scope: `### Demo book` through `## BYOK` (exclusive of the # BYOK section, which now follows the demo section before `## License`). -DEMO_SECTION="$(awk '/^### Demo book/,/^## BYOK/' "$README")" +DEMO_SECTION="$(awk '/^### Demo book/,/^## BYOK/' "$DOC")" # --------------------------------------------------------------------------- # c1: Live demo-book URL exact literal, trailing slash pinned (demo section); @@ -65,10 +68,10 @@ if printf '%s' "$DEMO_SECTION" | grep -qF 'https://mcmullarkey.github.io/blendtu else ko "live demo-book URL literal missing from demo section" fi -if grep -qF 'https://mcmullarkey.github.io/blendtutor/demo/' "$README"; then - ko "dead /demo/ URL still present in README (demo-standalone removed by #227)" +if grep -qF 'https://mcmullarkey.github.io/blendtutor/demo/' "$README" "$DOC"; then + ko "dead /demo/ URL still present in README or whole-game (demo-standalone removed by #227)" else - ok "dead /demo/ URL absent from whole README" + ok "dead /demo/ URL absent from README and whole-game" fi # --------------------------------------------------------------------------- @@ -136,10 +139,10 @@ fi # c6: Pyodide accuracy guard — whole README # --------------------------------------------------------------------------- echo "== c6: pyodide no-COI accuracy guard ==" -if grep -qE 'pyodide.*(do not|doesn.?t|no).*COI|Pyodide-only.*do not need COI' "$README"; then - ok "README states pyodide needs no COI" +if grep -qE 'pyodide.*(do not|doesn.?t|no).*COI|Pyodide-only.*do not need COI' "$DOC"; then + ok "whole-game states pyodide needs no COI" else - ko "pyodide accuracy — no pyodide-no-COI statement anywhere in README" + ko "pyodide accuracy — no pyodide-no-COI statement anywhere in whole-game" fi # --------------------------------------------------------------------------- @@ -157,13 +160,13 @@ fi # c8: Extend-don't-duplicate — count pins (whole README) # --------------------------------------------------------------------------- echo "== c8: extend-don't-duplicate count pins ==" -count_coi_phrase="$(grep -cF 'COI does not function in Quarto' "$README" || true)" +count_coi_phrase="$(grep -cF 'COI does not function in Quarto' "$DOC" || true)" if [ "$count_coi_phrase" -eq 1 ]; then ok "'COI does not function in Quarto' appears exactly once" else ko "'COI does not function in Quarto' count != 1 (got $count_coi_phrase — duplicated or deleted)" fi -count_book_heading="$(grep -c 'Book-mode limitation' "$README" || true)" +count_book_heading="$(grep -c 'Book-mode limitation' "$DOC" || true)" if [ "$count_book_heading" -eq 1 ]; then ok "'Book-mode limitation' appears exactly once" else @@ -171,30 +174,25 @@ else fi # --------------------------------------------------------------------------- -# c9: Region pin — live demo-book URL at line >= 60 and < 150 (whole README). -# Issue #225 reslimmed the README (383 → 149 lines); the old 288-342 -# region encoded the pre-slim layout. Band widened to 60-150 so -# legitimate edits above/below the demo section don't fail the pin; -# content pins (c1-c8) + the c12 ceiling carry the real contract. +# c9: README points at the whole-game demo section (the prose lives there now) +# --------------------------------------------------------------------------- +echo "== c9: README links to the whole-game demo section ==" +if grep -qF 'whole-game.html#demo-book' "$README"; then + ok "README links whole-game.html#demo-book" +else + ko "README missing link to whole-game.html#demo-book" +fi + # --------------------------------------------------------------------------- -echo "== c9: demo section region pin ==" -for url in 'https://mcmullarkey.github.io/blendtutor/demo-book/'; do - line="$(grep -nF "$url" "$README" | cut -d: -f1 | head -n1 || true)" - if [ -n "$line" ] && [ "$line" -ge 60 ] && [ "$line" -lt 150 ]; then - ok "URL at line $line (60 <= line < 150): $url" - else - ko "URL line pin failed for $url (got: ${line:-missing})" - fi -done # --------------------------------------------------------------------------- # c10: ADR-0015 pointer + file exists # --------------------------------------------------------------------------- echo "== c10: ADR-0015 pointer ==" -if grep -qF 'docs/adr/0015-opt-in-coi-cross-origin.md' "$README"; then - ok "README links docs/adr/0015-opt-in-coi-cross-origin.md" +if grep -qF 'docs/adr/0015-opt-in-coi-cross-origin.md' "$DOC"; then + ok "whole-game links docs/adr/0015-opt-in-coi-cross-origin.md" else - ko "README missing ADR-0015 pointer (docs/adr/0015-opt-in-coi-cross-origin.md)" + ko "whole-game missing ADR-0015 pointer (docs/adr/0015-opt-in-coi-cross-origin.md)" fi if [ -f "$ADR_FILE" ]; then ok "ADR file exists ($ADR_FILE)" @@ -206,22 +204,22 @@ fi # c11: Distribution-doc pins survive (test_quarto_distribution.sh README group) # --------------------------------------------------------------------------- echo "== c11: distribution-doc pins survive ==" -if grep -qF 'python3 -m http.server 8000' "$README"; then +if grep -qF 'python3 -m http.server 8000' "$DOC"; then ok "serve-over-HTTP instruction present (python3 -m http.server 8000)" else ko "distribution pin — 'python3 -m http.server 8000' missing" fi -if ! grep -qF 'COI configuration' "$README"; then +if ! grep -qF 'COI configuration' "$DOC"; then ok "no overclaiming 'COI configuration' phrase" else ko "distribution pin — 'COI configuration' present (overclaim)" fi -if ! grep -qF 'PANDOC_SCRIPT_FILE' "$README"; then +if ! grep -qF 'PANDOC_SCRIPT_FILE' "$DOC"; then ok "no stale mechanism phrase 'PANDOC_SCRIPT_FILE'" else ko "distribution pin — 'PANDOC_SCRIPT_FILE' present (stale mechanism)" fi -if grep -qF 'type: book' "$README"; then +if grep -qF 'type: book' "$DOC"; then ok "COI caveat names Quarto type: book (README-wide)" else ko "distribution pin — 'type: book' not found" diff --git a/scripts/tests/test_quarto_distribution.sh b/scripts/tests/test_quarto_distribution.sh index 02e3b5d..d3f7f50 100644 --- a/scripts/tests/test_quarto_distribution.sh +++ b/scripts/tests/test_quarto_distribution.sh @@ -55,6 +55,10 @@ ok() { echo " PASS: $1"; PASS=$((PASS + 1)); } ko() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); } README="README.md" +# The Quarto extension docs moved from README into the whole-game chapter; the +# Group 1 content clauses accept either file (README keeps install + pointers). +WHOLE_GAME="docs/book/src/whole-game.md" +README_DOCS=("$README" "$WHOLE_GAME") DEMO_BOOK_DIR="demo-book" CI_FILE=".github/workflows/ci.yml" @@ -69,7 +73,7 @@ echo "== Clause 1: install command ==" if [ ! -f "$README" ]; then ko "install command — README not found: $README" else - if grep -qF 'quarto add mcmullarkey/blendtutor' "$README"; then + if grep -qF 'quarto add mcmullarkey/blendtutor' "${README_DOCS[@]}"; then ok "install command present (quarto add mcmullarkey/blendtutor)" else ko "install command present — 'quarto add mcmullarkey/blendtutor' not found in README" @@ -82,14 +86,14 @@ if [ ! -f "$README" ]; then ko "syntax both languages — README not found" else # Check for R exercise syntax (language="r") - if grep -qF 'language="r"' "$README" || grep -qF "language='r'" "$README"; then + if grep -qF 'language="r"' "${README_DOCS[@]}" || grep -qF "language='r'" "${README_DOCS[@]}"; then ok "R exercise syntax shown in README" else ko "R exercise syntax shown in README — language=\"r\" not found" fi # Check for Python exercise syntax (language="python") - if grep -qF 'language="python"' "$README" || grep -qF "language='python'" "$README"; then + if grep -qF 'language="python"' "${README_DOCS[@]}" || grep -qF "language='python'" "${README_DOCS[@]}"; then ok "Python exercise syntax shown in README" else ko "Python exercise syntax shown in README — language=\"python\" not found" @@ -102,7 +106,7 @@ if [ ! -f "$README" ]; then ko "BYOK — README not found" else # Check for BYOK or "bring your own key" (case-insensitive) - if grep -qiE 'BYOK|bring.your.own.key' "$README"; then + if grep -qiE 'BYOK|bring.your.own.key' "${README_DOCS[@]}"; then ok "BYOK mentioned in README" else ko "BYOK mentioned in README — no BYOK or 'bring your own key' found" @@ -115,7 +119,7 @@ if [ ! -f "$README" ]; then ko "minimum Quarto version — README not found" else # Check for a Quarto version requirement (e.g., "Quarto 1.4", "Quarto >= 1.4", "Quarto 1.5+") - if grep -qiE 'quarto[[:space:]]*[>=]*[[:space:]]*1\.[0-9]' "$README"; then + if grep -qiE 'quarto[[:space:]]*[>=]*[[:space:]]*1\.[0-9]' "${README_DOCS[@]}"; then ok "minimum Quarto version stated in README" else ko "minimum Quarto version stated in README — no Quarto version requirement found" @@ -128,7 +132,7 @@ if [ ! -f "$README" ]; then ko "demo book link — README not found" else # Check for a link to the demo-book directory - if grep -qiE 'demo-book|demo.book' "$README"; then + if grep -qiE 'demo-book|demo.book' "${README_DOCS[@]}"; then ok "demo book link present in README" else ko "demo book link present in README — no demo-book reference found" @@ -141,21 +145,21 @@ if [ ! -f "$README" ]; then ko "install path — README not found" else # 6a: Install command survives - if grep -qF 'quarto add mcmullarkey/blendtutor' "$README"; then + if grep -qF 'quarto add mcmullarkey/blendtutor' "${README_DOCS[@]}"; then ok "install command present (quarto add mcmullarkey/blendtutor)" else ko "install command present — 'quarto add mcmullarkey/blendtutor' not found" fi # 6b: Correct install path stated (org/repo path, not bare repo name) - if grep -qF '_extensions/mcmullarkey/blendtutor/' "$README"; then + if grep -qF '_extensions/mcmullarkey/blendtutor/' "${README_DOCS[@]}"; then ok "install path stated (_extensions/mcmullarkey/blendtutor/)" else ko "install path stated — '_extensions/mcmullarkey/blendtutor/' not found" fi # 6c: Old wrong claim gone (bare 'your project's _extensions/blendtutor/' phrase) - if ! grep -qF "your project's \`_extensions/blendtutor/\`" "$README"; then + if ! grep -qF "your project's \`_extensions/blendtutor/\`" "${README_DOCS[@]}"; then ok "old wrong install path claim removed" else ko "old wrong install path claim removed — found 'your project's \`_extensions/blendtutor/\`'" @@ -164,7 +168,7 @@ else # 6d: Install-path independence stated (outcome-level only — assets deploy # alongside the rendered HTML; NOT the stale 'relative to the filter # script' mechanism, which issue #147 removed in lockstep with the README). - if grep -qiE 'install-path-independent|independent of.{0,40}install|regardless install' "$README"; then + if grep -qiE 'install-path-independent|independent of.{0,40}install|regardless install' "${README_DOCS[@]}"; then ok "install-path independence stated in README" else ko "install-path independence stated in README — no independence mention found" @@ -190,7 +194,7 @@ fi echo "== Clause 7: zero-bootstrap quick-start ==" QUICK_START="" if [ -f "$README" ]; then - QUICK_START=$(awk '/^#### Quick start/{flag=1;next} /^#### Auto-bootstrap opt-out/{flag=0;next} flag' "$README") + QUICK_START=$(awk '/^#+ Quick start/{flag=1;next} /^#+ Auto-bootstrap opt-out/{flag=0;next} flag' "$WHOLE_GAME") fi if [ -z "$QUICK_START" ]; then ko "quick-start — no '#### Quick start' subsection found in README" @@ -224,12 +228,12 @@ fi # Clause 9: auto-bootstrap opt-out documented (literal + prose). echo "== Clause 9: auto-bootstrap opt-out ==" -if [ -f "$README" ] && grep -qF 'bt-auto-bootstrap: false' "$README"; then +if [ -f "$README" ] && grep -qF 'bt-auto-bootstrap: false' "${README_DOCS[@]}"; then ok "opt-out literal present (bt-auto-bootstrap: false)" else ko "opt-out literal — 'bt-auto-bootstrap: false' not found" fi -if [ -f "$README" ] && grep -qiE 'opt-out' "$README"; then +if [ -f "$README" ] && grep -qiE 'opt-out' "${README_DOCS[@]}"; then ok "opt-out prose present" else ko "opt-out prose — no 'opt-out' mention found" @@ -242,33 +246,33 @@ fi # plain /^## BYOK/,/^## / range self-terminates on the heading — the heading # matches both patterns). echo "== Clause 10: feedback auto-mount ==" -if [ -f "$README" ] && grep -qF 'exercise-feedback.js' "$README"; then +if [ -f "$README" ] && grep -qF 'exercise-feedback.js' "${README_DOCS[@]}"; then ok "feedback names exercise-feedback.js" else ko "feedback — 'exercise-feedback.js' not found" fi -if [ -f "$README" ] && grep -qF 'mountAllFeedback' "$README"; then +if [ -f "$README" ] && grep -qF 'mountAllFeedback' "${README_DOCS[@]}"; then ok "feedback auto-mounts via mountAllFeedback" else ko "feedback — 'mountAllFeedback' not found" fi -if [ -f "$README" ] && grep -qiE 'auto-mounted|auto-mounts' "$README"; then +if [ -f "$README" ] && grep -qiE 'auto-mounted|auto-mounts' "${README_DOCS[@]}"; then ok "feedback stated auto-mounted (not manual opt-in)" else ko "feedback — no 'auto-mounted'/'auto-mounts' statement found" fi -if [ -f "$README" ] && ! grep -qiE 'not auto-mounted|manual opt-in' "$README"; then +if [ -f "$README" ] && ! grep -qiE 'not auto-mounted|manual opt-in' "${README_DOCS[@]}"; then ok "no stale 'manual opt-in' / 'not auto-mounted' claim survives" else ko "feedback — stale 'not auto-mounted'/'manual opt-in' claim still present" fi -if [ -f "$README" ] && grep -qF 'ANTHROPIC_API_KEY' "$README"; then +if [ -f "$README" ] && grep -qF 'ANTHROPIC_API_KEY' "${README_DOCS[@]}"; then ok "CLI ANTHROPIC_API_KEY env-var doc survives (rig/ADR-0006)" else ko "CLI ANTHROPIC_API_KEY doc — 'ANTHROPIC_API_KEY' not found in README" fi # BYOK section is Fireworks-only: zero ANTHROPIC_API_KEY inside it. -BYOK_SEC="$(awk '/^## BYOK/{f=1} f{print} f && /^## / && !/^## BYOK/{exit}' "$README")" +BYOK_SEC="$(awk '/^## BYOK/{f=1} f{print} f && /^## / && !/^## BYOK/{exit}' "$WHOLE_GAME")" if [ -f "$README" ] && ! printf '%s' "$BYOK_SEC" | grep -q 'ANTHROPIC_API_KEY'; then ok "BYOK section has zero ANTHROPIC_API_KEY (Fireworks-only)" else @@ -283,22 +287,22 @@ fi # clauses, incl. exact live-URL literals + capability mapping). This grep is # kept, not retired, for distribution-group coverage of the README contract. echo "== Clause 11: COI book-mode caveat ==" -if [ -f "$README" ] && grep -qF 'type: book' "$README"; then +if [ -f "$README" ] && grep -qF 'type: book' "${README_DOCS[@]}"; then ok "COI caveat names Quarto type: book" else ko "COI caveat — 'type: book' not found" fi -if [ -f "$README" ] && grep -qiE 'does not function|does not take effect|cannot cover' "$README"; then +if [ -f "$README" ] && grep -qiE 'does not function|does not take effect|cannot cover' "${README_DOCS[@]}"; then ok "COI caveat states book-mode does not function" else ko "COI caveat — no book-mode limitation statement found" fi -if [ -f "$README" ] && grep -qF '_output/' "$README"; then +if [ -f "$README" ] && grep -qF '_output/' "${README_DOCS[@]}"; then ok "COI caveat explains _output/ scope" else ko "COI caveat — '_output/' not mentioned" fi -if [ -f "$README" ] && ! grep -qF 'COI configuration' "$README"; then +if [ -f "$README" ] && ! grep -qF 'COI configuration' "${README_DOCS[@]}"; then ok "demo book no longer overclaims 'COI configuration'" else ko "demo book overclaim — 'COI configuration' still present" @@ -308,22 +312,22 @@ fi # consistency (matches ci.yml:148 full org/repo command, _extension.yml:3 # version 0.2.0, no short-form install). echo "== Clause 12: no stale mechanism, command/version consistency ==" -if [ -f "$README" ] && ! grep -qiE 'relative to.*filter|PANDOC_SCRIPT_FILE|locates its assets' "$README"; then +if [ -f "$README" ] && ! grep -qiE 'relative to.*filter|PANDOC_SCRIPT_FILE|locates its assets' "${README_DOCS[@]}"; then ok "stale mechanism claim removed (relative to filter script / PANDOC_SCRIPT_FILE)" else ko "stale mechanism claim still present (relative to.*filter|PANDOC_SCRIPT_FILE|locates its assets)" fi -if [ -f "$README" ] && ! grep -qE 'import .*exercise-runtime\.js' "$README"; then +if [ -f "$README" ] && ! grep -qE 'import .*exercise-runtime\.js' "${README_DOCS[@]}"; then ok "no stale runtime bootstrap import (import .*exercise-runtime.js)" else ko "stale runtime bootstrap import found (import .*exercise-runtime.js)" fi -if [ -f "$README" ] && ! grep -qE 'quarto add[[:space:]]+blendtutor([^/]|$)' "$README"; then +if [ -f "$README" ] && ! grep -qE 'quarto add[[:space:]]+blendtutor([^/]|$)' "${README_DOCS[@]}"; then ok "no short-form install command (full org/repo required, matches ci.yml:148)" else ko "short-form install command found ('quarto add blendtutor')" fi -if [ -f "$README" ] && grep -qF '0.2.0' "$README"; then +if [ -f "$README" ] && grep -qF '0.2.0' "${README_DOCS[@]}"; then ok "version stated matches _extension.yml (0.2.0)" else ko "version consistency — '0.2.0' not stated in README" @@ -333,17 +337,17 @@ fi # Part 3 — exercises require HTTP because file:// CORS-blocks ES modules; the # README must tell users how to serve the rendered book interactively). echo "== Clause 13: demo book serve-over-HTTP instructions ==" -if [ -f "$README" ] && grep -qF 'python3 -m http.server 8000' "$README"; then +if [ -f "$README" ] && grep -qF 'python3 -m http.server 8000' "${README_DOCS[@]}"; then ok "README serve-over-HTTP instruction present (python3 -m http.server 8000)" else ko "README serve-over-HTTP instruction — 'python3 -m http.server 8000' not found" fi -if [ -f "$README" ] && grep -qiE 'file://|ES modules|CORS' "$README"; then +if [ -f "$README" ] && grep -qiE 'file://|ES modules|CORS' "${README_DOCS[@]}"; then ok "README explains file:// blocks ES modules" else ko "README explains file:// blocks ES modules — no file:// / ES modules / CORS mention" fi -if [ -f "$README" ] && grep -qiE 'static exercise content|not interactive' "$README"; then +if [ -f "$README" ] && grep -qiE 'static exercise content|not interactive' "${README_DOCS[@]}"; then ok "README notes file:// shows static fallback content only" else ko "README notes file:// shows static fallback content only — no 'static exercise content'/'not interactive' mention" From b3ca18cc260daded98024f5f3c98e8dc2e8a780c Mon Sep 17 00:00:00 2001 From: Michael Mullarkey Date: Sun, 13 Sep 2026 16:49:27 -0600 Subject: [PATCH 4/4] fix(review): check-docs scrub message keeps its quoted section name --- scripts/check-docs.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/check-docs.sh b/scripts/check-docs.sh index db93cf1..ee912bf 100755 --- a/scripts/check-docs.sh +++ b/scripts/check-docs.sh @@ -100,7 +100,7 @@ fi # this pin fails closed on any /Users/ leak regardless of where it came from. if [ -d docs/evals ]; then if rg -l '/Users/' docs/evals/ >/dev/null; then - echo "docs: /Users/ absolute path leaked into docs/evals/ (scrub per whole-game.md "Eval report")" >&2 + echo "docs: /Users/ absolute path leaked into docs/evals/ (scrub per whole-game.md 'Eval report')" >&2 exit 1 fi fi