Add Autocomplete playground tool to Astro/Vue frontend - #7
Closed
gaurav wants to merge 129 commits into
Closed
Conversation
- Add IdentifierRecord dataclass to babel_xrefs.py (resolves TODO) - Add 89 tests across 3 files: test_downloader (26), test_babel_xrefs (31), test_nodenorm (23) - Unit tests (71) use mocks and run without network; integration tests (18) use real downloads/APIs - Add session-scoped fixtures in conftest.py for shared Parquet file downloads - Parametrize integration tests over tests/data/valid_curies.txt for easy expansion - Add integration and slow pytest markers to pyproject.toml - Update CLAUDE.md and README.md with testing documentation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a web UI (FastAPI + Jinja2 + htmx + Bootstrap 5) exposing NodeNorm, XRefs, IDs, and Test Concordance via browser forms, a JSON REST API, and CSV downloads. Launched with `babel-explorer web`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The NodeNorm and Test Concordance pages now show a select dropdown populated from NodeNorm.URLs (defaulting to NodeNorm Dev), with a "Custom URL..." option that reveals a free-text input. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CLI args are forwarded via environment variables so the reloaded subprocess picks up the same config. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add web route table and NodeNorm dropdown details to CLAUDE.md. Add web frontend section to README.md covering startup, REST API examples, and CSV download endpoints. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Rather than showing the full node.type hierarchy (Gene → BiologicalEntity → NamedThing…), compute "direct types" from the unique type values of each equivalent identifier. For non-conflated results this is a single type; for conflated results (e.g. Gene+Protein) it shows 2–5 types in first-appearance order. All type occurrences are now rendered as links to the biolink model docs (https://biolink.github.io/biolink-model/{Type}). The individual_types toggle is removed from the Advanced options UI since the display depends on it always being enabled. ResultsSummary type filter now uses direct types, so conflated CURIEs appear under all their type buckets. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ComparisonView was missed in the previous change: it still used node.type[0] for the type filter and node.type.slice(0,2) for the summary row badges. Now uses getDirectTypes() for both, consistent with the rest of the UI. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Browser-based NameRes tool with multi-instance comparison, expected CURIE validation via [[CURIE]] annotations, shareable URLs, and full-response JSON modal. Includes 46 new tests and API response fixtures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each instance panel in the expanded CURIE detail row now shows a small ↗ link next to the instance name. Clicking it opens the NodeNorm get_normalized_nodes GET response for that specific CURIE and instance in a new tab, using the same API options that were used for the query. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the copy-pasted instance-selection fieldset in NodeNormForm and NameResForm with a shared InstanceSelector component. UI changes: - Short env-level names: RENCI Dev, RENCI Experimental, ITRB CI, ITRB Test, ITRB Prod (full names kept in ComparisonView headers) - Primary envs (Dev, CI, Prod) always visible; Exp, Test, and custom URL input collapsed behind an "Extended environments" disclosure - Pipeline order documented in disclosure summary - Full endpoint URL shown as tooltip (title) on each label - Quick-select row: All | Primary | None - Custom URLs auto-get trailing slash; Enter key submits - "Save as default" persists selection to localStorage - Selection shared across NodeNorm/NameRes within the same session via a module-level sessionPrefs ref in instance-prefs.ts Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Live-as-you-type NameRes with debounce+abort, preset dropdown (Disease/Gene/SmallMolecule/Custom), latency badges, Solr highlighting, expected-CURIE rank check, and single- or multi-instance views with rank/label/types diff highlighting. State round-trips through the URL for shareable review links. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…abels Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rch"> label Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
happy-dom doesn't implement localStorage; add an in-memory mock so all InstanceSelector tests can run. Also add esCiInstance fixture, update Primary/All assertions, and add a test verifying the <abbr> renders in the primary section. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…on view In the comparison view, replace the per-instance label display under each CURIE with the most specific Biolink type (r.types[0]), linked to the Biolink model docs. If types differ across instances, show the union of distinct types in the CURIE column and add the per-instance type (italic) below each cell's label. Remove the now-redundant 'types differ' badge; the row highlight and inline types make the mismatch self-evident. Also add BiolinkTypeLink to the single-instance Types column so those badges are clickable links too. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rarchy - Add BiolinkTypeLink.vue to shared-components lists in CLAUDE.md and web/README.md — it was already used across NodeNorm, NameRes, and Autocomplete but undocumented - Clarify that NameResResult.types is a Biolink hierarchy (most-specific first), so types[0] is the direct type; add JSDoc in nameres-types.ts - Update Vitest test count 275 → 276 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… view The previous layout rendered both the CURIE and the Biolink type as plain blue hyperlinks, making them hard to tell apart. Now the type is rendered as a small muted badge on the same line as the CURIE (visually distinct), and the deduplicated label(s) from all instances sit below in small muted text — giving the row the same readable shape as before the type was added. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
"In-flight" is engineering jargon; "Cancel lookup" is clearer for a biomedical audience. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Each instance column header now shows a small "API↗" link beside the instance name that opens the raw NameRes /lookup JSON response in a new tab. The link reflects the current query and options (reactive computed) and uses the existing apiUrlFor() logic already present in AutocompleteApp. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds sortInstances() to instance-prefs.ts: Exp → Dev → CI → ES CI → Test → Prod → custom URLs (then alpha by URL). Applied to queriedInstances in all three apps (Autocomplete, NodeNorm, NameRes) so columns always appear in a predictable order regardless of selection order. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
NodeNorm's CI environment is switching to ElasticSearch. The old Redis CI is kept for side-by-side comparison as a named primary instance. - config: replace nodenorm.ci with es_ci (https://nodenorm-es.ci.transltr.io/) and redis_ci (https://nodenorm.ci.transltr.io/) - InstanceSelector: add redis_ci to PRIMARY_ENVS with "ITRB Redis CI" label and <abbr title="Redis"> rendering, matching the es_ci pattern - NodeNormApp: update ENV_LABELS to use es_ci/redis_ci instead of ci - instance-prefs: add redis_ci at sort position 4 (after es_ci, before test) - nodenorm.py: update Python env_labels for new keys Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix duplicate Data Flow section in CLAUDE.md - Document NodeNorm vs NameRes env key asymmetry (es_ci/redis_ci vs ci/es_ci) - Add sortInstances/ENV_ORDER to instance-prefs.ts description in lib listing - Add 'Adding a new named instance' checklist to Key Design Patterns - Add canonical instance ordering pattern note - Note pre-existing Python test failures (test_nodenorm_page, test_test_concord_page) - Update NodeNorm instance list in web/README.md (ES CI, Redis CI replace CI) - Document Autocomplete API↗ link and Biolink type badge in comparison view Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
NodeNorm's CI is now fully ES-backed, so calling its env key "es_ci" was misleading. Rename it to "ci" so both services use a symmetric "ci" key that means "the current CI deployment" — NodeNorm CI is ES-based, NameRes CI is Solr-based. The NameRes "es_ci" key now unambiguously identifies the experimental NameRes ElasticSearch instance. Also adds three AutocompleteApp regression tests that verify target=es_ci in a shareable URL correctly resolves to namelookup-es.ci.transltr.io (not the bare string "es_ci") and that results appear in the comparison view column for that instance. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gaurav
force-pushed
the
add-nameres-autocomplete
branch
from
September 1, 2026 05:45
da2edff to
02196ae
Compare
gaurav
added a commit
to TranslatorSRI/babel-validation
that referenced
this pull request
Sep 1, 2026
…lt around promotion drift (#120) The GitHub Pages site was an earlier, weaker version of what is now [Babel Explorer](https://github.com/TranslatorSRI/babel-explorer). This replaces it with a dashboard that runs the validation suite against every environment daily and publishes the result, organised around the question the data says actually matters: **where in the promotion pipeline did this appear?** Closes #113. Supersedes #118 (left as a draft until this merges) and #117, which is closed: its four code files were byte-identical to these and its prose is a subset of what is here. ## What's here **The old site is gone.** The Prefix Comparator moved into Babel (NCATSTranslator/Babel#889) and Autocomplete into Babel Explorer (TranslatorSRI/babel-explorer#7), so both pages are deleted, along with `website-vue3-vite/` — a never-deployed app that duplicated the Google Sheet test logic in the browser (44 files, ~10k lines). **The suite emits machine-readable results.** `pytest --report-jsonl` (a `pytest_runtest_logreport` hook in `tests/conftest.py`) writes one JSON line per test phase — raw pytest facts only. `src/babel_validation/tools/generate_report.py` turns those into `report.json` and appends one summary line to `history.jsonl`; all classification into passed/failed/xfailed/xpassed/skipped/error happens there, where it is unit-testable, rather than in the fixture that produced it. A raw file is treated as damageable: one unparseable or malformed record costs that record and nothing else. A result whose node ID names no environment is counted as unattributed and published as a count, never as a table row, because the site has no column to render it in. **A daily workflow runs it** at 06:30 UTC: the suite against every environment, then the report, the site build and a deploy to `gh-pages`. Three things about that loop are worth knowing, because each was a bug first: - **The list of environments comes from `read_targets()`**, not from a second copy in the YAML. When those disagreed, the extra target reached the site as a permanently unreachable column, sorted after prod, and as an always-empty position in every `?sig=` signature — which silently invalidates previously shared links. - **A failing test run and a broken one are told apart.** Failing tests are the artifact this workflow exists to publish, so exit 0 or 1 passes silently — but only when that target actually wrote results, because the command is `uv run pytest` and uv exits 1 for its own failures too, before pytest starts. `timeout` firing (124) is annotated as a warning, because the environment hung and the report already says so; anything else — a collection error, a usage error, nothing collected — is an error annotation. Every broken target fails the job in a final step *after* the deploy, so the targets that did work still publish. `generate_report` already refused to publish when *every* target was unreachable; this is the partial case, which is the one that looks fine. - **Only a manual dispatch cancels a run in progress.** A blanket `cancel-in-progress` would let a push, or the next day's cron, kill a scheduled run mid-flight — and that run is what appends the day's line to `history.jsonl`. **Three pages, not one.** *Dashboard* (run banner, per-environment pipeline cards, promotion-drift panel, environment detail matrix), *Results* (a sticky filter bar over the test matrix, filterable by category, source and environment, with shareable URL state), and *History* (what changed since the previous run, then a row per run). **The design follows the data.** Measured against the 2026-09-01 report: of **4,584** results, **528** are interesting, and **519 of those differ across environments** — only 9 fail everywhere. Failures per environment in promotion order run exp 21 → dev 79 → ci 81 → ci-es 222 → test 80 → prod 327, and the top outcome patterns are `pass ×5 FAIL` (148), `pass pass pass FAIL pass FAIL` (100) and `xfail ×5 XPASS` (67). Drift dominates the file, and it used to be hidden behind a checkbox labelled "interesting only". The redesign promotes it to structure. **Bootstrap 5.3 with a small theme layer, not PrimeVue.** A component library only reaches inside the Vue islands, the matrix is a pivot we would hand-write regardless, and the default view is ~530 rows, so virtual scroll buys nothing. **The report is untrusted input, and is treated as such.** No `v-html` anywhere; report values render only through `{{ }}`; links are built from generator-validated parts; blocklist rows withhold their detail in the expanded-row markup as well as the label; no Google Sheet ID or link appears anywhere in the output. A facet is as public as a cell, so the filter dropdowns exclude blocklist rows exactly as the table does. The URL parameters are treated the same way: they are read before the report loads, so there is nothing to check them against, and every one of them goes through `Object.hasOwn` rather than indexing a JSON-parsed object directly — a rule now in `CLAUDE.md`, because it has been two bugs in the same file. **The Google Sheet IDs are out of the repository, and have been rotated.** They live in `BABEL_VALIDATION_SHEET_ID` and `BABEL_VALIDATION_BLOCKLIST_SHEET_ID` (`.env` locally, repository secrets in Actions), resolved through `resolve_sheet_id()`. This matters more than it looks: the sheets are shared as "anyone with the link" because the CSV export is unauthenticated, so the ID *is* the credential — and the old one is checked into `main` and sits in this repository's public history, where `git rm` cannot reach it. Both sheets have been restricted, both old IDs now return 401 credential-free on the `gviz` CSV export and on `/pub?output=csv`, and both secrets hold new IDs as of 2026-09-01 (#126). A new `env.default` documents every variable and what each one turns off when missing, so `cp env.default .env` is the whole setup, and `CLAUDE.md` now tells coding agents not to read `.env` back — everything an agent reads lands in a transcript that is stored, replayed and pasted into issues. ## What it produces A run takes **19–36 minutes**, almost all of it the six sequential per-target pytest invocations; the report, build and deploy together are about 30 seconds. The most recent run published 4,152 Google Sheet results, 313 blocklist results and 96 GitHub issue results across six environments. Tests: **46 vitest** specs across `results`, `history`, `driftPanel`, `statusMatrix`, `overview` and `reportData`, run in CI beside the Python unit tests, which are now **192**. ## What it deliberately does not do - **No NameRes ES in Test.** Test matches CI — NodeNorm ES against the Solr-backed NameRes — because NameRes ES is being validated in CI (`[ci-es]`) first. The Redis-backed `nodenorm.test.transltr.io` was switched off on 2026-08-31, so there is no Redis NodeNorm left in Test to compare against; dev, exp and prod still run one. - **No sparklines on History** until there are enough runs to plot (~14); marked with a `ponytail:` note in `History.vue`. - **The six per-target runs stay sequential.** A matrix job would cut the wall clock from ~26 minutes to roughly 7, but it restructures the job graph and the `gh-pages` deploy, so it is #122 rather than another commit here. - **`DEPLOYMENT_ORDER` stays hardcoded.** Promotion order is a semantic fact about the pipeline that no config file states, so unlike the workflow's target list it cannot be derived from `targets.ini`. What it no longer does is fail quietly: an unrecognised environment used to sort last, which is indistinguishable from being the end of the pipeline, so the Dashboard now names it instead. ## Before merging Nothing outstanding. All four items that were blocking are done: the temporary `push:` trigger is removed, #117 is resolved, the sheet IDs and repository secrets are rotated, and the dashboard has been run once against the new secrets — [run 33469165720](https://github.com/TranslatorSRI/babel-validation/actions/runs/33469165720), whose published report carries 4,152 sheet rows and 313 blocklist rows, which is the evidence that both new IDs resolved. A green tick alone would not have shown that: a nonzero pytest exit is normal here. One thing to watch on the first run after merge: `workflow_dispatch` only starts working once `dashboard.yaml` is on the default branch, so the exit-code triage above has been verified by running the step's script under `bash -e` against stubbed exits, but has never executed on a runner. That applies to the empty-results check as well. ## Follow-on work Filed, on the **Babel Validation v1.0** milestone: - #122 — run the six per-environment test runs as a matrix job (~26 minutes → ~7). - #123 — a shared `?sig=` link silently matches nothing once the environment list changes, because the signature is positional. - #124 — the root README's example pytest transcripts are stale (`configfile: pytest.ini`, old tool versions). - #125 — the run history can be silently truncated: each run rebuilds `history.jsonl` from the *published* copy fetched over HTTP, so one failed or CDN-stale fetch publishes a one-line file, and the deploy replaces the branch rather than appending to it. 🤖 Generated with Claude Code
This was referenced Sep 1, 2026
gaurav
added a commit
that referenced
this pull request
Sep 1, 2026
…kDB and NodeNorm (#20) babel-explorer is a CLI for asking Babel *why* two identifiers are considered the same thing. It reads Babel's intermediate Parquet files through DuckDB and, optionally, enriches the results with labels from NodeNorm. `BabelDownloader` handles caching and freshness, `BabelXRefs` handles querying, `NodeNorm` handles labels, and `cli.py` wires them together with Click. Supersedes #1, which GitHub closed and refused to reopen after this branch's history was rewritten. Closes #12. ## What's here Three commands: - **`xrefs`** — cross-references for one or more CURIEs. `--recurse` expands transitively through a single `WITH RECURSIVE` DuckDB query; `--paths` shows the shortest paths connecting the given CURIEs; `--labels` adds NodeNorm labels and Biolink types. - **`ids`** — identifier records from `Identifiers.parquet`, with `--labels`. - **`test-concord`** — compare a proposed concordance change against NodeNorm's current cliques. `xrefs` and `ids` also take `--format json|tsv|csv` for machine-readable output. `--paths` is console-only, and both of its preconditions — a console format, and at least two CURIEs — are checked before anything is downloaded. Getting either wrong otherwise costs a multi-gigabyte `Concord.parquet` download and a full recursive query before the run is rejected, because `--paths` implies `--recurse`. Failures from the two services this tool talks to are reported as errors rather than tracebacks. `MissingBabelFileError` explains that a release does not publish the DuckDB files; `requests.RequestException` reaching the top means NodeNorm, since the downloader handles its own network failures. ## Configuring which Babel release to query A Babel release is addressed as a **releases directory plus a version**, which is how both the public and internal trees are actually laid out — one subdirectory per release, plus a `latest/` symlink: | Variable | CLI option | Default | |---|---|---| | `BABEL_RELEASES_URL` | `--babel-releases-url` | `https://stars.renci.org/var/babel/` | | `BABEL_VERSION` | `--babel-version` | `latest` | | — | `--babel-url` | *(overrides both)* | Pinning a release is a one-word change rather than a URL edit, which matters because pinning is the honest fix for a NodeNorm version mismatch. Precedence runs **flag > environment variable > `.env` > built-in default**, and only public URLs are committed. `--babel-url` takes a complete URL for a tree that does not follow that layout. It is **command-line only, with no `envvar=`, deliberately**: two variables already feed the composed URL, and a third that silently outranked both would make "which release am I actually querying?" unanswerable from the environment alone. `cli()` warns if the pre-refactor `BABEL_URL` is still set, so a stale `.env` fails loudly instead of silently pointing somewhere else. `compose_babel_url` lives in `core/downloader.py` rather than `cli.py` because `tests/constants.py` needs the identical composition and must not import Click to get it. The committed template is `env.default` — visible in a plain `ls`, unlike a dotfile. ## Babel version handling The release is resolved from `VERSION.txt`, falling back to the final URL path segment — which under this scheme is exactly `BABEL_VERSION`, so a pinned release still resolves when `VERSION.txt` is unreachable, while `latest` yields `None` as before. `BABEL_LOCAL_DIR` holds one release at a time. When it changes, only `last_checked` is cleared from the `.meta` sidecars, so the existing ETag path re-checks each file and re-downloads only what actually changed — the Parquet files are never deleted, and an unchanged file costs one HEAD rather than a fresh multi-gigabyte download. The `.babel-version` marker records the release the server *resolved* to rather than the one requested, so `latest` and an equivalent pinned version share a cache instead of thrashing it. That marker is written **after** the cache catches up, not when the change is spotted. It claims "the local cache holds this release", which is only true once every cached file has been re-validated against it, so `_write_version_marker_if_synced()` stamps it once no `.meta` sidecar is still missing its `last_checked`. Writing it up front would leave a run interrupted between `Concord.parquet` and `Identifiers.parquet` with a marker naming the new release over a half-old cache, and the next run would see it match and skip the refresh entirely. The cost is that a cached file nobody asks for holds the marker back indefinitely, at one HEAD per run; that is the honest answer, since the file really is still from the previous release. `--check-download never` suppresses re-checks *within* a release, not across one. `_is_within_freshness()` therefore tests for a missing `last_checked` **before** the `float("inf")` shortcut — reversed, `never` would hand back the previous release's Parquet with no network call at all, under a marker naming the new release, and nothing would ever notice. `--labels` refuses to mix a NodeNorm built from one release with cross-references from another, since the result would be silently wrong rather than obviously wrong. `--allow-version-mismatch` overrides it; pinning `BABEL_VERSION` fixes it properly. `--recurse` never consults NodeNorm, so it does not trigger the check. ## Partial downloads cannot corrupt the cache A corrupt Parquet here is *permanent*: whatever lands on disk gets stamped with the correct remote ETag and then passes every later freshness check. Five routes to that are closed: - A `.tmp` left by a killed process is discarded before each download rather than resumed. Resume is by byte offset, the only way to reach the download at all is that the remote bytes changed, and an orphaned `.tmp` carries no record of which version its bytes came from. Cleanup catches `BaseException`, so Ctrl-C leaves nothing resumable behind. - In-run resumes send `If-Range`, so a file rebuilt mid-download restarts instead of splicing two versions together. - HTTP 416 counts as "already complete" only once the local size matches the remote `Content-Length` — 416 is also what a server returns when the file *shrank* below the resume offset. - A stream ending short of `Content-Length` raises `IncompleteDownloadError` and is retried, rather than being promoted as complete. - A failed HEAD returns "unknown", not "unchanged", and no longer refreshes `last_checked`. One flaky HEAD could otherwise pin the previous release's Parquet as freshly validated for the whole freshness window. `.tmp` files are deleted in two places on purpose; `CLAUDE.md` records which one is the safety guarantee and which is housekeeping, so neither gets removed later as redundant. ## Querying DuckDB connections are ephemeral and in-memory, but "in-memory" is not "touches no disk": a larger-than-memory query spills, and DuckDB's default `temp_directory` is `.tmp` in the *current working directory*. The recursive expansion materialises the whole Concord relation, so every connection goes through `BabelXRefs._connect()`, which points `temp_directory` at `<BABEL_LOCAL_DIR>/duckdb-spill/` — the directory the user already chose to hold multi-gigabyte files, rather than wherever they happened to be standing. ## What it deliberately does not do - **Resume a download across runs.** A leftover `.tmp` cannot be proven to belong to the file being fetched, so it is discarded. Making it safe means persisting the validator alongside the `.tmp`; scoped in #15. - **Enforce the no-private-URLs rule outside `env.default`.** `TestCommittedConfigTemplate` guards the template, but the original leak came through a default value in `cli.py`. A tree-wide scan is #25. - **Reuse a DuckDB connection across queries.** A known performance cost, deferred to #13. (Batching NodeNorm lookups *did* ship — `normalize_curies()` collapses a clique into one request per 100 CURIEs, which is why this closes #12.) ## Known limitations at v0.1.0 This ships knowingly non-functional against its own defaults, so people can try the tool now rather than after the Babel side catches up. The CLI says so clearly rather than failing mid-download. - Public Babel releases do not publish `duckdb/Concord.parquet` or `duckdb/Identifiers.parquet`. **#16 — must close before v1.0.0.** - NodeNorm dev reports Babel `2025sep1` against public `2025dec11`, so `--labels` fails the skew check against the defaults. #17 - CI therefore cannot run the 28 Parquet integration tests. #18 - NodeNorm integration tests fail rather than skip when the API is unreachable. #19 ## Testing 261 unit tests and 13 NodeNorm integration tests pass. 28 Parquet-dependent integration tests skip without a Babel release publishing the files — that is the expected result, not a broken environment. Verified end to end against a Translator releases tree: the composed URL resolves `2026jul22`, downloads `Concord.parquet`, writes a correct `.babel-version` marker, and returns cross-references for `MONDO:0004979`. The version-marker sequencing was verified separately by simulating a `2025dec11` → `2026jul22` change under `--check-download never`: both Parquet files are re-fetched, and the marker flips only after the second one lands. Note that `not slow` is **not** a promise of "small". `Concord.parquet` is 4.6 GB in `2026jul22` and its tests are not marked `slow`, because excluding them would leave the non-slow integration set covering nothing that touches real data. This matters for sizing #18. ## Nothing is blocking this merge Everything outstanding is tracked in #12, #13, #15, #16, #17, #18, #19 and #25. None of it makes what ships here wrong — #16 and #17 constrain what the defaults can *do*, and both are stated plainly in the README, `CHANGELOG.md` and the CLI's own error messages. <details> <summary><b>Review history</b> — three review rounds and a history rewrite. Kept for anyone tracing why a particular line looks the way it does; the durable conclusions are in the code comments, CLAUDE.md and the sections above.</summary> **First review round** found six defects, each fixed in its own commit with a regression test: `--recurse` triggered the NodeNorm version check it no longer needed; `ids --labels` silently dropped the NodeNorm label because Identifiers.parquet's own `label` column overwrote it; a version change forced a full re-download instead of an ETag re-check; a stale `.tmp` could splice two releases into one corrupt Parquet; the HTTP 416 fast path persisted the error response's headers as the file's metadata; and the integration skip probe did not normalise a slashless `BABEL_URL`, so it probed `.../latestduckdb/...` and silently skipped the entire integration suite. **Second review round** found six more, all but one in the downloader's resume path — the five now described under "Partial downloads cannot corrupt the cache", plus `record_to_dict` dropping Identifiers.parquet's own `label` column whenever it was empty, because the omit-when-absent rule matched on a `label` name suffix rather than the three NodeNorm-derived field names. That made `label` present on some rows of a json/tsv/csv run and missing on others. The first round's `.tmp` fix was narrower than it looked: it swept `.tmp` files only on a Babel *version* change, which does not cover a content change within one release or a rebuild in place. The second round replaced it with an unconditional discard before every download. Both deletes are kept, for different reasons, which is why `CLAUDE.md` spells out which is which. **Copilot review** found four threads and one suppressed comment, all five real. Two were in the resume path and are folded into the rules above: a retry sent a bare `Range` whenever no validator was known, which is exactly the case where a splice cannot be detected afterwards; and a 416 whose HEAD carried no `Content-Length` was read as "already complete" when it is equally the answer for a file that shrank. Two existing tests had encoded the first behaviour by seeding a partial file with no validator — a state `get_downloaded_file` never produces — and were rewritten to reach the resume the way production does. The others: ten `get_curie_xref.cache_clear()` calls left over from an `lru_cache` that no longer exists, raising `AttributeError` in integration tests that skip and so never ran; `BABEL_ALLOW_VERSION_MISMATCH` missing from `env.default`, which survived because the test guarding that rule listed the settings by hand rather than reading them off the CLI; and this PR's own batched NodeNorm lookups still listed as future work. **Third review round** found seven, in five commits. Two were the same defect from opposite ends and are the reason the version-marker rules above are stated so explicitly: `_is_within_freshness()` returned `True` on `float("inf")` before looking at `last_checked`, so the whole cross-release refresh was a no-op under `--check-download never`; and the marker was stamped before anything had been re-downloaded, so an interrupted refresh looked complete to the next run. Both let `Concord.parquet` and `Identifiers.parquet` be read together across two Babel builds, which is precisely what the marker exists to prevent. The other three code findings: `xrefs --paths` rejected a single CURIE only inside `_print_paths`, after the 4.6 GB download; NodeNorm's deliberately-uncaught HTTP errors reached the user as a stack trace; and DuckDB spilled into the working directory. Two were documentation drifting from the code — `CLAUDE.md` described `@functools.lru_cache` in three places where none exists (disk caching by ETag, `cached_property`, and per-instance dicts respectively), and `CHANGELOG.md` carried a hard-coded test count that was already wrong, in a repo whose own `CLAUDE.md` says not to record them because they drift silently and then mislead. **History rewrite (2026-09-01).** The internal releases URL had been the hardcoded default from the initial commit until it moved into `.env`, leaving it in 178 of 180 commits across six branches of a public repository. Every commit was rewritten to remove it, verified by four independent checks before anything was pushed, with every changed line confirmed to be that substitution and nothing else. GitHub then refused to reopen the affected PRs, because their original head commits no longer exist — hence #20 here, and #21-#24 replacing #4, #6, #7 and #11. The rewrite does not un-publish anything: GitHub retains pre-rewrite objects reachable by SHA, and the tree the URL pointed at still serves 200 unauthenticated. Both are being handled outside this repo. **Verifying the release caught a stale claim.** `Concord.parquet` is 4.6 GB in `2026jul22`, not the ~626 MB a fixture docstring had claimed, so `pytest -m "integration and not slow"` — documented in two places as avoiding 2 GB+ downloads — would in fact pull 4.6 GB. The docs were corrected rather than the marker, and hard byte figures were dropped from the fixture docstrings for the same reason the repo already gives for test counts: they drift silently and then mislead. </details>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Live-as-you-type NameRes with debounce+abort, preset dropdown (Disease/Gene/SmallMolecule/Custom), latency badges, Solr highlighting, expected-CURIE rank check, and single- or multi-instance views with rank/label/types diff highlighting. State round-trips through the URL for shareable review links.
WIP should be merged after PR #6