Add NodeNorm web frontends: GitHub Pages and Python-based - #4
Closed
gaurav wants to merge 112 commits into
Closed
Conversation
- Add IdentifierRecord dataclass to babel_xrefs.py (resolves TODO) - Add 89 tests across 3 files: test_downloader (26), test_babel_xrefs (31), test_nodenorm (23) - Unit tests (71) use mocks and run without network; integration tests (18) use real downloads/APIs - Add session-scoped fixtures in conftest.py for shared Parquet file downloads - Parametrize integration tests over tests/data/valid_curies.txt for easy expansion - Add integration and slow pytest markers to pyproject.toml - Update CLAUDE.md and README.md with testing documentation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a web UI (FastAPI + Jinja2 + htmx + Bootstrap 5) exposing NodeNorm, XRefs, IDs, and Test Concordance via browser forms, a JSON REST API, and CSV downloads. Launched with `babel-explorer web`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The NodeNorm and Test Concordance pages now show a select dropdown populated from NodeNorm.URLs (defaulting to NodeNorm Dev), with a "Custom URL..." option that reveals a free-text input. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CLI args are forwarded via environment variables so the reloaded subprocess picks up the same config. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add web route table and NodeNorm dropdown details to CLAUDE.md. Add web frontend section to README.md covering startup, REST API examples, and CSV download endpoints. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…D, type fixes - nodenorm.py: Identifier is now frozen=True; rewrite from_dict as one-shot constructor to avoid post-construction mutation of lru_cache'd objects - nodenorm.py: remove **kwargs from get_clique_identifiers — unhashable and unused, would raise TypeError if any kwarg was ever passed - downloader.py: download to .tmp then os.replace() so the final file is never partially written; clean up .tmp on failure - downloader.py: _etag_matches returns True (fail open) on HEAD network error instead of False, avoiding spurious 2GB re-downloads on transient failures - cli.py: add nodenorm_url: str annotation in xrefs and test_concord; move test_concord inline comment to docstring - tests: update test_returns_false_on_request_error → test_returns_true_on_request_error - FUTURE.md: track CLI option deduplication refactor Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix --expand → --recurse (the actual flag name) in Data Flow and Key Design Patterns - BabelXRefs: remove false claim about writing DuckDB databases to disk; all connections are in-memory (duckdb.connect() with no path) - Remove 'Generated DuckDB databases' entry from File Locations (nothing on disk) - Update test count table: numbers were stale and test_cli.py was missing entirely - Add Identifier to Key Dataclasses (now frozen=True as of recent fix) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gaurav
changed the title
- babel_xrefs: dedup by key tuple instead of hashing LabeledCrossReference
(list[str] fields made it unhashable, crashing label_curies=True non-recursive path)
- routes: isinstance(LabeledCrossReference) instead of hasattr("subj_label");
intern NodeNorm instances in app.state.nodenorm_cache so lru_cache persists
across requests with a custom nodenorm_url
- NodeNormApp: hasResults as computed(() => resultsByInstance.size > 0)
so it resets automatically when results are cleared on a new query
- downloader: replace double os.path.exists in __init__ with makedirs+isdir;
replace exists-then-remove TOCTOU with direct remove + FileNotFoundError catch
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…vanced options
- NodeNormApp: wrap form in a query card (card mb-4) and results in a
results card with card-header, card-body, card-footer; move results
heading and ColumnVisibility into card-header; add Download JSON button
in card-footer that exports {queried_curies, instances, results} keyed
by CURIE → instance name
- NodeNormForm: split API options into main (Conflate, Drug/Chemical
Conflate, always visible) and advanced (Description, Individual Types,
Include Taxa) under a <details> collapsible; auto-opens when any
advanced option differs from its default
The two-card pattern (query card / results card) establishes the reuse
convention for future GitHub Pages tools.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The toggle only affected EquivalentIdTable (visible only when a row is
expanded), so from the user's perspective clicking the buttons appeared
to do nothing. Fix: gate the type badge block in ComparisonView's main
summary row with v-if="visibleColumns.has('type')" so toggling Biolink
Type immediately hides/shows the badges in the main results table.
Add 27 tests covering:
- ColumnVisibility: rendering for various Set states, emits, reactivity
when prop is replaced with a new Set instance
- EquivalentIdTable: each column header appears/hides per visibleColumns,
row count, reactivity via setProps
- ComparisonView: type badges present/absent based on visibleColumns,
updates when prop changes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ResultsSummary: rewrite typeCounts to use type[0] per CURIE (most-specific type, de-duped by distinct CURIEs); add selectedTypes prop + toggle-type-filter / clear-type-filter emits; render type badges as clickable buttons with active/inactive styling and a "clear" link - ComparisonView: add typeFilter prop; compute visibleCuries to hide rows not matching the active filter; show empty-state message when all rows are filtered - NodeNormApp: add typeFilter ref; wire toggleTypeFilter/clearTypeFilter; reset filter on new query; pass typeFilter to ComparisonView and selectedTypes to ResultsSummary - Tests: fix 2 ResultsSummary tests that relied on old ancestor-type semantics; add 9 new tests covering filter interaction, button styling, emits, and reactivity in both components Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…/babel-explorer into basic-implementation-in-uv
- list→tuple on Identifier and LabeledCrossReference fields so frozen
dataclasses are hashable (was a TypeError crash in get_curie_xrefs)
- NodeNorm(''): add early return in normalize_curie so empty URL truly
skips all network calls as documented
- BabelDownloader: auto-append trailing slash to url_base so urljoin
can't silently drop path segments
- CI: fix push trigger branch master → main
- Remove dead get_downloaded_dir method (lru_cache + NotImplementedError)
- parse_duration: reject negative values with a clear BadParameter error
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Display unique taxa from equivalent identifiers in the accordion header, controlled by the existing Taxa toggle (now on by default) - Rename "Show columns:" label to "Show:" since toggles now affect both the detail table and the compressed card view Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
NodeNormForm initializes its internal state at creation time, before onMounted fires in the parent. Moving readQueryState() to synchronous setup code ensures the form receives the correct initial values on its first render rather than falling back to the placeholder defaults. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Builds the Astro app and pushes to the gh-pages branch on every push to main. Uses force_orphan so each deploy is a clean slate with no leftover files from previous deployments. Pull request trigger is included temporarily for testing — remove before merging. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rather than showing the full node.type hierarchy (Gene → BiologicalEntity → NamedThing…), compute "direct types" from the unique type values of each equivalent identifier. For non-conflated results this is a single type; for conflated results (e.g. Gene+Protein) it shows 2–5 types in first-appearance order. All type occurrences are now rendered as links to the biolink model docs (https://biolink.github.io/biolink-model/{Type}). The individual_types toggle is removed from the Advanced options UI since the display depends on it always being enabled. ResultsSummary type filter now uses direct types, so conflated CURIEs appear under all their type buckets. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ComparisonView was missed in the previous change: it still used node.type[0] for the type filter and node.type.slice(0,2) for the summary row badges. Now uses getDirectTypes() for both, consistent with the rest of the UI. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Each instance panel in the expanded CURIE detail row now shows a small ↗ link next to the instance name. Clicking it opens the NodeNorm get_normalized_nodes GET response for that specific CURIE and instance in a new tab, using the same API options that were used for the query. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gaurav
force-pushed
the
basic-implementation-in-uv
branch
from
September 1, 2026 05:45
092eaf0 to
b55081f
Compare
This was referenced Sep 1, 2026
gaurav
added a commit
that referenced
this pull request
Sep 1, 2026
…kDB and NodeNorm (#20) babel-explorer is a CLI for asking Babel *why* two identifiers are considered the same thing. It reads Babel's intermediate Parquet files through DuckDB and, optionally, enriches the results with labels from NodeNorm. `BabelDownloader` handles caching and freshness, `BabelXRefs` handles querying, `NodeNorm` handles labels, and `cli.py` wires them together with Click. Supersedes #1, which GitHub closed and refused to reopen after this branch's history was rewritten. Closes #12. ## What's here Three commands: - **`xrefs`** — cross-references for one or more CURIEs. `--recurse` expands transitively through a single `WITH RECURSIVE` DuckDB query; `--paths` shows the shortest paths connecting the given CURIEs; `--labels` adds NodeNorm labels and Biolink types. - **`ids`** — identifier records from `Identifiers.parquet`, with `--labels`. - **`test-concord`** — compare a proposed concordance change against NodeNorm's current cliques. `xrefs` and `ids` also take `--format json|tsv|csv` for machine-readable output. `--paths` is console-only, and both of its preconditions — a console format, and at least two CURIEs — are checked before anything is downloaded. Getting either wrong otherwise costs a multi-gigabyte `Concord.parquet` download and a full recursive query before the run is rejected, because `--paths` implies `--recurse`. Failures from the two services this tool talks to are reported as errors rather than tracebacks. `MissingBabelFileError` explains that a release does not publish the DuckDB files; `requests.RequestException` reaching the top means NodeNorm, since the downloader handles its own network failures. ## Configuring which Babel release to query A Babel release is addressed as a **releases directory plus a version**, which is how both the public and internal trees are actually laid out — one subdirectory per release, plus a `latest/` symlink: | Variable | CLI option | Default | |---|---|---| | `BABEL_RELEASES_URL` | `--babel-releases-url` | `https://stars.renci.org/var/babel/` | | `BABEL_VERSION` | `--babel-version` | `latest` | | — | `--babel-url` | *(overrides both)* | Pinning a release is a one-word change rather than a URL edit, which matters because pinning is the honest fix for a NodeNorm version mismatch. Precedence runs **flag > environment variable > `.env` > built-in default**, and only public URLs are committed. `--babel-url` takes a complete URL for a tree that does not follow that layout. It is **command-line only, with no `envvar=`, deliberately**: two variables already feed the composed URL, and a third that silently outranked both would make "which release am I actually querying?" unanswerable from the environment alone. `cli()` warns if the pre-refactor `BABEL_URL` is still set, so a stale `.env` fails loudly instead of silently pointing somewhere else. `compose_babel_url` lives in `core/downloader.py` rather than `cli.py` because `tests/constants.py` needs the identical composition and must not import Click to get it. The committed template is `env.default` — visible in a plain `ls`, unlike a dotfile. ## Babel version handling The release is resolved from `VERSION.txt`, falling back to the final URL path segment — which under this scheme is exactly `BABEL_VERSION`, so a pinned release still resolves when `VERSION.txt` is unreachable, while `latest` yields `None` as before. `BABEL_LOCAL_DIR` holds one release at a time. When it changes, only `last_checked` is cleared from the `.meta` sidecars, so the existing ETag path re-checks each file and re-downloads only what actually changed — the Parquet files are never deleted, and an unchanged file costs one HEAD rather than a fresh multi-gigabyte download. The `.babel-version` marker records the release the server *resolved* to rather than the one requested, so `latest` and an equivalent pinned version share a cache instead of thrashing it. That marker is written **after** the cache catches up, not when the change is spotted. It claims "the local cache holds this release", which is only true once every cached file has been re-validated against it, so `_write_version_marker_if_synced()` stamps it once no `.meta` sidecar is still missing its `last_checked`. Writing it up front would leave a run interrupted between `Concord.parquet` and `Identifiers.parquet` with a marker naming the new release over a half-old cache, and the next run would see it match and skip the refresh entirely. The cost is that a cached file nobody asks for holds the marker back indefinitely, at one HEAD per run; that is the honest answer, since the file really is still from the previous release. `--check-download never` suppresses re-checks *within* a release, not across one. `_is_within_freshness()` therefore tests for a missing `last_checked` **before** the `float("inf")` shortcut — reversed, `never` would hand back the previous release's Parquet with no network call at all, under a marker naming the new release, and nothing would ever notice. `--labels` refuses to mix a NodeNorm built from one release with cross-references from another, since the result would be silently wrong rather than obviously wrong. `--allow-version-mismatch` overrides it; pinning `BABEL_VERSION` fixes it properly. `--recurse` never consults NodeNorm, so it does not trigger the check. ## Partial downloads cannot corrupt the cache A corrupt Parquet here is *permanent*: whatever lands on disk gets stamped with the correct remote ETag and then passes every later freshness check. Five routes to that are closed: - A `.tmp` left by a killed process is discarded before each download rather than resumed. Resume is by byte offset, the only way to reach the download at all is that the remote bytes changed, and an orphaned `.tmp` carries no record of which version its bytes came from. Cleanup catches `BaseException`, so Ctrl-C leaves nothing resumable behind. - In-run resumes send `If-Range`, so a file rebuilt mid-download restarts instead of splicing two versions together. - HTTP 416 counts as "already complete" only once the local size matches the remote `Content-Length` — 416 is also what a server returns when the file *shrank* below the resume offset. - A stream ending short of `Content-Length` raises `IncompleteDownloadError` and is retried, rather than being promoted as complete. - A failed HEAD returns "unknown", not "unchanged", and no longer refreshes `last_checked`. One flaky HEAD could otherwise pin the previous release's Parquet as freshly validated for the whole freshness window. `.tmp` files are deleted in two places on purpose; `CLAUDE.md` records which one is the safety guarantee and which is housekeeping, so neither gets removed later as redundant. ## Querying DuckDB connections are ephemeral and in-memory, but "in-memory" is not "touches no disk": a larger-than-memory query spills, and DuckDB's default `temp_directory` is `.tmp` in the *current working directory*. The recursive expansion materialises the whole Concord relation, so every connection goes through `BabelXRefs._connect()`, which points `temp_directory` at `<BABEL_LOCAL_DIR>/duckdb-spill/` — the directory the user already chose to hold multi-gigabyte files, rather than wherever they happened to be standing. ## What it deliberately does not do - **Resume a download across runs.** A leftover `.tmp` cannot be proven to belong to the file being fetched, so it is discarded. Making it safe means persisting the validator alongside the `.tmp`; scoped in #15. - **Enforce the no-private-URLs rule outside `env.default`.** `TestCommittedConfigTemplate` guards the template, but the original leak came through a default value in `cli.py`. A tree-wide scan is #25. - **Reuse a DuckDB connection across queries.** A known performance cost, deferred to #13. (Batching NodeNorm lookups *did* ship — `normalize_curies()` collapses a clique into one request per 100 CURIEs, which is why this closes #12.) ## Known limitations at v0.1.0 This ships knowingly non-functional against its own defaults, so people can try the tool now rather than after the Babel side catches up. The CLI says so clearly rather than failing mid-download. - Public Babel releases do not publish `duckdb/Concord.parquet` or `duckdb/Identifiers.parquet`. **#16 — must close before v1.0.0.** - NodeNorm dev reports Babel `2025sep1` against public `2025dec11`, so `--labels` fails the skew check against the defaults. #17 - CI therefore cannot run the 28 Parquet integration tests. #18 - NodeNorm integration tests fail rather than skip when the API is unreachable. #19 ## Testing 261 unit tests and 13 NodeNorm integration tests pass. 28 Parquet-dependent integration tests skip without a Babel release publishing the files — that is the expected result, not a broken environment. Verified end to end against a Translator releases tree: the composed URL resolves `2026jul22`, downloads `Concord.parquet`, writes a correct `.babel-version` marker, and returns cross-references for `MONDO:0004979`. The version-marker sequencing was verified separately by simulating a `2025dec11` → `2026jul22` change under `--check-download never`: both Parquet files are re-fetched, and the marker flips only after the second one lands. Note that `not slow` is **not** a promise of "small". `Concord.parquet` is 4.6 GB in `2026jul22` and its tests are not marked `slow`, because excluding them would leave the non-slow integration set covering nothing that touches real data. This matters for sizing #18. ## Nothing is blocking this merge Everything outstanding is tracked in #12, #13, #15, #16, #17, #18, #19 and #25. None of it makes what ships here wrong — #16 and #17 constrain what the defaults can *do*, and both are stated plainly in the README, `CHANGELOG.md` and the CLI's own error messages. <details> <summary><b>Review history</b> — three review rounds and a history rewrite. Kept for anyone tracing why a particular line looks the way it does; the durable conclusions are in the code comments, CLAUDE.md and the sections above.</summary> **First review round** found six defects, each fixed in its own commit with a regression test: `--recurse` triggered the NodeNorm version check it no longer needed; `ids --labels` silently dropped the NodeNorm label because Identifiers.parquet's own `label` column overwrote it; a version change forced a full re-download instead of an ETag re-check; a stale `.tmp` could splice two releases into one corrupt Parquet; the HTTP 416 fast path persisted the error response's headers as the file's metadata; and the integration skip probe did not normalise a slashless `BABEL_URL`, so it probed `.../latestduckdb/...` and silently skipped the entire integration suite. **Second review round** found six more, all but one in the downloader's resume path — the five now described under "Partial downloads cannot corrupt the cache", plus `record_to_dict` dropping Identifiers.parquet's own `label` column whenever it was empty, because the omit-when-absent rule matched on a `label` name suffix rather than the three NodeNorm-derived field names. That made `label` present on some rows of a json/tsv/csv run and missing on others. The first round's `.tmp` fix was narrower than it looked: it swept `.tmp` files only on a Babel *version* change, which does not cover a content change within one release or a rebuild in place. The second round replaced it with an unconditional discard before every download. Both deletes are kept, for different reasons, which is why `CLAUDE.md` spells out which is which. **Copilot review** found four threads and one suppressed comment, all five real. Two were in the resume path and are folded into the rules above: a retry sent a bare `Range` whenever no validator was known, which is exactly the case where a splice cannot be detected afterwards; and a 416 whose HEAD carried no `Content-Length` was read as "already complete" when it is equally the answer for a file that shrank. Two existing tests had encoded the first behaviour by seeding a partial file with no validator — a state `get_downloaded_file` never produces — and were rewritten to reach the resume the way production does. The others: ten `get_curie_xref.cache_clear()` calls left over from an `lru_cache` that no longer exists, raising `AttributeError` in integration tests that skip and so never ran; `BABEL_ALLOW_VERSION_MISMATCH` missing from `env.default`, which survived because the test guarding that rule listed the settings by hand rather than reading them off the CLI; and this PR's own batched NodeNorm lookups still listed as future work. **Third review round** found seven, in five commits. Two were the same defect from opposite ends and are the reason the version-marker rules above are stated so explicitly: `_is_within_freshness()` returned `True` on `float("inf")` before looking at `last_checked`, so the whole cross-release refresh was a no-op under `--check-download never`; and the marker was stamped before anything had been re-downloaded, so an interrupted refresh looked complete to the next run. Both let `Concord.parquet` and `Identifiers.parquet` be read together across two Babel builds, which is precisely what the marker exists to prevent. The other three code findings: `xrefs --paths` rejected a single CURIE only inside `_print_paths`, after the 4.6 GB download; NodeNorm's deliberately-uncaught HTTP errors reached the user as a stack trace; and DuckDB spilled into the working directory. Two were documentation drifting from the code — `CLAUDE.md` described `@functools.lru_cache` in three places where none exists (disk caching by ETag, `cached_property`, and per-instance dicts respectively), and `CHANGELOG.md` carried a hard-coded test count that was already wrong, in a repo whose own `CLAUDE.md` says not to record them because they drift silently and then mislead. **History rewrite (2026-09-01).** The internal releases URL had been the hardcoded default from the initial commit until it moved into `.env`, leaving it in 178 of 180 commits across six branches of a public repository. Every commit was rewritten to remove it, verified by four independent checks before anything was pushed, with every changed line confirmed to be that substitution and nothing else. GitHub then refused to reopen the affected PRs, because their original head commits no longer exist — hence #20 here, and #21-#24 replacing #4, #6, #7 and #11. The rewrite does not un-publish anything: GitHub retains pre-rewrite objects reachable by SHA, and the tree the URL pointed at still serves 200 unauthenticated. Both are being handled outside this repo. **Verifying the release caught a stale claim.** `Concord.parquet` is 4.6 GB in `2026jul22`, not the ~626 MB a fixture docstring had claimed, so `pytest -m "integration and not slow"` — documented in two places as avoiding 2 GB+ downloads — would in fact pull 4.6 GB. The docs were corrected rather than the marker, and hard byte figures were dropped from the fixture docstrings for the same reason the repo already gives for test counts: they drift silently and then mislead. </details>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a web interface to babel-explorer, exposing all four tools — NodeNorm, XRefs, IDs, and Test Concordance — through a browser UI, a JSON REST API, and CSV downloads. It also includes the full CLI implementation, core query engine, and comprehensive test suite that the web frontend builds on.
WIP. TODO:
python/?) for better organization, or at the least excludingweb/from being packaged into a future PyPI package.Why FastAPI?
We considered several options for the web framework:
What's included
How to test