diff --git a/.claude/development-notes/brainstorming.md b/.claude/development-notes/brainstorming.md index af2618a..03c50fb 100644 --- a/.claude/development-notes/brainstorming.md +++ b/.claude/development-notes/brainstorming.md @@ -256,3 +256,33 @@ The catalogue's two headers already separate layout from content — `#!index-fo After 3.0.0, and nothing about it is urgent while Z is the only person developing modules. **One thing gets harder by waiting, and the cost of getting it wrong is bounded.** `MODULE_INDEX_FORMAT="1"` compiles into every released copy and `require_module_index_format` is an exact-match refusal, so catalogue columns added later — `frame` and `environment`, which is what would let the repository hand an older pipeline a version that still runs on it — are unreadable by 3.0.0. The consequence is not data loss or a broken install: a 3.0.0 user who wants a module published later is told to upgrade. That may simply be acceptable for the first release of the module system, and it is Z's call rather than a deadline anyone has to meet. + +--- + +## The catalogue's columns are matched by name and read by position — Z, 2026-09-10 + +Added when the `kind` column landed and cost thirteen failing cases. **Z deferred it explicitly: "We can look at the name-position thing later. Add it to the list of post v3.1.0 tasks."** + +`index.tsv` was designed so that adding a column is free. Its own header says so, at length: *"THE COLUMNS ARE READ BY NAME, out of the header row below, so their order here does not matter and a column a release has never heard of is ignored rather than misread. Adding one later is therefore safe and needs no layout bump."* That is true, and it is the reason `#!index-format` is not bumped for an addition. + +**It is true of every release except the one doing the adding.** `module_index_rows()` reorders each row into `MODULE_INDEX_COLUMNS` order and joins it with `$'\037'`; every consumer then reads it back with a positional `IFS="$MODULE_INDEX_SEP" read -r name version contract frame environment url sha summary`. So the by-name matching happens once, at the top, and everything downstream is positional. Inserting `kind` second shifted six `read` destructurings, an `awk` filter keyed on `$2`, and a `sort -t"$SEP" -k2,2Vr` that silently stopped sorting by version and started sorting by kind. + +### What it would buy + +A column addition that costs one line instead of thirteen cases. More to the point, **the failure mode is silent where it matters most**: the `sort` did not error, it just returned the wrong newest version. Two of the thirteen failures were assertions going vacuous rather than red, and one of those only announced itself because it carried a `rows > 0` guard. A wrong column is not a crash, it is a plausible answer. + +### The shape it probably wants + +An accessor rather than a destructuring — `row_field "$row" version` reading the same `MODULE_INDEX_COLUMNS` list that built the row. Then a column's position is knowledge held in exactly one place instead of at every call site, and `MODULE_INDEX_COLUMNS` becomes the single declaration it was always meant to be. + +The cost is a subshell per field per row where there is now one `read` per row. `available` renders every row in the catalogue, so this is measurable and worth measuring before committing to it; a catalogue is tens of rows today and the answer may simply be that it does not matter. + +### What it would break + +Nothing published. The wire format is unchanged — this is entirely how the wrapper reads a row it already has. It does not touch `#!index-format`, so no released copy notices. + +**It does touch every consumer at once**, which is exactly the change that wants a full suite behind it rather than a `--case` run. That is the argument for after a release rather than during one. + +### Where it sits + +**After v3.1.0.** Nothing is wrong today: the columns and the destructurings agree, and `00_static` checks every row against the file it advertises. This is paying down the next addition's cost, not fixing a defect — and the next addition is not scheduled. diff --git a/.claude/development-notes/check-split-and-layout.md b/.claude/development-notes/check-split-and-layout.md new file mode 100644 index 0000000..df253f9 --- /dev/null +++ b/.claude/development-notes/check-split-and-layout.md @@ -0,0 +1,58 @@ +# `check install` / `check project`, and the directory move + +**Written 2026-09-10 against the working tree on top of `6d38c88`.** Z drove this one directly, in a series of short instructions; the reasoning below is what each of them settled. + +## The shape problem Z named + +`install/check_install.sh` had grown to check three things: the tools, the `bin/` helpers, and whether `parameters.config` parses — resolving the tool list out of `params.software` when a project was present and falling back to a canonical list when it was not. Taking `parameters.config` out of it (Z: *"We changed how install worked entirely. Checking parameters.config is just going to cause confusion"*) left a hardcoded `CANONICAL` list that duplicated the template's `software` block, and left nothing checking a project at all. + +Z, on being shown that: ***"It is not the error I'm concerned about it is the shape. The tools were checked against the params.config."*** + +Moving `params.software` into `nextflow.config` was tried and **reverted** — Z: ***"Tools remain in params.config."*** The answer was two commands instead: ***"We need a separate check install and check project."*** + +## What each one is + +| | | +|---|---| +| `check install` | the tools a release is built to run, **and that each comes from the release's own conda environment**; every helper in `bin/`. Reads no `parameters.config` and needs no project. | +| `check project` | `parameters.config` is current for this release and parses; `metadata.csv` and the run table parse, through the parsers step 0 uses; and every command **as `params.software` names it**. | + +**A bare `check` is refused.** Z chose this over keeping it as an alias: whichever one it picked would leave the other unchecked while reporting success. `check` is the second subcommand after `analysis` to carry a word of its own, which the argument block at the top of the wrapper had to be taught — every other subcommand takes none, so `check install` was rejected before reaching its arm. + +**`run_check()` is a function and not a nested `case`.** Two suite cases read the wrapper's own case arms and compare them against its usage line; a nested `install)` at the same indentation is read as a top-level subcommand. The top-level usage says `check `, following the `analysis ` convention, because a nested `{install|project}` breaks a flat split on `|`. + +## The environment check, which is the part that was silently wrong + +`check_tool` asked `command -v` and accepted whatever `PATH` returned. **Every tool in `CANONICAL` is pinned in `install/environment.yml`** — verified, all fourteen including `gawk` and `python` — so one resolving from anywhere else means the environment is missing a package and the machine's own copy is standing in, at another version, on this machine only. The run works and reproduces nowhere. + +It now compares each resolved path against `CONDA_PREFIX` and reports `OUTSIDE THE ENVIRONMENT` as a failure. When the environment is not active there is nothing to compare against, and the header says so rather than checking `PATH` and calling that an answer. + +**`check project` deliberately does NOT apply that rule.** Repointing a tool at a system binary is a thing a project is allowed to do, and that check is where you see the result. + +**A fixture that disabled the check under test.** The first version of these cases named the fake environment directory `env` while passing `ENV_NAME=check-install-env`; the script only compares paths when `basename $CONDA_PREFIX` matches `ENV_NAME`, so the comparison was off and both cases passed over nothing. The fixture directory is named for the environment now, and the passing case asserts the `from ` header line — which appears only when the script decided it knows which environment it is in, and is therefore what stops the case going vacuous again. + +## The directory move + +Z: ***"We also should move check_install, check_analysis_install and check_projects to bin"*** and ***"install only contains environment yml files. citations and bib file should move to another folder citations."*** + +``` +bin/ every script that is RUN rather than sourced, the three checkers included +lib/ every file that is SOURCED +install/ the two pinned environment files, and nothing else +citations/ the pipeline's own references.bib and the citations.json generated from it +``` + +`analysis/` keeps its own `citations.json` and `references.bib`, and so does each module — **every separately publishable unit keeps its bib beside it**. The pipeline's pair had no home but `install/`, which is what `citations/` fixes. + +Two things the move broke and how: + +- **`check_install.sh` enumerates `bin/*` as pipeline helpers**, so it would have reported itself and its two siblings as helpers a run depends on. The three are skipped by name. +- **`07_analysis_frame`'s one-way-dependency case** greps `bin/` for any mention of the analysis layer, and `check_analysis_install.sh` is full of them. It is excluded **by name**, not by pattern, and a second case asserts exactly one file matches `bin/check_analysis*` — otherwise a rename or a second script widens the exclusion silently, since `--exclude` takes a glob and a name matching nothing is not an error. + +`citations` was added to `PAYLOAD_ITEMS`, to the pipeline sandbox's copy list, and to `00_static`'s archive case — which now also asserts four **named files** and not only their directories, because a directory traveling empty satisfies every `assert_contains` on a path prefix. + +## State when this was written + +`--cost static` 272 passing, `--fast` 322 passing, `nextflow lint` 32 files clean, manual 41 pages / 361 anchors, language sweep clean, every analysis version current. + +**`dev/scripts/verify-archive.sh` defaults to `HEAD`**, so it passed against the old layout and proves nothing about this one until the move is committed. Re-run it then. diff --git a/.claude/development-notes/module-optionality.md b/.claude/development-notes/module-optionality.md new file mode 100644 index 0000000..3eb1151 --- /dev/null +++ b/.claude/development-notes/module-optionality.md @@ -0,0 +1,74 @@ +# Modules are optional — the design change, and the audited blast radius + +**Written 2026-09-10 against the tree at `91e027f` plus three uncommitted edits.** v3.0.0 was released earlier the same day, *with* the modules inside it. Nothing here is a defect in the released version: v3.0.0 works exactly as shipped. This is about what the NEXT release has to be, and why. + +## Z's ruling + +Z, 2026-09-10: *"The modules should not ship with the release. That's the whole idea."* And, when the analysis environment came up: *"The whole idea was to make the modules optional."* + +Three parts follow from it: + +1. **No module ships inside a release tarball.** The store starts empty; every module is installed from the catalogue. +2. **Each module declares its own conda pins** in its manifest, and `modules install` puts them into the shared analysis environment — the E8 machinery, which already exists. +3. **The baseline analysis environment slims** to what the FRAME needs, and stops carrying module dependencies. + +## What was actually wrong, stated precisely + +**The architecture was never wrong. The data was.** + +The per-module conda machinery is built and was proven against real conda on 2026-09-10 by `dev/scripts/check-module-packages.sh` — every check `ok`: a fixture module's pins installed under `--freeze-installed`, R imported them, a second module sharing a pin moved nothing, uninstalling one left the other's packages alive, and the baseline never lost a package. `PoolSeqFlow:786-794` is the deferred install; `:1433` is the reconcile that `analysis install` performs over modules already in the store. Neither is missing anything. + +**The three shipped modules declare `packages: []`.** That was TRUE while they shipped inside the release, because the same release shipped a 191-package environment holding everything they need. It became FALSE the moment they were published independently — which happened on 2026-09-09/10. They were never really modules; they were release components wearing a module's shape, and they are the one case that never exercised the machinery built for them. + +## The measured package split + +Evidence: every `library()`/`requireNamespace()`/`pkg::` reference across `analysis/lib/` and `analysis/modules/`. + +| | | +|---|---| +| the frame uses | `jsonlite`, `knitr`, `rmarkdown` — plus `pandoc` and `typst` for the PDF report | +| the modules use | `doFuture`, `ggplot2`, `Rcpp`, and `data.table` (basicstats only) | +| used by nothing at all | `r-optparse`, `r-pheatmap` — pinned since the first guess at the E4c roster | + +`foreach` and `future` arrive as `doFuture` dependencies. **The audit corrected two of my assumptions**: `mds` DOES need `ggplot2` unconditionally, and the compiled path needs the conda C++ toolchain, which `r-rcpp` does not pull — so the toolchain placement is part of the split, not incidental. + +## The audited blast radius — seven seams, 2026-09-10 + +A seven-dimension audit with adversarial verification of every blocker and major finding. **Five verifier verdicts, zero refutations or corrections** — the findings below survived independent checking. + +### Blockers + +- **The clone-install route still ships all three modules.** `.gitattributes export-ignore` governs the TARBALL; `install` copies from a checkout tree where `analysis/modules/*` still exist. "The store starts empty" is false for every user following the manual's clone route. **This is the one I would have missed.** +- **The three published catalogue rows become install-clean, fail-at-first-run.** Their `environment=3.0.0` passes the `version_at_most` check on any later release, their manifests declare no packages, so they install cleanly onto a slim baseline and then die missing `ggplot2`/`doFuture`/`Rcpp`. The `environment` field's semantics assume baselines only ever GAIN packages. +- **`00_static`'s `no package leaves a shipped environment file` cannot legitimately pass a slimmed baseline** — and it is the guard added on 2026-09-09 for exactly the typst class of bug. It compares against the last release tag, so it stays red for the whole dev cycle. `export-environment.sh --allow-removals` is the sanctioned escape for the export; the test needs its own answer. +- **`00_static` asserts every repo manifest's `environment` EQUALS the release version.** With no module shipping, that coupling is backwards — it forces a bump nobody's needs justify. +- **The manual teaches the old design in at least three places**: "Four ship with the release", the `# Shipped Modules` section, and "No module shipped with this release names one \[a package\]". +- **`RELEASING.md` step 6**'s justification for automatic manifest rewriting — "travels in the same tarball as the analysis environment it names" — is exactly the premise being removed. + +### Major + +- **`bump-version.sh` now writes a wrong claim.** Automatically setting `environment := new release` is right for a shipped module and wrong for an independent one, whose minimum should move only when its needs do. Landed 2026-09-10 at Z's request; the request was correct under the old design. +- **No gate checks that a module's declared packages cover what its R actually loads.** Under-declaration is silent — and the three published tarballs declare NONE. This is the gate that would have caught tonight. +- **The module suites SKIP rather than fail when `TEST_ANALYSIS_ENV` lacks a package**, so a slim environment silently retires the compiled and parallel coverage instead of reporting it. +- **`PoolSeqFlow analysis cite` cites zero R packages today** — a live bug, independent of any of this, in how the cite arm reads `analysis_r_packages`. +- **`analysis check` would report "All checks passed" on an environment missing every module-declared package**, because it only verifies the baseline. +- **The slim baseline list must add `python3` and `rsync`** (the frame invokes both) and probably drop `samtools`/`bcftools`/`htslib`, which the audit found nothing in the frame invoking — verify before acting. +- **Module package arrays must be name-disjoint from the baseline's set**, or `export-environment.sh`'s refusal fires permanently on a name the baseline owns. +- **Two tests fail against the uncommitted wrapper edit**: `02_launcher`'s modules-list case and `test_modules_list_reports_the_store` both plant a store module with no `.source`, which now prints `ships with this release`. +- **`test_an_unknown_module_refuses_before_any_task`** asserts the literal roster `Available here: association, basicstats, mds, verify`, read from the store at runtime — it only holds while the checkout's store has the three. +- **The store README** contradicts the new design in three places, including rule 6e: "Leave the field out when the release's own environment suffices — which is true of every module shipped here". +- **Offline/air-gapped installs regress**, which matters because this runs on HPC clusters: today a cluster user gets three working modules in the tarball; afterwards they need HTTPS to the catalogue AND to each tarball. A `file://` catalogue does not solve it — the rows carry `https` URLs independently of how the index was fetched. + +## The uncommitted work, and what to do with it + +Three files, all on `dev`, none committed: + +- **`.gitattributes`** — `analysis/modules/*/ export-ignore`, keeping `README.md`. **Correct and verified**: `git archive --worktree-attributes` leaves only `analysis/modules.nf` and the README. +- **`dev/scripts/verify-archive.sh`** — `excluded` widened to `analysis/modules/*/`, and the positive assertion strengthened to "no module directory in the archive AND the README is". **Correct**; `00_static` passes at 46 against the working tree. +- **`PoolSeqFlow`** — `module_was_installed()` plus `ships with this release` labels and the uninstall warning. **This encodes the ABANDONED middle design** and should probably be reverted: once nothing ships, everything in the store was installed, and the label distinction is dead weight. It also breaks two tests. `.source` remains a useful marker; the labels built on it do not. + +## Sequencing, when this is picked up + +The blockers interlock, so order matters. The clone-install path has to be settled before "the store starts empty" is true for anyone. The published catalogue rows need a decision — remove them, or republish at bumped versions declaring their packages — before the baseline slims, or a next release strands anyone who installed one. The two `00_static` gates need their own answers before the slim can be committed at all, or the suite is red for the whole cycle. + +Estimated at roughly one working day, about half of it unattended suite and environment runs. diff --git a/.claude/development-notes/modules-and-libraries.md b/.claude/development-notes/modules-and-libraries.md new file mode 100644 index 0000000..8caddd9 --- /dev/null +++ b/.claude/development-notes/modules-and-libraries.md @@ -0,0 +1,115 @@ +# Modules and libraries are optional, and installed — the rework + +**Written 2026-09-10 against the tree at `6d38c88`.** Supersedes `module-optionality.md`, which recorded the audit that led here; that note's findings are all addressed below or listed as still open. v3.0.0 was released the same morning WITH the modules inside it, so everything here describes the release after it. + +## The root cause, in one line + +Z, 2026-09-10, on seeing that `analysis/modules/` was both the git source directory and the install store: ***"That was the whole problem."*** + +Every symptom traced to that: the three modules shipped in the tarball because they were sources sitting in the install path; `modules list` reported them `installed` because the store contained them; the analysis environment carried their dependencies because they were release components. **One cause, four symptoms.** + +## The layout now + +| | | +|---|---| +| `modules//` | module sources, tracked, `export-ignore`d from releases | +| `modules/lib//` | library sources, same | +| `modules/repo/` | the published catalogue and tarballs | +| `analysis/` | the frame ALONE — `lib/nf/`, `lib/rmd/`, the entry pieces | +| `analysis/modules/` | THE STORE. Gitignored, empty in a checkout, empty in a fresh install | +| `analysis/modules/lib/` | the library store, under a name no module may take | + +**The repo path and the install path deliberately differ.** A module's frame import is `'../../lib/nf/plan.nf'`, correct from the store and meaningless from `modules//`. I first "fixed" that by moving the store to `$INSTALL/modules/` so the two matched — **wrong**, because the store would then land on the sources in a checkout install. Z caught it. `00_static` lints modules in an assembled store layout instead. + +**The published URL did NOT move** when `modules-repo/` became `modules/repo/`. `MODULE_INDEX_URL` compiles into v3.0.0 and asks for `/PoolSeqFlow/modules-repo/` for as long as that release exists. `build_docs.py` and `publish-module.sh` each carry two constants — source directory and published path — with a comment saying why. + +## Libraries + +Eleven files in `analysis/lib/R` + three `.cpp` became **five libraries**, grouped by what a module must take together: + +`n_eff` (n_eff, pool_n_eff, harmonic_mean) · `chunk_ranges` · `allele_frequencies` (+cpp) · `site_diversity` (+cpp) · `nei_distance` (nei_distance, add_distance, mean_distance, +cpp) + +**Measured, not assumed:** no library calls another. Every apparent dependency was a comment reference. Z's correction stands anyway — ***"They are standalone now. They can be used by other modules as we increase the size. Don't make assumptions."*** — so the manifest carries `libraries` regardless and resolution is transitive. + +**Two files had no consumer and moved to `test/tools/`**, on Z's rule *"If only test suite is using it it should be in test suite not here"*: `split_counts.R` and `pool_sensitivity.R`. Neither is dead — `split_counts` is the **oracle** the vectorized and compiled paths are checked against (mutating its split character fails 19 of 140 checks), and `pool_sensitivity` duplicated `poolSensitivity()` in `analysis/lib/nf/pools.nf`, which is the one modules actually read off `target.pools`. + +**`libraryFiles()` and `compiledFiles()` are gone from every `main.nf`.** The frame resolves both from the manifest — `moduleLibraryFiles(name)`, `moduleCompiledFiles(name)` — so the list exists once. `00_static:707` used to police the duplicate; there is no duplicate now. + +## Packages: declare everything + +Z: ***"Modules declare everything including doFuture, ggplot2 and data.table."*** And the rule that makes it safe, in Z's words: *"to uninstall take a diff between the uninstalled package and union of all installed packages + analysis base environment."* + +**That was a live bug.** `uninstall_module` computed `going = mine − others` with **no baseline subtraction**, so a module declaring `r-ggplot2` would have stripped ggplot2 out of the shared environment on its way out, breaking the frame and every module beside it. Now `going = mine − (others + baseline)`, with `baseline_packages()` reading the shipped `environment-analysis.yml` — the shipped file, not the live environment, which has module additions merged in and cannot say which packages are the release's own. + +`doFuture`, `ggplot2`, `data.table` and `Rcpp` **stay in the baseline** — Z: *"Rcpp should remain in the frame. I settled this one before."* Modules declare them anyway. A library declares `r-rcpp` only if it ships a `.cpp`. + +## The catalogue + +Gained a **`kind` column** (`module` | `library`), additive and matched by name, so an older release ignores it and an empty value means `module`. `available` lists modules only; a library is never asked for by name. + +**Every positional consumer had to move with it** — six `read` destructurings, an awk filter, and a `sort -k2,2Vr` that was then sorting by kind instead of version. Thirteen cases failed and pointed straight at them. **The columns are matched by name; the consumers destructure positionally. Adding a column is not free.** + +## Bugs made and caught while building this + +- **`def LIBRARY_DIR = 'lib'` at the top level of a `.nf`** — the strict parser allows only declarations there. It is `libraryDirName()`. +- **`git log -1 --format=%ct `** returns nothing: a tree has no commit and no date. `--mtime="@"` made tar substitute a nonsense year. `publish-module.sh` now **refuses** rather than publishing something unreproducible — which is why publishing must follow a commit. +- **A `sed 1,/^-->$/d`** whose anchor was not at line start deleted the entire release-notes boilerplate. Caught by reading the output; the new case mutation-tests exactly that. +- **`select-tests.py` listed files from the git index**, so a moved file still counted as a source. I masked it by staging before noticing. It now lists only what exists. +- **My own `every catalogue row` case went vacuous** for the positional reason above — and the `rows > 0` guard I had put in it made it **skip loudly** instead of passing. + +## What is still open + +- **Nothing is published.** The three module tarballs carry `20260910.001` manifests with no `libraries` and no `packages`; the repo is at `.002`. The five libraries have never been published. `RELEASING.md` step 10 is the procedure. +- **The JVM and pipeline tiers have never run against this layout.** `moduleLibraryFiles()` resolving a module's libraries at run time is new code exercised only statically. Z: *"we should not run the full suite until we are done with the work today."* +- **The tarball extract/install/check step** (Z's step 3) is unbuilt. A spec exists and came back with four blockers including two vacuous assertions; it needs rewriting, not applying. +- **The docs pass is partial.** The manual's roster, modules section, package paragraph, store-wipe paragraph and library references are corrected; `modules/README.md` gained a libraries section and rule 6e was inverted; `RELEASING.md` gained step 10. **An exhaustive read of the manual was still running when this was written** — the directory-layout diagram and the Upgrading section are known to be untouched. + +--- + +# The follow-up pass, 2026-09-10, against `6d38c88` plus the working tree + +Written when the question "can we publish now?" was asked. The answer was no, and finding out why turned up a class of defect the rework had left behind everywhere. + +## Publishing had to wait, and the frame version is why + +**v3.0.0 compiled `MODULE_INDEX_COLUMNS="name version contract frame environment url sha256 summary"` — no `kind`.** It matches by name, drops what it does not know, and reads all eight rows as modules. Two gates then stand between a v3.0.0 user and a module built for the release after it: `contract`, which is `freq-1` on both sides and passes, and `frame`. + +**`analysis/frame.version` was still `20260908.003`** — the exact value v3.0.0 shipped — after `analysis/lib/R/` and `analysis/lib/cpp/` were deleted out from under it. So the frame gate passed too, and publishing then would have offered every v3.0.0 user a `.002` module that installs cleanly, brings no library (v3.0.0 has no `install_library`), and dies at run time on a `moduleLibraryFiles()` its frame does not define. `install/environment-analysis.yml` is byte-identical to v3.0.0, so `environment: 3.0.0` is honest and cannot refuse anything: **the frame version was the only gate, and it was open.** + +Now `20260910.001`, with all eight manifests declaring it — modules at `.003`, libraries at `.002`. `check-analysis-versions.sh` had been reporting `BEHIND` the whole time; nothing was reading it because it is a release gate rather than a suite case, which is correct and is also how it sat unnoticed for a day. + +## THE CLASS: a glob pointed at the install store passes over nothing + +`analysis/modules/` is gitignored and empty in a checkout. **Every script and case that still globbed it kept working, kept exiting 0, and checked nothing.** Nine of them, and two were release gates: + +| | | +|---|---| +| `check-analysis-versions.sh` | the per-module version loop — **zero modules examined**; a release could ship any manifest unbumped | +| `export-environment.sh` | half of the guard the plan calls *"the sharpest trap in the whole feature"* | +| `check-module-packages.sh` | the only real conda solve of the modules' own pins, asking it of an empty set | +| `bib2citations.py` | `--check` said *"every citations.json matches (2 files)"* — three module bibliographies unread | +| `00_static` ×5 | the DOI check, three suite-list loops, the manifest-anchor glob | +| `07_analysis_frame` | asserted the three modules **ship**, which the rework inverted | +| `08_analysis_rlib` | the no-package grep, which printed `grep: ... No such file` and passed | +| the three module suites | `cat analysis/lib/R/*.R` — **0 passed, 6 failed** in `basicstats` once actually run | + +**The suite agreed with the bug rather than catching it.** Both version fixtures in `00_static` planted their `demo` module *into the store*, so the gate found it there and the case went green over a loop that finds nothing in reality. A fixture that matches the defect is worse than no fixture: it is a standing assertion that the wrong thing is right. + +Guards added so it cannot repeat quietly: a `ghost` module planted in the store that must never be reported, a `helper` library that must be, a `seen > 0` count in the frame's tracked-files loop, and a `fail_case` when `modules/lib` is missing. Each mutation-tested. + +## Two defects in shipped code, found while fixing the gates + +**`module_packages()` and `module_libraries()` did not terminate their last line.** `store_packages` runs one per module and concatenates, so the last spec of one manifest and the first of the next arrive as **one token** — `r-shared=2.0r-shared=2.0`, `chunk_rangesn_eff`. `cut -d= -f1` then keeps the first name and the second is gone from the keep-list, so **`uninstall` removes a package a still-installed module declares.** Reproduced with three modules where the shared package sits at a list boundary: `r-fst` dropped while `gamma` still needed it. Fixed by ending both with `awk -F'"' 'NF > 1 { print $2 }'`, which terminates its records. + +It did not bite the three shipped modules, by luck: their library lists overlap enough that every glued name also appears cleanly somewhere else in the `sort -u`. Two modules with a boundary name in common and nothing else would lose it. + +**`ANALYSIS_ENV_FILE` was defined in `PoolSeqFlow` and read in `lib/wrapper_lib.sh`.** So `baseline_packages()` returned **nothing** for any `dev/` script that sourced the library on its own — silently, through `2>/dev/null` on an empty path — and an empty baseline subtracts nothing. Now defaulted in `wrapper_lib.sh` beside the function that reads it. + +`test_the_keep_list_survives_more_than_one_other_module` in `02_launcher` covers both; the existing case could not, because with a single other module there is no join to get wrong. + +## Still open after this pass + +- **Nothing is published, and publishing waits for the release.** `RELEASING.md` step 10 already puts it after the version bump, and the frame version is why that ordering is load-bearing rather than tidy. +- **The JVM and pipeline tiers still have not run.** `--fast` is 313 passing, `--cost static` 264 — but every Nextflow half is skipped, and `moduleLibraryFiles()` resolving a module's libraries at run time has still never executed. +- **The tarball extract/install/check step** (Z's step 3) is unbuilt. +- **The docs pass.** The four-way audit finished, 39 agents; the manual's own claims about the shared library are corrected in `analysis/references.bib` and regenerated, but the directory-layout diagram and the Upgrading section are untouched. diff --git a/.gitattributes b/.gitattributes index 4132f6f..bf27180 100644 --- a/.gitattributes +++ b/.gitattributes @@ -27,15 +27,16 @@ Project/ export-ignore # be checked against a known dataset, kept out of releases because a user never runs it. test/ export-ignore -# The same, for the cases a module ships with itself. They are the module's and travel with it -# when it is published on its own; inside the pipeline tarball they are what test/ is - files a -# user has no way to run and no reason to. -analysis/modules/*/test/ export-ignore - -# The module catalogue, which `modules available` and `modules install` read over the network -# from this branch. A copy inside a tarball would be a second answer to what can be installed, -# frozen on the day the release was built, and modules are published after it. -modules-repo/ export-ignore +# NO MODULE SHIPS INSIDE A RELEASE. A module is published on its own timetable and installed +# from the catalogue, and that is the whole point of the module system - a release that carried +# three of them would make those three privileged, installed by nobody, and impossible to +# replace with a newer version without first removing what the release provided. +# +# The whole directory: modules and the libraries they install alongside themselves. They are +# sources here, and published artifacts in modules/repo/, which the site serves. A release +# carries neither: a catalogue inside a tarball would be a second answer to what can be +# installed, frozen on the day the release was built. +modules/ export-ignore # How the project is built rather than what it does: the agent's instruction file and the # development notes. Tracked, and linked from the manual's Development section, but a release diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 0d6526a..d8e02c6 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -18,7 +18,7 @@ on: - "manual/**" # The module repository is deployed with the site, so publishing a module is a change # here. Without this line the catalogue would advertise a tarball no deploy had carried. - - "modules-repo/**" + - "modules/repo/**" - "mkdocs.yml" - "CHANGELOG.md" - "dev/scripts/build_docs.py" @@ -26,7 +26,7 @@ on: pull_request: paths: - "manual/**" - - "modules-repo/**" + - "modules/repo/**" - "mkdocs.yml" - "CHANGELOG.md" - "dev/scripts/build_docs.py" diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 1bf42ce..12d04da 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -69,11 +69,10 @@ jobs: fi fi - # Refuses to publish a version the changelog does not describe. - if ! grep -q "^## \[$declared\]" CHANGELOG.md; then - echo "CHANGELOG.md has no '## [$declared]' section" >&2 - exit 1 - fi + # Refuses to publish a version the changelog does not describe. The same extractor + # that writes the release body, so the gate cannot pass over a section the body step + # would then fail to read. + dev/scripts/changelog-section.sh "$declared" > /dev/null echo "version=$declared" >> "$GITHUB_OUTPUT" echo "name=PoolSeqFlow-$declared" >> "$GITHUB_OUTPUT" @@ -101,6 +100,23 @@ jobs: ( cd dist && sha256sum ./*.tar.gz > SHA256SUMS ) ls -lh dist/ + - name: Build the release notes + env: + VERSION: ${{ steps.version.outputs.version }} + NAME: ${{ steps.version.outputs.name }} + run: | + # This version's CHANGELOG section, then the standing tail. Runs on every trigger and + # not only on a tag, so a rehearsal proves the notes extract before the tag exists. + dev/scripts/changelog-section.sh "$VERSION" > dist/RELEASE_NOTES.md + printf '\n' >> dist/RELEASE_NOTES.md + sed -e "1,/^-->$/d" -e "s|@VERSION@|${VERSION}|g" -e "s|@NAME@|${NAME}|g" \ + dev/release-notes-tail.md >> dist/RELEASE_NOTES.md + + [ -s dist/RELEASE_NOTES.md ] || { echo "the release notes came out empty" >&2; exit 1; } + echo "::group::release body" + cat dist/RELEASE_NOTES.md + echo "::endgroup::" + - name: Publish the release if: github.ref_type == 'tag' uses: softprops/action-gh-release@v3 @@ -108,31 +124,4 @@ jobs: files: | dist/*.tar.gz dist/SHA256SUMS - body: | - ## Download and install - - ```bash - curl -LO https://github.com/ozankiratli/PoolSeqFlow/releases/download/${{ github.ref_name }}/${{ steps.version.outputs.name }}.tar.gz - tar -xzf ${{ steps.version.outputs.name }}.tar.gz - cd ${{ steps.version.outputs.name }} - - cp parameters.config.template parameters.config - ./PoolSeqFlow install - ``` - - Then edit `parameters.config` and `metadata.csv` for your data and run - `./PoolSeqFlow run`. - - Verify the download with `sha256sum -c SHA256SUMS`. - - `PoolSeqFlow.tar.gz` is the same archive under a stable name, for - scripted installs: - `https://github.com/ozankiratli/PoolSeqFlow/releases/latest/download/PoolSeqFlow.tar.gz` - - **Upgrading an existing project?** Your `parameters.config` is not - touched by a new version and can be missing parameters this release - expects. Run `./PoolSeqFlow migrate_config` and read what it reports — - see [Upgrading](https://ozankiratli.github.io/PoolSeqFlow/getting-started/upgrading/). - - Full documentation: - Changes in this release: see `CHANGELOG.md`. + body_path: dist/RELEASE_NOTES.md diff --git a/.gitignore b/.gitignore index af4d7e4..b8d9282 100644 --- a/.gitignore +++ b/.gitignore @@ -76,3 +76,9 @@ test/**/*.pyc # Build output from dev/scripts/verify-archive.sh. dist/ + +# THE MODULE STORE. Running ./PoolSeqFlow out of a checkout makes the checkout the installation, +# so `analysis modules install` unpacks into analysis/modules/ here - installed artifacts, not +# sources. The sources live in modules/, which is tracked and export-ignored; this is where they +# arrive after being published and installed back. +analysis/modules/ diff --git a/CLAUDE.md b/CLAUDE.md index 83f236b..9400f26 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -82,7 +82,15 @@ Z, 2026-09-08: *"We keep everything but abandoned ideas. They carry a different Not one line of code may change. Per file, diff the non-comment lines against `HEAD`. For Python, a docstring is not a `#` comment and the line count misleads — prove it with an AST comparison that strips docstrings. -Then `nextflow lint .` (zero errors **and** zero warnings) and `bash test/run_tests.sh --fast`, which is under a minute. Check both counts — files linted, cases passed — against the run before it rather than only the exit status: a filter that matches nothing also reports success. Neither number is written down here, because both move with every file added. +Then lint and `bash test/run_tests.sh --fast`, which is under a minute. Check both counts — files linted, cases passed — against the run before it rather than only the exit status: a filter that matches nothing also reports success. Neither number is written down here, because both move with every file added. + +**Lint the tree without `modules/`**, and expect zero errors and zero warnings: + +``` +nextflow lint analysis analysis.nf dryrun.nf poolseqflow.nf scripts +``` + +`nextflow lint .` **cannot pass and is not the command.** A module's `main.nf` imports the frame as `'../../lib/nf/plan.nf'` — correct from `analysis/modules//`, where it is installed, and unresolvable from `modules//`, where it is written. The path is right and the tree is wrong for it, so linting from the repository root reports one `Invalid include source` per import on every module. **`00_static` is what lints them**: it assembles a store layout in a sandbox and lints that, which is the only place those imports resolve. ## What to run while building @@ -98,16 +106,16 @@ bash test/run_tests.sh --suite 07_analysis --case citation | you changed | run | |---|---| -| `bin/` | `05_helpers` | -| `PoolSeqFlow`, install/uninstall | `02_launcher` | +| `bin/` | `05_helpers` — except the three `check_*.sh`, which are `02_launcher` | +| `PoolSeqFlow`, install/uninstall, the check scripts | `02_launcher` | | `bin/config_migrate.sh`, the templates | `01_migrate` | | step 0, parameter resolution, the change guards | `04_guards` | | wiring, channels, promotion, a step's script | `03_pipeline` | | `dryrun.nf`, `dryrun`/`dryclean` | `06_dryrun` | | version strings, packaging, syntax | `00_static` | -| the shared R library under `analysis/lib/R` | `analysis_rlib` — no JVM, 3 seconds | +| a module library under `modules/lib/` | `analysis_rlib` — no JVM, 3 seconds | | `analysis/lib/nf/`, the frame | the analysis seam you touched: `analysis_frame`, `analysis_plan`, `analysis_verify`, `analysis_design`, `analysis_time`, `analysis_series`, `analysis_modules`, `analysis_results` | -| a module | `--suite `; its cases ship with it under `analysis/modules//test/` | +| a module | `--suite `; its cases travel with it under `modules//test/` | `--fast` runs everything that does not start a JVM; what it skips is `03_pipeline`, `04_guards`, and the pipeline halves of `06_dryrun` and the analysis suites. diff --git a/PoolSeqFlow b/PoolSeqFlow index c98e16c..616d2bb 100755 --- a/PoolSeqFlow +++ b/PoolSeqFlow @@ -5,7 +5,7 @@ # email : ozankiratli@protonmail.com # # Wrapper script for PoolSeqFlow -# Usage: ./PoolSeqFlow {install|init|init_multi|check|run|dryrun|dryclean|migrate_config|clean|reset|analysis |version|cite|list|uninstall|uninstall_all} +# Usage: ./PoolSeqFlow {install|init|init_multi|check |run|dryrun|dryclean|migrate_config|clean|reset|analysis |version|cite|list|uninstall|uninstall_all} # install - creates this release's conda environment and installs the pipeline # init - populates the current directory as a project # init_multi - the same, plus multiRun on @@ -46,7 +46,7 @@ eval "$(conda shell.bash hook)" VERSION="3.0.0" usage() { - echo "Usage: $0 {install|init|init_multi|check|run|dryrun|dryclean|migrate_config|clean|reset|analysis |version|cite|list|uninstall|uninstall_all}" + echo "Usage: $0 {install|init|init_multi|check |run|dryrun|dryclean|migrate_config|clean|reset|analysis |version|cite|list|uninstall|uninstall_all}" exit 1 } @@ -133,18 +133,13 @@ require_project_config() { # else: migrate_config is the fix and must never be refused by this, and clean and dryclean read # no parameter it covers. require_migrated_config() { - grep -qE '^[[:space:]]*storageDir[[:space:]]*=' parameters.config && return 0 + config_is_current parameters.config && return 0 echo "ERROR: parameters.config was written for an older release of PoolSeqFlow." >&2 echo "" >&2 echo "It does not set storageDir, which every config for this release sets." >&2 - # Named to show this is a recognized older file rather than a damaged one. Advisory only: - # the refusal above turns on storageDir alone, and migrate_config reports the full set. - local old found="" - for old in projectDir diploidy rgTagsFile rgTagsPath; do - grep -qE "^[[:space:]]*${old}[[:space:]]*=" parameters.config && found="$found $old" - done + local found; found=$(config_stale_parameters parameters.config) [ -n "$found" ] && { echo "Parameters in it that this release renamed or removed:" >&2 echo " $found" >&2; } @@ -159,7 +154,7 @@ require_migrated_config() { # What an installation consists of. Must match what `git archive` puts in a release tarball. PAYLOAD_ITEMS="PoolSeqFlow poolseqflow.nf dryrun.nf analysis.nf \ -nextflow.config scripts bin lib analysis install parameters.config.template \ +nextflow.config scripts bin lib analysis install citations parameters.config.template \ metadata.csv.template multi-run.csv.example manual README.md CHANGELOG.md LICENSE" # Payload files installed read-only. These are wiring rather than settings: a run reads them, @@ -570,8 +565,12 @@ available_modules() { echo "" printf ' %-16s %-10s %-12s %s\n' NAME VERSION CONTRACT '' local frame_here; frame_here=$(installed_frame_version) - while IFS="$MODULE_INDEX_SEP" read -r name version row_contract row_frame row_env url sha summary; do + while IFS="$MODULE_INDEX_SEP" read -r name row_kind version row_contract row_frame row_env url sha summary; do [ -n "$name" ] || continue + # A library is a module's dependency, installed with whatever declares it and never + # asked for by name. An empty kind is a module, which is what a catalogue written before + # the column existed says. + [ -z "$row_kind" ] || [ "$row_kind" = "module" ] || continue rows=$((rows + 1)) if [ -n "$contract" ] && [ "$row_contract" != "$contract" ]; then state="reads $row_contract - not this release" @@ -600,6 +599,71 @@ available_modules() { # Installs one module into this installation's store. Takes the name, and optionally the exact # version; without one, the newest the catalogue offers for this release's contract. +# One library into the library store, from a catalogue already fetched and format-checked. +# +# Takes the newest version THIS release can run, the same rule install_module applies to a +# module: a library published for a newer frame is skipped rather than installed and then found +# wanting at run time. Prints what it did; returns non-zero and says why on failure, and leaves +# nothing behind when it does. +install_library() { + local lib="$1" index="$2" + local row version frame_req env_req url sha kind + local best_v="" best_url="" best_sha="" + local frame_here; frame_here=$(installed_frame_version) + + while IFS="$MODULE_INDEX_SEP" read -r row_name row_kind row_version row_contract \ + row_frame row_env row_url row_sha row_summary; do + [ "$row_name" = "$lib" ] || continue + [ "${row_kind:-module}" = "library" ] || continue + version_at_most "$row_frame" "$frame_here" || continue + version_at_most "$row_env" "$VERSION" || continue + if [ -z "$best_v" ] || ! version_at_most "$row_version" "$best_v"; then + best_v="$row_version"; best_url="$row_url"; best_sha="$row_sha" + fi + done < <(module_index_rows "$index") + + if [ -z "$best_v" ]; then + echo "ERROR: the catalogue has no library '$lib' this release can run." >&2 + echo " This release is v$VERSION with analysis frame $frame_here." >&2 + return 1 + fi + + local tmp; tmp=$(mktemp -d) + local tarball="$tmp/$lib-$best_v.tar.gz" + echo " library $lib v$best_v" + if ! fetch_url "$best_url" "$tarball"; then + echo "ERROR: could not download the library $lib v$best_v." >&2 + echo " Tried: $best_url" >&2 + rm -rf "$tmp"; return 1 + fi + # Verified before anything is unpacked, exactly as a module's archive is. + if [ -n "$best_sha" ]; then + local got; got=$(file_sha256 "$tarball") + if [ -z "$got" ] || [ "$got" != "$best_sha" ]; then + echo "ERROR: the library $lib v$best_v does not match the catalogue's checksum." >&2 + echo " expected $best_sha" >&2 + echo " got ${got:-no checksum tool on this machine}" >&2 + rm -rf "$tmp"; return 1 + fi + fi + if ! tar -xzf "$tarball" -C "$tmp"; then + echo "ERROR: the library $lib v$best_v could not be unpacked." >&2 + rm -rf "$tmp"; return 1 + fi + if [ ! -f "$tmp/$lib/manifest.json" ]; then + echo "ERROR: the $lib archive has no $lib/manifest.json, so it is not a library." >&2 + rm -rf "$tmp"; return 1 + fi + + mkdir -p "$(library_store)" + mv "$tmp/$lib" "$(library_store)/$lib" + printf 'name\t%s\nversion\t%s\nurl\t%s\nsha256\t%s\ncatalogue\t%s\n' \ + "$lib" "$best_v" "$best_url" "$best_sha" \ + "$(module_index_header "$index" index-version)" > "$(library_store)/$lib/.source" + rm -rf "$tmp" + return 0 +} + install_module() { local name="$1" want="${2:-}" index contract line version row_contract url sha summary local tmp tarball unpacked dest specs spec @@ -636,11 +700,15 @@ install_module() { # Every row for this name whose contract this release speaks, or the exact version asked # for, newest first. + # Fields are in MODULE_INDEX_COLUMNS order: name kind version contract frame environment... + # A row with an empty kind is a module, which is what a catalogue written before the column + # existed yields; a library is never installed by name and is skipped here. candidates=$(module_index_rows "$index" | awk -F"$MODULE_INDEX_SEP" -v n="$name" -v c="$contract" -v w="$want" ' $1 != n { next } - w != "" && $2 != w { next } - c != "" && $3 != c { next } - { print }' | sort -t"$MODULE_INDEX_SEP" -k2,2Vr) + $2 != "" && $2 != "module" { next } + w != "" && $3 != w { next } + c != "" && $4 != c { next } + { print }' | sort -t"$MODULE_INDEX_SEP" -k3,3Vr) if [ -z "$candidates" ]; then echo "ERROR: no module '$name'${want:+ at version $want} is published for this release." >&2 @@ -657,7 +725,7 @@ install_module() { while IFS= read -r row; do [ -n "$row" ] || continue [ -n "$newest" ] || newest="$row" - IFS="$MODULE_INDEX_SEP" read -r _n _v _c row_frame row_env _rest <&2 @@ -682,12 +750,12 @@ ROW exit 1 fi - IFS="$MODULE_INDEX_SEP" read -r name version row_contract row_frame row_env url sha summary <&2 + echo "ERROR: $name v$version needs the library '$lib', which could not be" >&2 + echo " installed. The module was not installed." >&2 + exit 1 + fi + done + fi + mkdir -p "$MODULE_STORE" dest="$MODULE_STORE/$name" mv "$unpacked/$name" "$dest" @@ -808,13 +900,30 @@ uninstall_module() { # A package another installed module also declares stays: the environment is shared, and # what leaves with this module is what nothing left behind asks for. With no analysis # environment there is nothing to take them out of, so nothing does. + # What leaves is what this module declared, less everything still asking for it: the other + # installed modules, AND the release's own baseline. Without the baseline in that subtraction + # a module declaring a package the release also ships - r-ggplot2, r-rcpp - would take it out + # of the shared environment on its way out and break the frame and every module beside it. mine=$(module_packages "$dir/manifest.json" | cut -d= -f1 | sort -u) others=$(store_packages "$MODULE_STORE" "$name" | cut -d= -f1 | sort -u) - going=$(printf '%s\n' "$mine" | grep -vxF "$others" || true) + keep=$(printf '%s\n%s\n' "$others" "$(baseline_packages)" | grep -v '^$' | sort -u) + going=$(printf '%s\n' "$mine" | grep -vxF "$keep" || true) if ! env_exists "$ANALYSIS_ENV"; then going=""; fi echo "" + # Libraries this module brought that nothing else still declares. Same arithmetic as the + # packages below, one level up: what is mine, less what everything remaining asks for. + local mylibs otherlibs libs_going + mylibs=$(module_libraries "$dir/manifest.json" | sort -u) + otherlibs=$(store_libraries "$MODULE_STORE" "$name") + libs_going=$(printf '%s\n' "$mylibs" | grep -v '^$' | grep -vxF "$otherlibs" || true) + echo "This removes:" echo " $dir" + if [ -n "$libs_going" ]; then + echo "" + echo "and these libraries, which no other installed module declares:" + for lib in $libs_going; do echo " $(library_store)/$lib"; done + fi if [ -n "$going" ]; then echo "" echo "and these packages from '$ANALYSIS_ENV', which nothing else installed here asks for:" @@ -849,6 +958,7 @@ uninstall_module() { fi fi rm -rf "$dir" + for lib in $libs_going; do rm -rf "$(library_store)/$lib"; done echo "Removed '$name'. Install it again with the same command that installed it." } @@ -942,28 +1052,70 @@ EOF return 1 } +# Two different questions, and neither answers the other: `install` asks whether the tools a +# release is built to run are present in its own environment, `project` asks whether the files +# and the commands THIS project names are good. A bare `check` is refused rather than assumed, +# because whichever it picked would leave the other unchecked while reporting success. +# +# A function and not a nested `case`, so the arms below stay one level deep: two suite cases +# read the wrapper's own arms to check them against its usage line, and a nested `install)` +# would be read as a top-level subcommand. +run_check() { + case "${1:-}" in + install) + require_install + require_env + bash "$INSTALL/bin/check_install.sh" + ;; + project) + require_install + require_project_config + require_migrated_config + # Last: nextflow config and the version probes both need the environment, and the + # refusals above must be reachable without one. + require_env + bash "$INSTALL/bin/check_project.sh" + ;; + *) + echo "Usage: $(basename "$0") check {install|project}" >&2 + echo "" >&2 + echo " install the tools and helpers this release is built to run," >&2 + echo " and that each comes from this release's own environment" >&2 + echo " project the configuration and the commands this project names," >&2 + echo " read from the directory you are standing in" >&2 + exit 1 + ;; + esac +} + if [ "$#" -lt 1 ]; then usage fi COMMAND=$1 -# `analysis` is the one subcommand that carries words of its own - an analysis command, or the -# module to run. How many follow that first word is each arm's own business, enforced there. -# Every other subcommand takes none. +# Two subcommands carry a word of their own. `analysis` takes an analysis command or the module +# to run, and how many words follow that is each arm's own business, enforced there. `check` +# takes exactly one, saying which check; an absent or unknown one is reported by its own arm, +# which can name the two by name, so anything at all is passed through from here. Every other +# subcommand takes none. ANALYSIS_COMMAND="" ANALYSIS_ARGS=() +CHECK_TARGET="" if [ "$COMMAND" = "analysis" ]; then [ "$#" -ge 2 ] || usage_analysis ANALYSIS_COMMAND=$2 shift 2 ANALYSIS_ARGS=("$@") +elif [ "$COMMAND" = "check" ]; then + [ "$#" -le 2 ] || usage + CHECK_TARGET="${2:-}" elif [ "$#" -ne 1 ]; then usage fi # What the user typed, for the messages that tell them to type it again somewhere else. -INVOCATION="$COMMAND${ANALYSIS_COMMAND:+ $ANALYSIS_COMMAND}" +INVOCATION="$COMMAND${ANALYSIS_COMMAND:+ $ANALYSIS_COMMAND}${CHECK_TARGET:+ $CHECK_TARGET}" # One environment per release, named from VERSION above so a bump carries it. ENV_NAME="PoolSeqFlow-${VERSION}" @@ -1081,7 +1233,7 @@ case $COMMAND in report_other_envs # Against the DEPLOYED copy, which is the one that will run. conda activate "$ENV_NAME" - bash "$(install_prefix)/opt/PoolSeqFlow-${VERSION}/install/check_install.sh" + bash "$(install_prefix)/opt/PoolSeqFlow-${VERSION}/bin/check_install.sh" echo "Installation complete." echo "" echo "The analysis layer was copied with everything else, so 'PoolSeqFlow analysis'" @@ -1106,9 +1258,7 @@ case $COMMAND in ;; check) - require_install - require_env - bash "$INSTALL/install/check_install.sh" + run_check "$CHECK_TARGET" ;; run|resume) # Resuming is what `run` already does: every step skips itself when its outputs are @@ -1425,14 +1575,14 @@ EOF echo "" # Just created, so it is activated without the existence check the other arms make. conda activate "$ANALYSIS_ENV" - bash "$INSTALL/install/check_analysis_install.sh" + bash "$INSTALL/bin/check_analysis_install.sh" ;; check) analysis_takes_no_more require_analysis_install require_analysis_env - bash "$INSTALL/install/check_analysis_install.sh" + bash "$INSTALL/bin/check_analysis_install.sh" ;; complete) diff --git a/README.md b/README.md index 84a5bb5..ef5ac88 100644 --- a/README.md +++ b/README.md @@ -91,7 +91,8 @@ Full walkthrough: [Install](https://ozankiratli.github.io/PoolSeqFlow/getting-st | `./PoolSeqFlow install` | Create the conda environment, install the pipeline, then verify both | | `./PoolSeqFlow init` | Populate the current directory as a project | | `./PoolSeqFlow init_multi` | The same, for a project running several parameter sets over one set of reads | -| `./PoolSeqFlow check` | Verify an existing installation — tools, helpers, config | +| `./PoolSeqFlow check install` | Verify an installation — the tools and helpers it is built to run | +| `./PoolSeqFlow check project` | Verify a project — its configuration, and the commands it names | | `./PoolSeqFlow run` | Start — or resume — the pipeline | | `./PoolSeqFlow dryrun` | Create the directory tree a run would write, empty, before any compute is spent | | `./PoolSeqFlow dryclean` | Remove that preview | diff --git a/analysis/citations.json b/analysis/citations.json index 61b17bb..cb1a062 100644 --- a/analysis/citations.json +++ b/analysis/citations.json @@ -32,7 +32,7 @@ "pages": "315--330", "year": "2018", "doi": "10.1534/genetics.118.300900", - "note": "The effective sample size the analysis layer weights by, n_eff = n*d / (n + d - 1) for a pool of n chromosomes read to depth d. The paper does not write it in that form - it defines D2 as the sum of (d + n - 1)/n, which is the sum of d / n_eff under this form and under no other. Its pools are parameterized by HAPLOID size, which is what leaves n_eff, and everything weighted by it, general over ploidy. It is HERE rather than in a module because analysis/lib/R/n_eff.R is library code and every module that weights anything calls it." + "note": "The effective sample size the analysis layer weights by, n_eff = n*d / (n + d - 1) for a pool of n chromosomes read to depth d. The paper does not write it in that form - it defines D2 as the sum of (d + n - 1)/n, which is the sum of d / n_eff under this form and under no other. Its pools are parameterized by HAPLOID size, which is what leaves n_eff, and everything weighted by it, general over ploidy. It is HERE rather than in a module because it is the n_eff library, installed with whatever module declares it, and every module that weights anything calls it." }, "nei1972": { "name": "Nei's minimum genetic distance", @@ -46,7 +46,7 @@ "pages": "283--292", "year": "1972", "doi": "10.1086/282771", - "note": "The distance the analysis layer places pools by, D_m = (J_X + J_Y)/2 - J_XY, where J is the probability that two chromosomes carry the same allele. It is the MINIMUM distance of this paper and not the standard distance D, which is defined in the same one and is a log of a ratio; the minimum distance is linear in the J terms, so averaging over loci and averaging over sites are the same operation and no ratio-of-averages question arises. What is applied here beyond the paper is the sampling correction: each J is replaced by its unbiased estimator from a sample of n_eff chromosomes, and J_XY takes none because the two pools are sequenced independently. It is HERE rather than in a module because analysis/lib/R/nei_distance.R is library code." + "note": "The distance the analysis layer places pools by, D_m = (J_X + J_Y)/2 - J_XY, where J is the probability that two chromosomes carry the same allele. It is the MINIMUM distance of this paper and not the standard distance D, which is defined in the same one and is a log of a ratio; the minimum distance is linear in the J terms, so averaging over loci and averaging over sites are the same operation and no ratio-of-averages question arises. What is applied here beyond the paper is the sampling correction: each J is replaced by its unbiased estimator from a sample of n_eff chromosomes, and J_XY takes none because the two pools are sequenced independently. It is HERE rather than in a module because it is the nei_distance library, installed with whatever module declares it." }, "ggplot2": { "name": "ggplot2", diff --git a/analysis/frame.version b/analysis/frame.version index e02e6e7..707b3c3 100644 --- a/analysis/frame.version +++ b/analysis/frame.version @@ -24,4 +24,4 @@ # not touch this file. Each module carries its own version in the same form, in its manifest.json. # # Blank lines and lines starting with # are ignored; the first line left is the version. -20260908.003 +20260910.001 diff --git a/analysis/lib/nf/citations.nf b/analysis/lib/nf/citations.nf index ce3c162..45177d0 100644 --- a/analysis/lib/nf/citations.nf +++ b/analysis/lib/nf/citations.nf @@ -3,7 +3,7 @@ // One published analysis carries the result, the script that produced it, the record that // cleared the folder, and this: CITATIONS.md and references.bib for the software behind it. // -// Three sources are merged: PoolSeqFlow and Nextflow from install/citations.json, the analysis +// Three sources are merged: PoolSeqFlow and Nextflow from citations/citations.json, the analysis // layer's own from analysis/citations.json, and the module's, from its own directory. nextflow.enable.dsl=2 @@ -11,7 +11,7 @@ nextflow.enable.dsl=2 include { installDir } from './paths.nf' include { moduleStore } from './modules.nf' -// The entries install/citations.json holds that an ANALYSIS run also invokes. It lists every +// The entries citations/citations.json holds that an ANALYSIS run also invokes. It lists every // tool the pipeline can call, and an analysis calls almost none of them. def sharedCitationKeys() { return ['poolseqflow', 'nextflow'] @@ -34,7 +34,7 @@ def readCitations(String path) { // Everything one invocation of `module` should cite, keyed as write_citations.py expects. def mergedCitations(String module) { def merged = [:] - readCitations("${installDir()}/install/citations.json").each { key, entry -> + readCitations("${installDir()}/citations/citations.json").each { key, entry -> if (sharedCitationKeys().contains(key)) merged[key] = entry } merged.putAll(readCitations("${installDir()}/analysis/citations.json") diff --git a/analysis/lib/nf/modules.nf b/analysis/lib/nf/modules.nf index d9604c2..5e30c8e 100644 --- a/analysis/lib/nf/modules.nf +++ b/analysis/lib/nf/modules.nf @@ -17,6 +17,48 @@ def moduleStore() { return "${installDir()}/analysis/modules".toString() } +// Where installed libraries live, inside the module store under a name no module may take. +// A library is a module's dependency rather than something a project runs, so it is not in the +// roster and `analysis ` never resolves to one. +// A function and not a top-level constant: the strict parser allows only declarations at the +// top level of a script. +def libraryDirName() { + return 'lib' +} + +def libraryStore() { + return "${moduleStore()}/${libraryDirName()}".toString() +} + +// The R files a module's script must source, in the order its libraries are declared. Read from +// the module's own manifest so the list exists once: main.nf repeating it was a second list to +// keep equal, and a disagreement between them was silent. +def moduleLibraryFiles(Object name) { + def manifest = readManifest(file("${moduleStore()}/${name}")) + if (manifest == null) return [] + return manifest.libraries.collect { lib -> + def dir = file("${libraryStore()}/${lib}") + if (!dir.exists()) { + throw new IllegalStateException( + "module '${name}' declares the library '${lib}', which is not installed in\n" + + " ${libraryStore()}\n" + + "Reinstall the module so its libraries come with it.") + } + dir.listFiles().findAll { f -> f.name.endsWith('.R') }.sort { a, b -> a.name <=> b.name } + }.flatten().collect { f -> "${f}".toString() } +} + +// The .cpp a module's libraries offer, resolved the same way as their R. A module that offers a +// compiled path publishes these beside its result whether or not the run used them. +def moduleCompiledFiles(Object name) { + def manifest = readManifest(file("${moduleStore()}/${name}")) + if (manifest == null) return [] + return manifest.libraries.collect { lib -> + file("${libraryStore()}/${lib}").listFiles() + .findAll { f -> f.name.endsWith('.cpp') }.sort { a, b -> a.name <=> b.name } + }.flatten().collect { f -> "${f}".toString() } +} + // The published-table contract a module declares it speaks. Bumped when a column's name or // meaning changes. def contractVersion() { @@ -33,6 +75,7 @@ def builtinModules() { contract: contractVersion(), license : 'Apache-2.0', needs : [], + libraries: [], gates : [], outputs : [], packages: [], @@ -140,6 +183,7 @@ def readManifest(Object dir) { frame : "${parsed.frame}".toString(), environment: "${parsed.environment}".toString(), needs : parsed.needs ?: [], + libraries : (parsed.libraries ?: []).collect { lib -> "${lib}".toString() }, gates : parsed.gates ?: [], outputs : parsed.outputs ?: [], packages : (parsed.packages ?: []).collect { spec -> "${spec}".toString() }, @@ -178,7 +222,7 @@ def moduleRoster() { def roster = builtinModules() def store = file(moduleStore()) if (store.exists()) { - store.listFiles().findAll { entry -> entry.isDirectory() } + store.listFiles().findAll { entry -> entry.isDirectory() && entry.name != libraryDirName() } .sort { a, b -> "${a}" <=> "${b}" } .each { dir -> def found = readManifest(dir) diff --git a/analysis/references.bib b/analysis/references.bib index 40c3aeb..7259046 100644 --- a/analysis/references.bib +++ b/analysis/references.bib @@ -28,7 +28,7 @@ @article{hivert2018poolseq pages = {315--330}, year = {2018}, doi = {10.1534/genetics.118.300900}, - note = {The effective sample size the analysis layer weights by, n_eff = n*d / (n + d - 1) for a pool of n chromosomes read to depth d. The paper does not write it in that form - it defines D2 as the sum of (d + n - 1)/n, which is the sum of d / n_eff under this form and under no other. Its pools are parameterized by HAPLOID size, which is what leaves n_eff, and everything weighted by it, general over ploidy. It is HERE rather than in a module because analysis/lib/R/n_eff.R is library code and every module that weights anything calls it.}, + note = {The effective sample size the analysis layer weights by, n_eff = n*d / (n + d - 1) for a pool of n chromosomes read to depth d. The paper does not write it in that form - it defines D2 as the sum of (d + n - 1)/n, which is the sum of d / n_eff under this form and under no other. Its pools are parameterized by HAPLOID size, which is what leaves n_eff, and everything weighted by it, general over ploidy. It is HERE rather than in a module because it is the n_eff library, installed with whatever module declares it, and every module that weights anything calls it.}, } @article{nei1972distance, @@ -43,7 +43,7 @@ @article{nei1972distance pages = {283--292}, year = {1972}, doi = {10.1086/282771}, - note = {The distance the analysis layer places pools by, D_m = (J_X + J_Y)/2 - J_XY, where J is the probability that two chromosomes carry the same allele. It is the MINIMUM distance of this paper and not the standard distance D, which is defined in the same one and is a log of a ratio; the minimum distance is linear in the J terms, so averaging over loci and averaging over sites are the same operation and no ratio-of-averages question arises. What is applied here beyond the paper is the sampling correction: each J is replaced by its unbiased estimator from a sample of n_eff chromosomes, and J_XY takes none because the two pools are sequenced independently. It is HERE rather than in a module because analysis/lib/R/nei_distance.R is library code.}, + note = {The distance the analysis layer places pools by, D_m = (J_X + J_Y)/2 - J_XY, where J is the probability that two chromosomes carry the same allele. It is the MINIMUM distance of this paper and not the standard distance D, which is defined in the same one and is a log of a ratio; the minimum distance is linear in the J terms, so averaging over loci and averaging over sites are the same operation and no ratio-of-averages question arises. What is applied here beyond the paper is the sampling correction: each J is replaced by its unbiased estimator from a sample of n_eff chromosomes, and J_XY takes none because the two pools are sequenced independently. It is HERE rather than in a module because it is the nei_distance library, installed with whatever module declares it.}, } @misc{wickham2016ggplot2, diff --git a/install/check_analysis_install.sh b/bin/check_analysis_install.sh similarity index 100% rename from install/check_analysis_install.sh rename to bin/check_analysis_install.sh diff --git a/install/check_install.sh b/bin/check_install.sh similarity index 59% rename from install/check_install.sh rename to bin/check_install.sh index d3ee52c..2df300f 100755 --- a/install/check_install.sh +++ b/bin/check_install.sh @@ -2,22 +2,19 @@ # # Verify a PoolSeqFlow installation before a run depends on it. # -# Usage: ./PoolSeqFlow check (the wrapper activates the environment first) +# Usage: ./PoolSeqFlow check install (the wrapper activates the environment first) # -# Checks three things: +# Checks two things: # 1. Every command the pipeline invokes resolves and runs, with its version. # 2. Every helper in bin/ is present and executable. -# 3. If parameters.config exists, that Nextflow can parse it. # -# With a parameters.config present the commands come from `params.software` through -# `nextflow config`, so a command repointed at a system binary is checked as configured. -# Without one, the canonical list below is used. +# IT KNOWS NOTHING ABOUT parameters.config. That file belongs to a project and this verifies an +# INSTALLATION, which a project need not exist for; the tool list is the canonical one below. +# A project that repoints a tool at a system binary is a project's business, and `run` is where +# that resolves. set -uo pipefail -# Two directories: the installation holds the helpers and nextflow.config, the directory this -# was invoked from is the project and holds parameters.config. Captured before the cd. -PROJECT_DIR="$PWD" cd "$(dirname "$0")/.." || exit 1 INSTALL_DIR="$PWD" @@ -51,6 +48,17 @@ CANONICAL="java cutadapt fastqc trim_galore samtools bamtools bwa bcftools vcfto } . "$INSTALL_DIR/lib/tool_version.sh" +# THE ENVIRONMENT'S OWN COPY, NOT WHATEVER PATH FINDS. Every tool in CANONICAL is pinned in +# install/environment.yml, so one resolving from outside means the environment is missing a +# package and the system's copy is standing in - at some other version, and only on this +# machine. Reporting that as OK is the failure this check exists to prevent, and it is silent: +# the run works here and does not reproduce anywhere else. +# +# Empty when the script is run without the environment active. Then there is nothing to compare +# against and the section below says so rather than checking PATH and calling it an answer. +ENV_PREFIX="${CONDA_PREFIX:-}" +[ "$(basename "${ENV_PREFIX:-.}")" = "$ENV_NAME" ] || ENV_PREFIX="" + check_tool() { local name="$1" cmd="$2" resolved version checked=$((checked + 1)) @@ -61,6 +69,18 @@ check_tool() { return fi + if [ -n "$ENV_PREFIX" ]; then + case $resolved in + "$ENV_PREFIX"/*) ;; + *) + printf ' %-14s %-12s %sOUTSIDE THE ENVIRONMENT%s %s\n' \ + "$name" "$cmd" "$RED" "$RESET" "$resolved" + missing=$((missing + 1)) + return + ;; + esac + fi + version=$(tool_version "$name" "$cmd") if [ -z "$version" ]; then # Resolved but reported no version. Not fatal; some tools have no version flag. @@ -78,34 +98,16 @@ echo # ----------------------------------------------------------------- 1. tools -- echo "Tools" -echo - -declare -a NAMES=() CMDS=() -source_note="" - -# Which list is in use, with the reason when it is the fallback. -if [ ! -f "$PROJECT_DIR/parameters.config" ]; then - source_note="canonical list - no parameters.config in $PROJECT_DIR" -elif ! command -v nextflow >/dev/null 2>&1; then - source_note="canonical list - nextflow not available to read parameters.config" +if [ -n "$ENV_PREFIX" ]; then + printf ' %sfrom %s%s\n' "$DIM" "$ENV_PREFIX" "$RESET" else - # Interpolated by Nextflow, so this reads what the pipeline will actually invoke. From the - # project directory against the installation, exactly as a run would. - while read -r n c; do - [ -n "$n" ] || continue - NAMES+=("$n"); CMDS+=("$c") - done < <(cd "$PROJECT_DIR" && nextflow config -flat "$INSTALL_DIR" 2>/dev/null | - sed -n "s|^params\.software\.\([A-Za-z_][A-Za-z0-9_]*\) = '\(.*\)'$|\1 \2|p") - if [ ${#NAMES[@]} -gt 0 ]; then - source_note="params.software in parameters.config" - else - source_note="canonical list - could not read params.software from parameters.config" - fi + printf ' %sfrom PATH: %s is not active, so nothing here says WHERE a tool came from%s\n' \ + "$YELLOW" "$ENV_NAME" "$RESET" fi +echo -if [ ${#NAMES[@]} -eq 0 ]; then - for n in $CANONICAL; do NAMES+=("$n"); CMDS+=("$n"); done -fi +declare -a NAMES=() CMDS=() +for n in $CANONICAL; do NAMES+=("$n"); CMDS+=("$n"); done check_tool nextflow nextflow for i in "${!NAMES[@]}"; do @@ -114,8 +116,6 @@ done check_tool python3 python3 check_tool awk awk -echo -echo " ${DIM}tool list from: ${source_note}${RESET}" echo # --------------------------------------------------------------- 2. helpers -- @@ -125,9 +125,14 @@ echo # Enumerated, not hand-listed. Everything in bin/ is run and needs its executable bit; # anything sourced lives in lib/ instead. +# +# The check scripts are in bin/ and are skipped here: they are run by the wrapper, not by a +# process script, so they are not on the list of helpers a run depends on - and this one +# reporting on itself says nothing, since it is already running. for path in bin/*; do f=$(basename "$path") [ -d "$path" ] && continue + case $f in check_install.sh|check_project.sh|check_analysis_install.sh) continue ;; esac checked=$((checked + 1)) if [ ! -f "bin/$f" ]; then @@ -144,28 +149,6 @@ for path in bin/*; do done echo -# ---------------------------------------------------------------- 3. config -- - -echo "Configuration" -echo - -if [ ! -f "$PROJECT_DIR/parameters.config" ]; then - printf ' %-28s %sNOT YET CREATED%s in %s\n' "parameters.config" "$YELLOW" "$RESET" "$PROJECT_DIR" - echo " cp $INSTALL_DIR/parameters.config.template $PROJECT_DIR/parameters.config" -elif ! command -v nextflow >/dev/null 2>&1; then - printf ' %-28s %sSKIPPED%s nextflow not available\n' "parameters.config" "$YELLOW" "$RESET" -else - checked=$((checked + 1)) - if err=$(cd "$PROJECT_DIR" && nextflow config "$INSTALL_DIR" 2>&1 >/dev/null); then - printf ' %-28s %sPARSES%s\n' "parameters.config" "$GREEN" "$RESET" - else - printf ' %-28s %sFAILED TO PARSE%s\n' "parameters.config" "$RED" "$RESET" - printf '%s\n' "$err" | sed 's/^/ /' - missing=$((missing + 1)) - fi -fi -echo - # ---------------------------------------------------------------- summary ---- if [ "$missing" -eq 0 ]; then diff --git a/bin/check_project.sh b/bin/check_project.sh new file mode 100755 index 0000000..cee7534 --- /dev/null +++ b/bin/check_project.sh @@ -0,0 +1,236 @@ +#!/bin/bash +# +# Verify a PoolSeqFlow PROJECT before a run depends on it. +# +# Usage: ./PoolSeqFlow check project (the wrapper activates the environment first) +# run from the project directory, which is where parameters.config lives +# +# Checks two things: +# 1. The project's files parse: parameters.config, metadata.csv, and the run table when +# multiRun is on. +# 2. Every command the pipeline will invoke, AS THIS PROJECT CONFIGURES IT. The list comes +# from params.software through `nextflow config`, so a command repointed at a system +# binary is checked the way the run will call it. +# +# THE INSTALLATION IS bin/check_install.sh's BUSINESS. That one asks whether the tools a +# release is built to run are present at all; this one asks whether the tools THIS PROJECT +# names resolve. A project that repoints nothing gets the same answer twice, which is the +# point: the difference between them is exactly the project's own configuration. + +set -uo pipefail + +# Two directories: the project is where this was invoked from and holds parameters.config, the +# installation holds nextflow.config and the helpers. Captured before the cd. +PROJECT_DIR="$PWD" +cd "$(dirname "$0")/.." || exit 1 +INSTALL_DIR="$PWD" + +INSTALL="$INSTALL_DIR" +# shellcheck source=../lib/wrapper_lib.sh +. "$INSTALL_DIR/lib/wrapper_lib.sh" || { + echo "ERROR: $INSTALL_DIR/lib/wrapper_lib.sh is missing." >&2 + echo " This installation is incomplete; reinstall it." >&2 + exit 1 +} +# Checked before use: this runs without `set -e`, so a missing library would leave tool_version +# undefined and every tool would report as present with no version. +[ -f "$INSTALL_DIR/lib/tool_version.sh" ] || { + echo "ERROR: $INSTALL_DIR/lib/tool_version.sh is missing." >&2 + echo " This installation is incomplete; reinstall it." >&2 + exit 1 +} +. "$INSTALL_DIR/lib/tool_version.sh" + +RED=''; GREEN=''; YELLOW=''; DIM=''; RESET='' +if [ -t 1 ]; then + RED=$'\033[31m'; GREEN=$'\033[32m'; YELLOW=$'\033[33m' + DIM=$'\033[2m'; RESET=$'\033[0m' +fi + +missing=0 +checked=0 + +CONFIG="$PROJECT_DIR/parameters.config" + +verdict() { printf ' %-28s %s%s%s%s\n' "$2" "$1" "$3" "$RESET" "${4:+ $4}"; } +pass() { verdict "$GREEN" "$1" "$2" "${3:-}"; } +warn() { verdict "$YELLOW" "$1" "$2" "${3:-}"; } +fail() { verdict "$RED" "$1" "$2" "${3:-}"; missing=$((missing + 1)); } +note() { printf ' %-28s %s%s%s\n' "$1" "$DIM" "$2" "$RESET"; } + +check_tool() { + local name="$1" cmd="$2" version + checked=$((checked + 1)) + + if ! command -v "$cmd" >/dev/null 2>&1; then + printf ' %-14s %-22s %sMISSING%s %s\n' "$name" "$cmd" "$RED" "$RESET" "not on PATH" + missing=$((missing + 1)) + return + fi + + version=$(tool_version "$name" "$cmd") + if [ -z "$version" ]; then + printf ' %-14s %-22s %sFOUND%s %s(version not reported)%s\n' \ + "$name" "$cmd" "$YELLOW" "$RESET" "$DIM" "$RESET" + else + printf ' %-14s %-22s %sOK%s %s\n' "$name" "$cmd" "$GREEN" "$RESET" "$version" + fi +} + +echo "PoolSeqFlow project check" +echo "=========================" +echo " $PROJECT_DIR" +echo + +# --------------------------------------------------------- 1. configuration -- + +echo "Configuration" +echo + +if [ ! -f "$CONFIG" ]; then + echo "${RED}No parameters.config in $PROJECT_DIR.${RESET}" >&2 + echo "" >&2 + echo "A project check needs a project. Make one and populate it:" >&2 + echo " cd $PROJECT_DIR" >&2 + echo " PoolSeqFlow init" >&2 + exit 1 +fi + +checked=$((checked + 1)) +if config_is_current "$CONFIG"; then + pass "parameters.config" "WRITTEN FOR THIS RELEASE" +else + stale=$(config_stale_parameters "$CONFIG") + fail "parameters.config" "WRITTEN FOR AN OLDER RELEASE" \ + "run: PoolSeqFlow migrate_config" + [ -n "$stale" ] && printf ' %sparameters it renamed or removed:%s%s\n' \ + "$DIM" "$RESET" "$stale" +fi + +# Nextflow's own parse, from the project against the installation, exactly as a run would. +# Everything below reads settings out of this, so a failure here is reported and the rest is +# skipped by name rather than passing over an empty answer. +PARSED=0 +checked=$((checked + 1)) +if ! command -v nextflow >/dev/null 2>&1; then + warn "parameters.config" "NOT PARSED" "nextflow is not on PATH" +else + # Both streams: Nextflow reports a config error on stdout, so capturing stderr alone leaves + # a failure with nothing to print. The whole capture is discarded when it succeeds. + if err=$(cd "$PROJECT_DIR" && nextflow config "$INSTALL_DIR" 2>&1); then + pass "parameters.config" "PARSES" + PARSED=1 + else + fail "parameters.config" "FAILED TO PARSE" + printf '%s\n' "$err" | sed 's/^/ /' + # The installation's nextflow.config interpolates the project's settings, so a missing + # or malformed one surfaces as a failure to parse THAT file. Reported as it comes and + # then explained: without this the message reads as a broken installation, which is + # the one thing it is not - a run fails here in exactly the same way. + case $err in + *"$INSTALL_DIR/nextflow.config"*) + printf ' %sThat is the installation'"'"'s own config, and it is not damaged: it\n' \ + "$DIM" + printf ' interpolates your settings, so a parameter missing from\n' + printf ' parameters.config fails while it is being read.%s\n' "$RESET" + ;; + esac + fi +fi + +# The two tables, each by the parser the pipeline itself uses, so a project is told here what +# step 0 would tell it. Their JSON goes nowhere; the exit status is the answer. +# +# WHICH FILES THEY ARE COMES OUT OF THE CONFIG, so a config that did not parse means they are +# not known - not that they are the defaults. Guessing a name here would report a file the +# project does not use, and `not in use` would be a claim about multiRun nobody read. +metadata_file="" +multirun_file="" +multirun_on=0 +if [ "$PARSED" -eq 1 ]; then + metadata_file=$(cd "$PROJECT_DIR" && nf_config_value "params.metadataFile") + multirun_file=$(cd "$PROJECT_DIR" && nf_config_value "params.multiRunFile") + value=$(cd "$PROJECT_DIR" && nf_config_value "params.multiRun") + [ "$value" = "true" ] && multirun_on=1 +fi + +check_table() { + local label="$1" path="$2" parser="$3" out + checked=$((checked + 1)) + if [ ! -f "$path" ]; then + fail "$label" "MISSING" "expected at $path" + return + fi + if out=$(python3 "$INSTALL_DIR/bin/$parser" "$path" 2>&1 >/dev/null); then + pass "$label" "PARSES" + else + fail "$label" "FAILED TO PARSE" + printf '%s\n' "$out" | sed 's/^/ /' + fi +} + +if [ "$PARSED" -eq 0 ]; then + note "the sample metadata" "not checked - parameters.config did not parse" + note "the run table" "not checked - parameters.config did not parse" +elif [ -z "$metadata_file" ]; then + fail "metadataFile" "NOT SET" "parameters.config names no sample metadata file" +else + check_table "$metadata_file" "$PROJECT_DIR/$metadata_file" parse_metadata.py + if [ "$multirun_on" -eq 0 ]; then + note "${multirun_file:-the run table}" "not in use - multiRun is false" + elif [ -z "$multirun_file" ]; then + fail "multiRunFile" "NOT SET" "multiRun is on and no run table is named" + else + check_table "$multirun_file" "$PROJECT_DIR/$multirun_file" parse_multirun.py + fi +fi + +echo + +# ----------------------------------------------------------------- 2. tools -- + +echo "Tools, as this project configures them" +echo + +declare -a NAMES=() CMDS=() +if [ "$PARSED" -eq 1 ]; then + # Interpolated by Nextflow, so this reads what the pipeline will actually invoke. + while read -r n c; do + [ -n "$n" ] || continue + NAMES+=("$n"); CMDS+=("$c") + done < <(cd "$PROJECT_DIR" && nextflow config -flat "$INSTALL_DIR" 2>/dev/null | + sed -n "s|^params\.software\.\([A-Za-z_][A-Za-z0-9_]*\) = '\(.*\)'$|\1 \2|p") +fi + +if [ ${#NAMES[@]} -eq 0 ]; then + # Never silently substituted with the canonical list. The whole reason this section exists + # is that it reads the PROJECT'S list, and one that could not be read is a finding. + checked=$((checked + 1)) + fail "params.software" "NOT READ" "so no tool was checked as this project configures it" + [ "$PARSED" -eq 1 ] && printf ' %sparameters.config parses but declares no software block%s\n' \ + "$DIM" "$RESET" +else + check_tool nextflow nextflow + for i in "${!NAMES[@]}"; do + check_tool "${NAMES[$i]}" "${CMDS[$i]}" + done + check_tool python3 python3 + check_tool awk awk + echo + printf ' %stool list from: params.software in parameters.config%s\n' "$DIM" "$RESET" +fi + +echo + +# ---------------------------------------------------------------- summary ---- + +if [ "$missing" -eq 0 ]; then + echo "${GREEN}All $checked checks passed.${RESET}" + exit 0 +fi + +echo "${RED}$missing of $checked checks failed.${RESET}" +echo +echo "This checks a project. To check the installation itself:" +echo " PoolSeqFlow check install" +exit 1 diff --git a/install/citations.json b/citations/citations.json similarity index 100% rename from install/citations.json rename to citations/citations.json diff --git a/install/references.bib b/citations/references.bib similarity index 100% rename from install/references.bib rename to citations/references.bib diff --git a/dev/RELEASING.md b/dev/RELEASING.md index ecd435d..dbae8a9 100644 --- a/dev/RELEASING.md +++ b/dev/RELEASING.md @@ -4,7 +4,7 @@ The order to do a release in, what each step must show, and what bites. **This is a living procedure, not a record.** Correct it when a release teaches you something; it is not dated and it does not describe a particular version. `.claude/development-notes/` is where the dated records go. -Everything happens on `dev` until step 4. Steps 4 to 9 happen on `main`. Step 10 brings `main` back. +Everything happens on `dev` until step 4. Steps 4 to 10 happen on `main`. Step 11 brings `main` back. **Two things are not written down here and are checked by the suite instead**: the release archive (`verify-archive.sh`), the docs and citation gates (`build_docs.py --check`, `bib2citations.py --check`), the analysis versions (`check-analysis-versions.sh`) and the version-consistency case all run inside `00_static`. If the suite is green they passed. What follows is only the work the suite cannot do for you. @@ -55,7 +55,7 @@ dev/scripts/check-module-packages.sh **Real conda, real network, minutes.** It is deliberately not in the suite, which is exactly why it gets skipped — it is on this checklist or it is lost. -It builds the baseline, installs what the shipped modules declare, and checks a module cannot move a version another module or the release itself is running on. It found on its first run that `--freeze-installed` does **less** than its name suggests: it refuses to change a package the solve reaches on its own, but a package named on the command line it installs at the version asked for, downgrading what is there. +It builds the baseline, installs what the modules published from this repository declare, and checks a module cannot move a version another module or the release itself is running on. It found on its first run that `--freeze-installed` does **less** than its name suggests: it refuses to change a package the solve reaches on its own, but a package named on the command line it installs at the version asked for, downgrading what is there. Every check must say `ok`. A failure here is a compatibility problem between this release and a module, and it is settled by publishing, not on a user's machine. @@ -103,7 +103,7 @@ dev/scripts/americanize.py # report; --fix rewrites the safe ones Open the merge request, review the diff as a whole, merge. -The release commits — the version bump and the CHANGELOG — are made on `main` after this, so `main` briefly holds the merge at the old version. That state is never tagged, so it costs nothing, and it keeps release-only commits off `dev` until step 10. +The release commits — the version bump and the CHANGELOG — are made on `main` after this, so `main` briefly holds the merge at the old version. That state is never tagged, so it costs nothing, and it keeps release-only commits off `dev` until step 11. ## 5. Run the analysis version gate @@ -125,11 +125,9 @@ Mid-development a version is legitimately behind, which is why the plain run onl dev/scripts/bump-version.sh ``` -It rewrites the version in the wrapper (header comment and `VERSION=`) and in `nextflow.config`'s manifest, sets `"environment"` in every shipped module's manifest and moves that module's own version with it, and prepends a CHANGELOG section listing every commit since the last release tag. It does not commit, tag or push — it prints those commands. +It rewrites the version in the wrapper (header comment and `VERSION=`) and in `nextflow.config`'s manifest, and prepends a CHANGELOG section listing every commit since the last release tag. It does not commit, tag or push — it prints those commands. -**The shipped manifests are part of the bump and no longer a separate step.** A module shipped inside a release travels in the same tarball as the analysis environment it names, so at a release the two agree by construction and there is nothing to decide. `00_static` still asserts `"environment"` equals `nextflow.config`'s version exactly, so a manifest that somehow disagrees still fails the next step — the check is unchanged, only the typing is gone. - -A module published **outside** a release is different and is still yours: its `"environment"` is a claim about which release it was built against, and `dev/scripts/bump-analysis-version.sh module ` is what moves its version when you change it. +**It does not touch a module or library manifest, and must not.** No module ships inside a release, so a module's `"environment"` is the oldest release its author says it needs — moved when its needs move, by whoever maintains it, not by a release bump acting on its behalf. `dev/scripts/bump-analysis-version.sh module ` is what moves a module's own version when you change it. ## 7. Run the full suite @@ -139,7 +137,17 @@ bash test/run_tests.sh On `main`, at the new version, with the frozen environments. This is the run that matters — everything before it tested a version string that is no longer the one shipping. -**Check the counts against the previous run, not only the exit status.** A filter that matches nothing also reports success. `nextflow lint .` must be at zero errors *and* zero warnings. +**Run it with conda on `PATH`.** The suite finds the analysis environment through `conda info --base`, and a shell without a working `conda` finds nothing: three cases then skip — the PDF report, the compiled hot path, and the compiled-and-parallel agreement — and the run still reports success. Set `TEST_ANALYSIS_ENV=/envs/PoolSeqFlow--analysis` if discovery cannot find it. + +**Check the counts against the previous run, not only the exit status** — cases passed *and* cases skipped. A filter that matches nothing also reports success, and a skip is how a case that should have run says so quietly. + +**Lint without `modules/`**, at zero errors and zero warnings: + +``` +nextflow lint analysis analysis.nf dryrun.nf poolseqflow.nf scripts +``` + +`nextflow lint .` cannot pass: a module's `main.nf` imports the frame as `'../../lib/nf/plan.nf'`, which resolves from the store it is installed into and not from `modules//`. `00_static` lints the modules, in an assembled store layout. ## 8. Write the CHANGELOG @@ -154,7 +162,23 @@ What the notes owe a reader, beyond the commits: anything a user has to *do*, an - **Zenodo mints a DOI for the version.** The citation machinery points at the all-versions DOI and tells a user to pick their version from it, so the version record has to exist for the citation the release prints to be answerable. - Verify the published archive installs from scratch on a machine that has never had it. -## 10. Return to `dev` +## 10. Publish the modules and libraries this release runs + +No module ships inside a release, so a release on its own leaves users with an empty store. Publishing is what makes the modules installable, and it is separate on purpose: a module moves on its own timetable, and one published tomorrow is installable into this release without re-releasing anything. + +``` +dev/scripts/publish-module.sh [ref] +``` + +It builds the tarball into `modules/repo/`, reads `kind`, `contract`, `frame`, `environment` and `summary` out of the thing's own manifest, appends the catalogue row and bumps `#!index-version`. It takes a module or a library by name and finds it in `modules/` or `modules/lib/`. + +**Publish from a commit.** The script refuses a source with no commit timestamp, because the tarball's reproducibility depends on it — archiving a tree stamps *now* and two builds of one ref stop matching. So this comes after the release is committed, not before. + +**The tarball and the row it advertises go out in one commit.** The site deploys `modules/repo/` wholesale at the published address `/PoolSeqFlow/modules-repo/`, so a row committed without its file advertises a download that 404s until the next deploy. + +**A published version is never rewritten.** Somebody may have installed it and its checksum is in the catalogue. Change means bumping the version and publishing that; `00_static` checks every row's file exists and its checksum matches. + +## 11. Return to `dev` Sync `dev` with `main` so the version bump and the CHANGELOG come back, then carry on. The first commits after a release are usually the things this protocol found and deferred. @@ -174,3 +198,27 @@ They look alike, and each one is cheap to spot once you know the shape: - **A literal that happens to be right still says PASS.** A case asserted `${EXPECTED_VERSION:-2.2.0}` against a variable set nowhere. Correct until the bump, then a failure that says nothing about what it tests. **A version bump is when this class surfaces**, because everything before it ran against a tree carrying the old version. That is why step 7 is after step 6 and not before it. + +--- + +## Post-release triage + +Things this protocol worked around rather than fixed. Each has a note saying what the workaround is, so a release is never blocked on one — and each is a gate that is weaker than it reads, so none of them should sit here long. + +**`run_tests.sh` reports success over three cases it never ran.** The analysis environment is discovered through `conda info --base`, so a run in a shell with no working `conda` finds none and silently skips the PDF report, the compiled hot path, and the compiled-and-parallel agreement. Measured on 2026-09-10: a full run said `549 passed, 3 skipped` and exit 0, and the three that skipped are among the least trivial in the suite — F1's Rcpp worker bug was caught by the combination of compiled *and* parallel and by nothing else. + +The workaround is step 7's, and it is a person remembering: run with conda on `PATH`, and read the skip count. What it should do instead is **refuse**, the way `check-analysis-versions.sh --release` refuses rather than answering — a suite that cannot find the environment for cases that need it should say so and exit non-zero when it was asked for a full run. `--fast` and `--cost static` legitimately skip those, so the refusal belongs to the unfiltered run alone. + +**A release step that shells out to `conda` cannot assume `conda` works.** On a machine where the shell function is set up for an interactive shell of a different family, a non-interactive `bash -c 'conda env list'` fails with `__conda_exe: permission denied` — and `env_exists()` in `prep-version.sh` is `conda env list | grep -qxF`, so a broken function reads as *the environment is not there* and the script refuses a release that had nothing wrong with it. A misconfigured plugin is the milder version of the same thing: `anaconda-anon-usage` prints an error line on every invocation while conda still works, which is noise in a log that a person is being asked to read carefully. + +The workaround is to `source /etc/profile.d/conda.sh` before running anything that needs conda. What the scripts should do instead is source it themselves, or resolve the real binary and stop depending on the shell at all — and `env_exists()` in particular should tell *conda said no* apart from *conda did not run*, because those are opposite problems wearing the same message. + +**Step 2 is skippable when nothing has drifted, and the protocol should say how to know.** Its expensive half updates and re-freezes both environments; when the last freeze already describes them, that work produces an identical file and an hour of nothing. The check is two exports to a scratch path and a diff: + +``` +dev/scripts/export-environment.sh PoolSeqFlow- /tmp/a.yml +dev/scripts/export-environment.sh PoolSeqFlow--analysis /tmp/b.yml +diff /tmp/a.yml install/environment.yml && diff /tmp/b.yml install/environment-analysis.yml +``` + +Identical both ways means the freeze still holds and the update-and-export cycle is redundant. **`check-module-packages.sh` is not part of that skip** — it asks a different question, whether the modules' own pins still solve against the frozen baseline, and a module's manifest can change on a day the environments do not. diff --git a/dev/release-notes-tail.md b/dev/release-notes-tail.md new file mode 100644 index 0000000..5b3eda4 --- /dev/null +++ b/dev/release-notes-tail.md @@ -0,0 +1,36 @@ + + +## Download and install + +```bash +curl -LO https://github.com/ozankiratli/PoolSeqFlow/releases/download/v@VERSION@/@NAME@.tar.gz +tar -xzf @NAME@.tar.gz +cd @NAME@ + +cp parameters.config.template parameters.config +./PoolSeqFlow install +``` + +Then edit `parameters.config` and `metadata.csv` for your data and run +`./PoolSeqFlow run`. + +Verify the download with `sha256sum -c SHA256SUMS`. + +`PoolSeqFlow.tar.gz` is the same archive under a stable name, for +scripted installs: +`https://github.com/ozankiratli/PoolSeqFlow/releases/latest/download/PoolSeqFlow.tar.gz` + +**Upgrading an existing project?** Your `parameters.config` is not +touched by a new version and can be missing parameters this release +expects. Run `./PoolSeqFlow migrate_config` and read what it reports — +see [Upgrading](https://ozankiratli.github.io/PoolSeqFlow/getting-started/upgrading/). + +Full documentation: +The full changelog, including every commit, is in `CHANGELOG.md` in the download and in the repository. diff --git a/dev/scripts/bench-compiled-paths.R b/dev/scripts/bench-compiled-paths.R index 09f81c4..c3c9485 100755 --- a/dev/scripts/bench-compiled-paths.R +++ b/dev/scripts/bench-compiled-paths.R @@ -24,12 +24,16 @@ sizes <- if (length(args) > 0) as.numeric(args) else 3.2e6 REPS <- 5 POOLS <- 6 -for (f in list.files(file.path(repo, "analysis/lib/R"), pattern = "[.]R$", full.names = TRUE)) { +# Every library, from the sources rather than from an installation. A library is a directory +# under modules/lib/ holding its .R and, where it has one, the .cpp beside it - so `recursive` +# is what reaches into them, and a library added later is picked up with no edit here. +libs <- file.path(repo, "modules/lib") +for (f in list.files(libs, pattern = "[.]R$", full.names = TRUE, recursive = TRUE)) { source(f) } -Rcpp::sourceCpp(file.path(repo, "analysis/lib/cpp/allele_frequencies.cpp")) -Rcpp::sourceCpp(file.path(repo, "analysis/lib/cpp/site_diversity.cpp")) -Rcpp::sourceCpp(file.path(repo, "analysis/lib/cpp/nei_distance.cpp")) +for (f in list.files(libs, pattern = "[.]cpp$", full.names = TRUE, recursive = TRUE)) { + Rcpp::sourceCpp(f) +} # A published depth table is overwhelmingly biallelic. A uniform draw over arities would give # the compiled path more digits to parse per site than a real cohort does. diff --git a/dev/scripts/bib2citations.py b/dev/scripts/bib2citations.py index e4396e0..9c262ad 100644 --- a/dev/scripts/bib2citations.py +++ b/dev/scripts/bib2citations.py @@ -12,9 +12,9 @@ Each `references.bib` compiles to the `citations.json` in the same directory: - install/references.bib -> install/citations.json + citations/references.bib -> citations/citations.json analysis/references.bib -> analysis/citations.json - analysis/modules//references.bib -> analysis/modules//citations.json + modules//references.bib -> modules//citations.json ## The format @@ -173,8 +173,11 @@ def compile_bib(path: Path) -> str: def sources() -> list[Path]: - found = [REPO / "install" / "references.bib", REPO / "analysis" / "references.bib"] - found += sorted((REPO / "analysis" / "modules").glob("*/references.bib")) + # `modules/` and not `analysis/modules/`: the latter is the install store, gitignored and + # empty in a checkout, so globbing it returns nothing and --check reports every module's + # citations fine without having read one. A library cites nothing and has no bib. + found = [REPO / "citations" / "references.bib", REPO / "analysis" / "references.bib"] + found += sorted((REPO / "modules").glob("*/references.bib")) return [path for path in found if path.exists()] diff --git a/dev/scripts/build_docs.py b/dev/scripts/build_docs.py index 2c65942..96edf41 100644 --- a/dev/scripts/build_docs.py +++ b/dev/scripts/build_docs.py @@ -57,7 +57,11 @@ # The module repository, copied into the site verbatim: the catalogue the wrapper fetches and # the tarballs its rows point at. Published in the same deploy as the page below, so a row can # never advertise a download that is not there yet. -REPO_DIR = REPO / "modules-repo" +# The source directory and the PUBLISHED path are deliberately different. The source moved under +# modules/ to keep everything module-related together; the published path may never move, because +# MODULE_INDEX_URL in lib/wrapper_lib.sh compiles into every release and asks for this address +# for as long as that release exists. +REPO_DIR = REPO / "modules" / "repo" REPO_PATH = "modules-repo" @@ -93,7 +97,7 @@ def module_repo_page(): if not rows: lines += [ - "No module is published yet. The modules that ship inside a release — `basicstats`, `association` and `mds` — are installed with it and are not listed here.", + "No module is published yet. No module ships inside a release either, so an installation starts with an empty store and stays that way until one is published here.", "", ] else: @@ -395,9 +399,9 @@ def render_nav(home: Page, sections: list[Section]) -> str: def reference_files() -> list[Path]: """Every references.bib: what a run cites, plus the manual's own context-only entries.""" found = [MANUAL.parent / "references.bib", - REPO / "install" / "references.bib", + REPO / "citations" / "references.bib", REPO / "analysis" / "references.bib"] - found += sorted((REPO / "analysis" / "modules").glob("*/references.bib")) + found += sorted((REPO / "modules").glob("*/references.bib")) return [path for path in found if path.exists()] diff --git a/dev/scripts/bump-analysis-version.sh b/dev/scripts/bump-analysis-version.sh index 5f8e80d..f52d623 100755 --- a/dev/scripts/bump-analysis-version.sh +++ b/dev/scripts/bump-analysis-version.sh @@ -9,8 +9,8 @@ # Three things carry one of these and each moves on its own: # # frame analysis/frame.version - frame.config and anything under analysis/lib/ -# index modules-repo/index.tsv - the #!index-version header, on every publish -# module analysis/modules//manifest.json +# index modules/repo/index.tsv - the #!index-version header, on every publish +# module modules//manifest.json, or modules/lib//manifest.json for a library # # The new value is today's UTC date and a counter: .001 the first time on a given day, then # .002 and so on. It writes one line and nothing else, and prints what it changed. @@ -48,9 +48,18 @@ next_version() { # The current value, per target. Empty when there is none to read. read_frame() { grep -vE '^[[:space:]]*(#|$)' "$REPO/analysis/frame.version" | head -1 | tr -d ' '; } read_index() { sed -n 's|^#![[:space:]]*index-version:[[:space:]]*\(.*\)$|\1|p' \ - "$REPO/modules-repo/index.tsv" | head -1 | tr -d ' '; } -read_module() { sed -n 's|.*"version"[[:space:]]*:[[:space:]]*"\([^"]*\)".*|\1|p' \ - "$REPO/analysis/modules/$1/manifest.json" | head -1; } + "$REPO/modules/repo/index.tsv" | head -1 | tr -d ' '; } +read_module() { sed -n 's|.*"version"[[:space:]]*:[[:space:]]*"\([^"]*\)".*|\1|p' "$1" | head -1; } + +# A module or a library, named the same way and versioned the same way. The caller gives a name +# and the repository says which it is. +module_manifest() { + local d + for d in "$REPO/modules/$1" "$REPO/modules/lib/$1"; do + [ -f "$d/manifest.json" ] && { printf '%s' "$d/manifest.json"; return 0; } + done + return 1 +} [ "$#" -ge 1 ] || usage TARGET="$1" @@ -67,7 +76,7 @@ case "$TARGET" in [ "$(read_frame)" = "$NEW" ] || { echo "ERROR: could not write $FILE." >&2; exit 1; } ;; index) - FILE="$REPO/modules-repo/index.tsv" + FILE="$REPO/modules/repo/index.tsv" [ -f "$FILE" ] || { echo "ERROR: $FILE not found." >&2; exit 1; } CURRENT=$(read_index) [ -n "$CURRENT" ] || { echo "ERROR: $FILE has no '#!index-version:' header." >&2; exit 1; } @@ -78,13 +87,15 @@ case "$TARGET" in module) [ "$#" -eq 2 ] || usage NAME="$2" - FILE="$REPO/analysis/modules/$NAME/manifest.json" - [ -f "$FILE" ] || { echo "ERROR: no module '$NAME' in $REPO/analysis/modules." >&2; exit 1; } - CURRENT=$(read_module "$NAME") + FILE=$(module_manifest "$NAME") || { + echo "ERROR: no module or library '$NAME'." >&2 + echo " Looked for modules/$NAME/manifest.json and modules/lib/$NAME/manifest.json" >&2 + exit 1; } + CURRENT=$(read_module "$FILE") [ -n "$CURRENT" ] || { echo "ERROR: $FILE has no 'version'." >&2; exit 1; } NEW=$(next_version "$CURRENT") sed -i -E "s|(\"version\"[[:space:]]*:[[:space:]]*\")[^\"]*(\")|\1${NEW}\2|" "$FILE" - [ "$(read_module "$NAME")" = "$NEW" ] || { echo "ERROR: could not write $FILE." >&2; exit 1; } + [ "$(read_module "$FILE")" = "$NEW" ] || { echo "ERROR: could not write $FILE." >&2; exit 1; } ;; *) usage diff --git a/dev/scripts/bump-version.sh b/dev/scripts/bump-version.sh index 18049bb..f971e60 100755 --- a/dev/scripts/bump-version.sh +++ b/dev/scripts/bump-version.sh @@ -5,12 +5,15 @@ # Usage: dev/scripts/bump-version.sh e.g. 1.0.2 # # Rewrites the version in the PoolSeqFlow wrapper (both the header comment and -# VERSION=) and in nextflow.config's manifest, sets the "environment" field of every -# shipped module's manifest and moves that module's own version with it, and prepends a +# VERSION=) and in nextflow.config's manifest, and prepends a # CHANGELOG section listing every commit since the last release tag under a "### Commits" # heading, along with the matching reference-link definition at the foot of the file. # Does not commit, tag, or push - it prints those commands for you. # +# It does NOT touch a module or library manifest. No module ships inside a release, so a +# module's "environment" is the oldest release its author says it needs, moved when its needs +# move - not a field a release bump may rewrite on its behalf. +# # Add release notes above that heading, not over it: the commit list stays in the # changelog as the record of what landed. @@ -106,35 +109,12 @@ sed -i -E "s|^(\s*version\s*=\s*)'.*'|\1'$NEW'|" "$NFCONFIG" grep -q "version *= *'$NEW'" "$NFCONFIG" || { echo "ERROR: could not update the manifest version in $NFCONFIG" >&2; exit 1; } -# Every module shipped inside the release declares the release whose analysis environment it was -# built against, and 00_static asserts that equals the manifest version in nextflow.config. A -# shipped module travels in the same tarball as the environment it names, so at a release the two -# are the same by construction and there is nothing to decide. -# -# Editing a manifest changes its module's directory, which its own YYYYMMDD.NNN version has to -# follow. That rule belongs to bump-analysis-version.sh and is called rather than repeated. -MODULES_BUMPED="" -for manifest in analysis/modules/*/manifest.json; do - [ -f "$manifest" ] || continue - name=$(basename "$(dirname "$manifest")") - grep -q "\"environment\": \"$NEW\"" "$manifest" && continue - sed -i -E "s|(\"environment\"[[:space:]]*:[[:space:]]*)\"[^\"]*\"|\1\"$NEW\"|" "$manifest" - grep -q "\"environment\": \"$NEW\"" "$manifest" || { - echo "ERROR: could not update the environment field in $manifest" >&2; exit 1; } - bash "$ROOT/dev/scripts/bump-analysis-version.sh" module "$name" > /dev/null - MODULES_BUMPED="$MODULES_BUMPED $name" -done - echo "$CURRENT -> $NEW" for wrapper in $WRAPPERS; do echo " $wrapper : $(grep -cF "$NEW" "$wrapper") references updated" done echo " $NFCONFIG : manifest version updated" echo " $LOG : $(printf '%s\n' "$COMMITS" | wc -l) commits since ${LAST_TAG:-start}, link definition added" -if [ -n "$MODULES_BUMPED" ]; then - echo " shipped modules :$MODULES_BUMPED" - echo " environment -> $NEW, and each version moved with it" -fi echo echo "Review, then:" echo " git add -A && git commit -m 'Version bump $NEW'" diff --git a/dev/scripts/changelog-section.sh b/dev/scripts/changelog-section.sh new file mode 100755 index 0000000..9d6389f --- /dev/null +++ b/dev/scripts/changelog-section.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# +# Print one version's section of CHANGELOG.md on stdout. +# +# Usage: dev/scripts/changelog-section.sh [--file ] +# +# The section is everything from `## []` up to whichever comes first: the next +# `## [` heading, the reference-link block at the foot of the file, or the end of the file. +# The `---` rule between sections belongs to neither, so it and the blank lines around it are +# trimmed; everything else is printed exactly as it stands, including the `### Commits` list. +# +# Exits non-zero with nothing on stdout when the version has no section. release.yml feeds +# this to the GitHub release body, so an absent section has to fail the release rather than +# publish an empty one. +# +# dev/ carries export-ignore, so this file never ships to a user. + +set -euo pipefail + +VERSION="" +FILE="" + +while [ $# -gt 0 ]; do + case "$1" in + --file) + FILE="${2-}" + [ -n "$FILE" ] || { echo "--file needs a path" >&2; exit 1; } + shift + ;; + -*) echo "unknown option: $1" >&2; exit 1 ;; + *) + [ -z "$VERSION" ] || { echo "unexpected argument: $1" >&2; exit 1; } + VERSION="$1" + ;; + esac + shift +done + +if [ -z "$VERSION" ]; then + echo "Usage: $0 [--file ] (e.g. 3.0.0)" >&2 + exit 1 +fi + +[ -n "$FILE" ] || FILE="$(git rev-parse --show-toplevel)/CHANGELOG.md" +[ -f "$FILE" ] || { echo "changelog-section.sh: no such file: $FILE" >&2; exit 1; } + +# The heading is matched as a literal, not a regular expression: a version is full of dots, and +# `## [3.0.1]` as a pattern would also match `## [3.0.10]`. The version reaches awk through +# ENVIRON rather than -v, which processes escape sequences in the value. +VERSION="$VERSION" awk ' + BEGIN { want = "## [" ENVIRON["VERSION"] "]"; n = 0 } + !inside && substr($0, 1, length(want)) == want { inside = 1 } + inside && NR > 1 { + # The next section, or the reference-link block that closes the file. + if (substr($0, 1, 4) == "## [" && substr($0, 1, length(want)) != want) exit + if ($0 ~ /^\[[0-9]+\.[0-9]+\.[0-9]+\]: /) exit + } + inside { body[n++] = $0 } + END { + if (n == 0) exit 3 + # The rule and the blank lines that bound one section from the next belong to neither. + last = n - 1 + while (last >= 0 && (body[last] == "" || body[last] == "---")) last-- + for (i = 0; i <= last; i++) print body[i] + if (last < 0) exit 4 + } +' "$FILE" || { + status=$? + if [ "$status" -eq 3 ]; then + echo "changelog-section.sh: ${FILE##*/} has no '## [$VERSION]' section" >&2 + elif [ "$status" -eq 4 ]; then + echo "changelog-section.sh: the '## [$VERSION]' section is empty" >&2 + fi + exit 1 +} diff --git a/dev/scripts/check-analysis-versions.sh b/dev/scripts/check-analysis-versions.sh index 801bd07..f5e734a 100755 --- a/dev/scripts/check-analysis-versions.sh +++ b/dev/scripts/check-analysis-versions.sh @@ -15,8 +15,9 @@ # bump - by hand while working, and as a release gate. # # frame analysis/frame.version covers analysis/frame.config and analysis/lib/ -# index the #!index-version header covers the rows in modules-repo/index.tsv -# module manifest.json's version covers that module's own directory +# index the #!index-version header covers the rows in modules/repo/index.tsv +# module manifest.json's version covers that module's or library's own directory, +# under modules/ and modules/lib/ # # It reads the working tree first and git second, so a change that is still uncommitted is # reported the same way as one that is already in. Exits 1 when anything is behind. @@ -161,7 +162,7 @@ fi # The catalogue. Its rows and its version live in ONE file, so the question is not which # changed last but whether the change that touched the rows also touched the header. -INDEX=modules-repo/index.tsv +INDEX=modules/repo/index.tsv index_rows() { grep -v '^[[:space:]]*#' "$1" | grep -v '^[[:space:]]*$' || true @@ -186,39 +187,43 @@ else fi # --------------------------------------------------------------------------------------- -# Each installed module, against its own manifest. - -if [ -d analysis/modules ]; then - for dir in analysis/modules/*/; do - [ -f "${dir}manifest.json" ] || continue - name=$(basename "$dir") - # A manifest that is not in HEAD yet is a module being added, and its version is new by - # construction - there is no earlier one it could have failed to move from. Without this - # every new module reports as behind, because `git diff HEAD` says nothing at all about - # an untracked file. - git cat-file -e "HEAD:${dir}manifest.json" 2>/dev/null || continue - # NOT THE MODULE'S OWN CASES. `analysis/modules/*/test/` carries export-ignore, so those - # files are in no published module and can change nothing a user installs - and the - # version is what an installation and every published result record the module BY. - if dirty "$dir" ":(exclude)${dir}test"; then - if ! git diff HEAD -- "${dir}manifest.json" | grep -q '^+.*"version"'; then - report "module '$name' changed and its manifest version did not" \ - "bump it: dev/scripts/bump-analysis-version.sh module $name" - fi - else - # Committed, which is the state a release is cut in: the last commit that touched - # the module has to be the one that moved its version. Without this the loop asks - # nothing at all of a clean tree, and every module passes a release unexamined. - last=$(git log -1 --format=%H -- "$dir" ":(exclude)${dir}test" 2>/dev/null || true) - if [ -n "${last:-}" ] \ - && ! git show "$last" -- "${dir}manifest.json" | grep -q '^+.*"version"'; then - report "module '$name' last changed in a commit that did not move its version" \ - "commit: $(git log -1 --format='%h %ad %s' --date=short -- "$dir" ":(exclude)${dir}test")" \ - "bump it: dev/scripts/bump-analysis-version.sh module $name" - fi +# Each module and each library SOURCE, against its own manifest. +# +# `analysis/modules/` is the install store: gitignored, empty in a checkout, and nothing git can +# say anything about. Reading it here checked nothing and reported success - so the paths below +# are the tracked sources, and `modules/lib/*/` is in the list because a library carries a +# manifest and a version exactly like a module. `modules/lib/` and `modules/repo/` are swept up +# by the first glob and drop out on the manifest test, which is what makes one loop enough. + +for dir in modules/*/ modules/lib/*/; do + [ -f "${dir}manifest.json" ] || continue + name=$(basename "$dir") + # A manifest that is not in HEAD yet is a module being added, and its version is new by + # construction - there is no earlier one it could have failed to move from. Without this + # every new module reports as behind, because `git diff HEAD` says nothing at all about + # an untracked file. + git cat-file -e "HEAD:${dir}manifest.json" 2>/dev/null || continue + # NOT THE MODULE'S OWN CASES. publish-module.sh drops `test/` from the tarball, so those + # files are in no published module and can change nothing a user installs - and the + # version is what an installation and every published result record the module BY. + if dirty "$dir" ":(exclude)${dir}test"; then + if ! git diff HEAD -- "${dir}manifest.json" | grep -q '^+.*"version"'; then + report "module '$name' changed and its manifest version did not" \ + "bump it: dev/scripts/bump-analysis-version.sh module $name" fi - done -fi + else + # Committed, which is the state a release is cut in: the last commit that touched + # the module has to be the one that moved its version. Without this the loop asks + # nothing at all of a clean tree, and every module passes a release unexamined. + last=$(git log -1 --format=%H -- "$dir" ":(exclude)${dir}test" 2>/dev/null || true) + if [ -n "${last:-}" ] \ + && ! git show "$last" -- "${dir}manifest.json" | grep -q '^+.*"version"'; then + report "module '$name' last changed in a commit that did not move its version" \ + "commit: $(git log -1 --format='%h %ad %s' --date=short -- "$dir" ":(exclude)${dir}test")" \ + "bump it: dev/scripts/bump-analysis-version.sh module $name" + fi + fi +done # --------------------------------------------------------------------------------------- diff --git a/dev/scripts/check-module-packages.sh b/dev/scripts/check-module-packages.sh index 617f16a..a35a1d6 100755 --- a/dev/scripts/check-module-packages.sh +++ b/dev/scripts/check-module-packages.sh @@ -92,10 +92,15 @@ conda env create -n "$ENV_NAME" -f "$BASELINE_FILE" >/dev/null BASELINE_PACKAGES=$(env_versions) printf ' %s packages\n' "$(printf '%s\n' "$BASELINE_PACKAGES" | wc -l | tr -d ' ')" -# Whatever the shipped modules declare. None does today, so this is a no-op that becomes the -# most important check in the script the moment one gains a dependency: it is the release's own -# environment being asked to take its own modules' pins. -SHIPPED=$(store_packages "$REPO_ROOT/analysis/modules") +# Whatever the modules and libraries this repository publishes declare - which is every one of +# them now, since a manifest states its dependencies whether or not the baseline already holds +# them. Read from the SOURCES: no module ships in a release any more, and analysis/modules/ is +# the install store, gitignored and empty here, so reading it asked this question of nothing. +# +# This is the release's own environment being asked to take its own modules' pins, and it is +# the only place the answer is a real conda solve rather than a stubbed command line. +SHIPPED=$( { store_packages "$REPO_ROOT/modules" + store_packages "$REPO_ROOT/modules/lib"; } | sort -u ) if [ -n "$SHIPPED" ]; then say "Installing what the shipped modules declare" printf ' %s\n' $SHIPPED diff --git a/dev/scripts/export-environment.sh b/dev/scripts/export-environment.sh index 0529d3e..ceda22f 100755 --- a/dev/scripts/export-environment.sh +++ b/dev/scripts/export-environment.sh @@ -78,18 +78,31 @@ fi # Only the module's OWN specs are looked for. What conda pulled in beneath them is not # distinguishable here from what the baseline needed anyway, which is exactly why the answer is # to rebuild the environment rather than to subtract from it. +# +# TWO STORES AND THE SOURCES. A module can reach this environment from an installation's store +# or by being installed out of this checkout, so both are read - and `modules/` with it, because +# `$REPO_ROOT/analysis/modules` is the checkout's own store, which is gitignored and empty and +# says nothing at all. Reading it alone left this guard answering over nothing. +# +# THE BASELINE IS SUBTRACTED. Every module declares what it needs whether or not the release +# already carries it, so `r-ggplot2` appears in a manifest and in environment-analysis.yml both. +# Without this the guard refuses every export, which is the same failure as refusing none. INSTALL="$REPO_ROOT" POOLSEQFLOW_INSTALLED_HOME="${POOLSEQFLOW_INSTALLED_HOME:-}" # shellcheck source=../../lib/wrapper_lib.sh . "$REPO_ROOT/lib/wrapper_lib.sh" INSTALLED_STORE="$(install_prefix)/opt/PoolSeqFlow-$VERSION/analysis/modules" +BASELINE=$(baseline_packages) DECLARED=$( { store_packages "$REPO_ROOT/analysis/modules" + store_packages "$REPO_ROOT/modules" + store_packages "$REPO_ROOT/modules/lib" store_packages "$INSTALLED_STORE"; } | sort -u ) HELD=$(conda_installed_packages "$ENV_NAME") CARRIED="" while IFS= read -r SPEC; do [ -n "$SPEC" ] || continue + printf '%s\n' "$BASELINE" | grep -qxF "${SPEC%%=*}" && continue if printf '%s\n' "$HELD" | grep -qxF "${SPEC%%=*}"; then CARRIED="$CARRIED $SPEC"$'\n' fi diff --git a/dev/scripts/prep-version.sh b/dev/scripts/prep-version.sh index 9927c8b..2731fa0 100755 --- a/dev/scripts/prep-version.sh +++ b/dev/scripts/prep-version.sh @@ -303,7 +303,7 @@ say " git diff install/environment.yml install/environment-analysis.yml" say " dev/scripts/bump-version.sh $NEW" say " ./PoolSeqFlow install # builds PoolSeqFlow-$NEW" say " ./PoolSeqFlow analysis install # builds PoolSeqFlow-$NEW-analysis" -say " ./PoolSeqFlow check" +say " ./PoolSeqFlow check install" say "" say "Both release environments are built by those installs, from the two exported files, so" say "what ships and what was tested are the same set - and the files, not long-lived" diff --git a/dev/scripts/publish-module.sh b/dev/scripts/publish-module.sh index 60e8bb2..d997e89 100755 --- a/dev/scripts/publish-module.sh +++ b/dev/scripts/publish-module.sh @@ -1,13 +1,13 @@ #!/usr/bin/env bash # -# Publish one analysis module into modules-repo/: build its tarball, add its catalogue row. +# Publish one analysis module into modules/repo/: build its tarball, add its catalogue row. # # Usage: dev/scripts/publish-module.sh [ref] # ref defaults to HEAD # -# Writes modules-repo/-.tar.gz, appends a row to modules-repo/index.tsv, and +# Writes modules/repo/-.tar.gz, appends a row to modules/repo/index.tsv, and # bumps the catalogue's #!index-version. It does not commit or push. The site deploys -# modules-repo/ wholesale, so the tarball and the row that advertises it go out together - +# modules/repo/ wholesale, so the tarball and the row that advertises it go out together - # which is why they are written in one step and not two. # # WHY A TARBALL IS A FILE HERE AND NOT SOMETHING GENERATED @@ -42,13 +42,26 @@ fi ROOT="$(git rev-parse --show-toplevel)" cd "$ROOT" -REPO_DIR="modules-repo" +# The source directory and the PUBLISHED path are different on purpose. The source sits under +# modules/ with everything else module-related; the published address may never move, because +# MODULE_INDEX_URL in lib/wrapper_lib.sh compiles into every release and asks for it for as long +# as that release exists. build_docs.py copies the one to the other. +REPO_DIR="modules/repo" +PUBLISHED_PATH="modules-repo" INDEX="$REPO_DIR/index.tsv" [ -f "$INDEX" ] || { echo "ERROR: $INDEX not found" >&2; exit 1; } -SRC="analysis/modules/$NAME" -git cat-file -e "$REF:$SRC/manifest.json" 2>/dev/null || { - echo "ERROR: no module '$NAME' at $REF ($SRC/manifest.json is not there)" >&2; exit 1; } +# A module or a library: both are a folder with a manifest, published the same way, and the +# manifest's own `kind` says which. Looked for in both places rather than taking a flag, so the +# caller names the thing and the repository says what it is. +SRC="" +for candidate in "modules/$NAME" "modules/lib/$NAME"; do + if git cat-file -e "$REF:$candidate/manifest.json" 2>/dev/null; then SRC="$candidate"; break; fi +done +[ -n "$SRC" ] || { + echo "ERROR: no module or library '$NAME' at $REF" >&2 + echo "Looked for modules/$NAME/manifest.json and modules/lib/$NAME/manifest.json" >&2 + exit 1; } # Straight out of the module's own manifest at that ref. The catalogue repeats what the manifest # says because the choice of which version to install is made before the tarball is downloaded. @@ -57,12 +70,16 @@ field() { | python3 -c "import json,sys; print(json.load(sys.stdin).get('$1',''))" } VERSION=$(field version) +KIND=$(field kind); [ -n "$KIND" ] || KIND="module" CONTRACT=$(field contract) FRAME=$(field frame) ENVIRONMENT=$(field environment) SUMMARY=$(field summary) -for pair in "version:$VERSION" "contract:$CONTRACT" "frame:$FRAME" \ +# A library that reads no published table declares no contract, and an empty column is read as +# "no requirement". Everything else is required of both kinds. +[ "$KIND" = "library" ] || REQUIRED_CONTRACT="contract:$CONTRACT" +for pair in "version:$VERSION" "${REQUIRED_CONTRACT:-version:$VERSION}" "frame:$FRAME" \ "environment:$ENVIRONMENT" "summary:$SUMMARY"; do [ -n "${pair#*:}" ] || { echo "ERROR: $NAME's manifest has no ${pair%%:*}" >&2; exit 1; } done @@ -90,8 +107,17 @@ fi # The commit that last touched the module at this ref, for a timestamp that follows the content # rather than the clock. -STAMP=$(git log -1 --format=%ct "$REF" -- "$SRC") -[ -n "$STAMP" ] || STAMP=$(git log -1 --format=%ct "$REF") +STAMP=$(git log -1 --format=%ct "$REF" -- "$SRC" 2>/dev/null || true) +[ -n "$STAMP" ] || STAMP=$(git log -1 --format=%ct "$REF" 2>/dev/null || true) +# A tree object has no commit and therefore no date of its own; fall back to the commit that +# last touched the source on the current branch. Without a timestamp tar stamps whatever it +# likes and two builds of one ref stop matching, which is the property this whole path exists +# for - so an empty stamp is refused rather than guessed. +[ -n "$STAMP" ] || STAMP=$(git log -1 --format=%ct HEAD -- "$SRC" 2>/dev/null || true) +[ -n "$STAMP" ] || { + echo "ERROR: no commit timestamp for $SRC at $REF, so the tarball would not be" >&2 + echo " reproducible. Commit the source and publish from a commit." >&2 + exit 1; } WORK=$(mktemp -d) trap 'rm -rf "$WORK"' EXIT @@ -99,7 +125,9 @@ mkdir -p "$WORK/$NAME" git archive --format=tar "$REF:$SRC" | tar -x -C "$WORK/$NAME" rm -rf "$WORK/$NAME/test" -for f in manifest.json main.nf citations.json; do +required="manifest.json" +[ "$KIND" = "module" ] && required="manifest.json main.nf citations.json" +for f in $required; do [ -f "$WORK/$NAME/$f" ] || { echo "ERROR: $NAME has no $f - install would refuse it" >&2 exit 1; } done @@ -109,23 +137,23 @@ tar --sort=name --format=gnu --owner=0 --group=0 --numeric-owner \ --mtime="@$STAMP" -C "$WORK" -cf - "$NAME" | gzip -n > "$TARBALL" SHA=$(sha256sum "$TARBALL" | awk '{print $1}') -URL="https://ozankiratli.github.io/PoolSeqFlow/$REPO_DIR/$(basename "$TARBALL")" +URL="https://ozankiratli.github.io/PoolSeqFlow/$PUBLISHED_PATH/$(basename "$TARBALL")" # Appended in the header's column order, which is the order the file declares and not one this # script decides. Read back and compared before anything else is written. HEADER=$(grep -v '^#' "$INDEX" | grep -v '^[[:space:]]*$' | head -1) -EXPECTED=$'name\tversion\tcontract\tframe\tenvironment\turl\tsha256\tsummary' +EXPECTED=$'name\tkind\tversion\tcontract\tframe\tenvironment\turl\tsha256\tsummary' [ "$HEADER" = "$EXPECTED" ] || { echo "ERROR: $INDEX's header is not the layout this script writes:" >&2 printf ' found: %s\n expected: %s\n' "$HEADER" "$EXPECTED" >&2 exit 1; } -printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \ - "$NAME" "$VERSION" "$CONTRACT" "$FRAME" "$ENVIRONMENT" "$URL" "$SHA" "$SUMMARY" >> "$INDEX" +printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \ + "$NAME" "$KIND" "$VERSION" "$CONTRACT" "$FRAME" "$ENVIRONMENT" "$URL" "$SHA" "$SUMMARY" >> "$INDEX" bash "$ROOT/dev/scripts/bump-analysis-version.sh" index > /dev/null -echo "Published $NAME $VERSION" +echo "Published $KIND $NAME $VERSION" echo " tarball : $TARBALL ($(stat -c%s "$TARBALL") bytes)" echo " sha256 : $SHA" echo " url : $URL" diff --git a/dev/scripts/select-tests.py b/dev/scripts/select-tests.py index e0dd6b6..5f49ac0 100644 --- a/dev/scripts/select-tests.py +++ b/dev/scripts/select-tests.py @@ -3,7 +3,7 @@ dev/scripts/select-tests.py # against the working tree dev/scripts/select-tests.py --ref HEAD~3 # against a commit - dev/scripts/select-tests.py scripts/7_vcf2freq.nf analysis/lib/R/n_eff.R + dev/scripts/select-tests.py scripts/7_vcf2freq.nf modules/lib/n_eff/n_eff.R dev/scripts/select-tests.py --command # print the run_tests.sh line and nothing else HOW IT DECIDES, in two halves. @@ -43,7 +43,7 @@ def suites(): """Every suite file, with the source paths its header claims.""" found = {} roots = [os.path.join(ROOT, "test", "suites")] - modules = os.path.join(ROOT, "analysis", "modules") + modules = os.path.join(ROOT, "modules") if os.path.isdir(modules): roots += [os.path.join(modules, m, "test") for m in sorted(os.listdir(modules))] for directory in roots: @@ -66,7 +66,12 @@ def sources(): listed = subprocess.run(["git", "ls-files"], capture_output=True, text=True, cwd=ROOT) extra = subprocess.run(["git", "ls-files", "--others", "--exclude-standard"], capture_output=True, text=True, cwd=ROOT) - return [p for p in (listed.stdout + extra.stdout).split("\n") if p] + # Only what is actually THERE. `git ls-files` reads the index, which still names a file + # that has been moved or deleted in the working tree until the change is staged - and a + # path nothing can land in any more is not a source. Without this, every check built on + # sources() reports files that do not exist. + return [p for p in (listed.stdout + extra.stdout).split("\n") + if p and os.path.exists(os.path.join(ROOT, p))] def graph(): diff --git a/dev/scripts/verify-archive.sh b/dev/scripts/verify-archive.sh index 74722e4..2419bb9 100755 --- a/dev/scripts/verify-archive.sh +++ b/dev/scripts/verify-archive.sh @@ -46,12 +46,12 @@ done # as well as out of the archive, and the check would pass. So an export-ignore fails here until # it is named in this list too. # -# modules-repo/ is the whole directory and not just the catalogue in it. It holds the published -# module tarballs as well, and those are served from the site: a copy inside a release would be -# a second answer to what can be installed, frozen on the day the release was built. +# modules/ covers the module and library sources AND modules/repo/, the published catalogue and +# the tarballs beside it. Those are served from the site: a copy inside a release would be a +# second answer to what can be installed, frozen on the day the release was built. excluded='docs/ .github/ mkdocs.yml .gitignore .gitattributes dev/ Project/ test/ .claude/ CLAUDE.md - analysis/modules/*/test/ modules-repo/' + modules/' # Whether one tracked path is meant to reach the archive at all. ships() { @@ -85,12 +85,21 @@ for f in docs .github mkdocs.yml .gitignore .gitattributes dev Project .claude C [ ! -e "$root/$f" ] || fail "should have been export-ignored: $f" done -# The same for a module's own cases, which no wildcard above would have caught: `ships` only -# says a path need not be there, and a rule that stopped working would go unnoticed without -# somebody looking for the file. -for d in "$root"/analysis/modules/*/test; do - [ ! -e "$d" ] || fail "should have been export-ignored: ${d#$root/}" -done +# NO MODULE SHIPS INSIDE A RELEASE, which is the point of the module system: a module carried +# in the tarball is one nobody installed, that `modules install` then refuses to replace with a +# newer version, and that `modules uninstall` would strip out of the release itself. +# +# Asserted by looking, not by the `excluded` list above - that list only says a path NEED NOT be +# in the archive, so a rule that stopped working would leave the modules in it and nothing would +# say so. +# +# Two directories, because the sources and the store are deliberately different places. modules/ +# is the tracked source of every module and of the libraries they install alongside themselves; +# analysis/modules/ is the STORE those are installed into, gitignored and empty in a checkout. +# While the two were one directory, a release shipped the modules because they were sources +# sitting in the install path - which is the whole reason they are apart now. +[ ! -e "$root/modules" ] || fail "modules/ shipped in the archive" +[ ! -e "$root/analysis/modules" ] || fail "the module store shipped in the archive" # Compiled Python must not ship. Checked here and not by the executable-bit loop below, which a # directory passes. diff --git a/dev/validation/README.md b/dev/validation/README.md index 9da75dd..16e3ee7 100644 --- a/dev/validation/README.md +++ b/dev/validation/README.md @@ -12,7 +12,7 @@ Nothing here runs in `run_tests.sh` and nothing here gates a commit. **What ship ## The discipline that makes it worth running -**The generator never calls `analysis/lib/R/`.** It draws with `rbinom` and arithmetic and nothing else. +**The generator never calls the libraries under `modules/lib/`.** It draws with `rbinom` and arithmetic and nothing else. This is the whole reason a measurement here is evidence. If the simulation drew its read counts using our own `n_eff`, a wrong `n_eff` would cancel on both sides and the harness would report agreement to twelve figures while being wrong about the world. The library is on trial; it does not get to write the exam. @@ -42,7 +42,7 @@ The same rule holds for every section added later: the truth is constructed from Both source `lib.R`, which holds every generator and estimator they share. They fit and permute identically or neither result says anything about the other, so nothing is defined twice. ``` -Rscript dev/validation/external.R analysis/lib/R /tmp/bp 1200 +Rscript dev/validation/external.R modules/lib /tmp/bp 1200 ``` BayPass is `g_baypass` on the PATH, and it lives in an environment of its own: @@ -59,9 +59,9 @@ conda activate poolseqflow-validation Run one section, or all of them: ``` -Rscript dev/validation/calibrate.R analysis/lib/R -Rscript dev/validation/calibrate.R analysis/lib/R n_eff -Rscript dev/validation/calibrate.R analysis/lib/R n_eff 500000 +Rscript dev/validation/calibrate.R modules/lib +Rscript dev/validation/calibrate.R modules/lib n_eff +Rscript dev/validation/calibrate.R modules/lib n_eff 500000 ``` The third argument is the replicate count per cell; it defaults low enough to run in seconds and high enough to separate the hypotheses. Every run prints the seed it used and is reproducible from it. diff --git a/dev/validation/calibrate.R b/dev/validation/calibrate.R index 958885e..31cd6f7 100755 --- a/dev/validation/calibrate.R +++ b/dev/validation/calibrate.R @@ -21,7 +21,9 @@ section <- if (length(args) > 1) args[2] else "all" replicates <- if (length(args) > 2) as.integer(args[3]) else 200000L seed <- if (length(args) > 3) as.integer(args[4]) else 20260907L -sources <- list.files(lib, pattern = "[.]R$", full.names = TRUE) +# `recursive` because a library is a DIRECTORY under modules/lib/ and its .R sits inside it, so +# a flat listing of the store or the source tree returns nothing at all. +sources <- list.files(lib, pattern = "[.]R$", full.names = TRUE, recursive = TRUE) if (length(sources) == 0) stop("no R sources in ", lib) for (path in sources) source(path) diff --git a/dev/validation/external.R b/dev/validation/external.R index 12ab78e..30db7af 100644 --- a/dev/validation/external.R +++ b/dev/validation/external.R @@ -28,7 +28,12 @@ work <- args[2] sites <- if (length(args) > 2) as.integer(args[3]) else 2000L seed <- if (length(args) > 3) as.integer(args[4]) else 20260907L -for (path in list.files(lib, pattern = "[.]R$", full.names = TRUE)) source(path) +# `recursive` because a library is a DIRECTORY under modules/lib/ and its .R sits inside it, so +# a flat listing returns nothing - and sourcing nothing here would score BayPass against an +# empty library rather than against ours, which is a result that looks like a finding. +sources <- list.files(lib, pattern = "[.]R$", full.names = TRUE, recursive = TRUE) +if (length(sources) == 0) stop("no R sources in ", lib) +for (path in sources) source(path) here <- dirname(sub("^--file=", "", grep("^--file=", commandArgs(FALSE), value = TRUE)[1])) source(file.path(here, "lib.R")) diff --git a/lib/wrapper_lib.sh b/lib/wrapper_lib.sh index 74988d5..0efcf3d 100644 --- a/lib/wrapper_lib.sh +++ b/lib/wrapper_lib.sh @@ -53,7 +53,7 @@ nf_config_value() { } # Zenodo all-versions DOI. A release's own DOI is reached through it. Also recorded in -# install/citations.json, which the per-run CITATIONS.md is built from. +# citations/citations.json, which the per-run CITATIONS.md is built from. CONCEPT_DOI="10.5281/zenodo.19245611" # The catalogue of modules that can be installed. It is export-ignored, so a release carries no @@ -112,7 +112,7 @@ module_contract() { } # The order this release reads a catalogue row in, whatever order the file writes them in. -MODULE_INDEX_COLUMNS="name version contract frame environment url sha256 summary" +MODULE_INDEX_COLUMNS="name kind version contract frame environment url sha256 summary" # What separates the fields of a normalized row, and it is NOT the tab the file uses. # @@ -235,6 +235,27 @@ analysis_r_packages() { sed -n 's/^ *- *r-\([^=]*\).*$/\1/p' "$ENV_FILE" | grep -vx base } +# Every package the release's own analysis environment is built from, one name per line, with no +# version. This is the baseline: what `analysis install` creates before any module is installed. +# +# NOTHING HERE MAY BE REMOVED BY A MODULE UNINSTALL. A module declares what it needs whether or +# not the baseline already has it - that is what makes its manifest a true statement of its +# dependencies rather than a statement about one release's environment - so the set a module +# declares and the baseline overlap by design, and the overlap belongs to the release. +# +# Reads the shipped file rather than the live environment: the live one has whatever modules +# added merged into it and cannot say which packages are the release's own. +# +# Defined here and not only in the wrapper, because baseline_packages() reads it and every +# dev/ script that sources this file needs the same answer. An unset path makes the sed below +# silently produce nothing, which reads as "the baseline is empty" and subtracts nothing. +ANALYSIS_ENV_FILE="${ANALYSIS_ENV_FILE:-${INSTALL:-}/install/environment-analysis.yml}" + +baseline_packages() { + sed -n 's/^ *- *\([A-Za-z0-9][A-Za-z0-9._-]*\).*$/\1/p' "$ANALYSIS_ENV_FILE" 2>/dev/null \ + | grep -vx 'pip' | sort -u +} + # The shape a module's `packages` entry must have: a name, one `=`, an exact version. No build # string, no range, no channel prefix. The analysis frame applies the same rule when a module # runs; this is what refuses one before it is installed. @@ -246,10 +267,48 @@ MODULE_SPEC_RE='^[a-z0-9][a-z0-9._-]*=[A-Za-z0-9][A-Za-z0-9._+]*$' module_packages() { local manifest="$1" [ -f "$manifest" ] || return 0 + # awk for the last step and not sed: it terminates its final record. store_packages runs + # this once per module and concatenates the results, so an unterminated last line arrives + # joined to the next module's first one as a single token. tr '\n' ' ' < "$manifest" \ | sed -n 's/.*"packages"[[:space:]]*:[[:space:]]*\[\([^]]*\)\].*/\1/p' \ | tr ',' '\n' \ - | sed -n 's/^[^"]*"\([^"]*\)".*/\1/p' + | awk -F'"' 'NF > 1 { print $2 }' +} + +# The libraries one module declares, one per line, read out of its manifest the same way as its +# packages. A module with no `libraries` field yields nothing. +module_libraries() { + local manifest="$1" + [ -f "$manifest" ] || return 0 + # awk for the last step and not sed, for the reason module_packages gives: store_libraries + # concatenates one of these per module and an unterminated last line glues two names. + tr '\n' ' ' < "$manifest" \ + | sed -n 's/.*"libraries"[[:space:]]*:[[:space:]]*\[\([^]]*\)\].*/\1/p' \ + | tr ',' '\n' \ + | awk -F'"' 'NF > 1 { print $2 }' +} + +# Where installed libraries live: inside the module store, under a name no module may take. +# analysis/lib/nf/modules.nf resolves the same path when a module runs. +LIBRARY_DIR_NAME="lib" + +library_store() { + printf '%s' "$MODULE_STORE/$LIBRARY_DIR_NAME" +} + +# Every library the modules in a store declare, sorted and deduplicated, optionally skipping one +# module by name. This is what says whether a library is still wanted after a module leaves. +store_libraries() { + local store="$1" skip="${2:-}" dir name + [ -d "$store" ] || return 0 + for dir in "$store"/*/; do + [ -d "$dir" ] || continue + name=$(basename "$dir") + if [ "$name" = "$LIBRARY_DIR_NAME" ]; then continue; fi + if [ -n "$skip" ] && [ "$name" = "$skip" ]; then continue; fi + module_libraries "$dir/manifest.json" + done | sort -u } # Every spec the modules in a store declare, sorted and deduplicated, optionally skipping one @@ -260,6 +319,7 @@ store_packages() { for dir in "$store"/*/; do [ -d "$dir" ] || continue name=$(basename "$dir") + if [ "$name" = "$LIBRARY_DIR_NAME" ]; then continue; fi if [ -n "$skip" ] && [ "$name" = "$skip" ]; then continue; fi module_packages "$dir/manifest.json" done | sort -u @@ -338,3 +398,22 @@ conda_remove_packages() { fi conda remove -n "$env" -y "$@" } + +# Is $1 a parameters.config written for THIS release? Every config for this release sets +# storageDir and no earlier one did, so that single key answers it. +# +# Shared because two callers ask the same question and must not drift: the wrapper refuses a +# stale config before any command that reads one, and `check project` reports it as a line. +config_is_current() { + grep -qE '^[[:space:]]*storageDir[[:space:]]*=' "$1" 2>/dev/null +} + +# The parameters this release renamed or removed that $1 still sets, space-separated. Advisory: +# it names an old file as recognized rather than damaged, and migrate_config reports the full set. +config_stale_parameters() { + local old found="" + for old in projectDir diploidy rgTagsFile rgTagsPath; do + grep -qE "^[[:space:]]*${old}[[:space:]]*=" "$1" 2>/dev/null && found="$found $old" + done + printf '%s' "$found" +} diff --git a/manual/PoolSeqFlow-manual.md b/manual/PoolSeqFlow-manual.md index abb1fee..315f481 100644 --- a/manual/PoolSeqFlow-manual.md +++ b/manual/PoolSeqFlow-manual.md @@ -191,7 +191,7 @@ Check the [requirements](#getting-started) first if you have not — in particul cd PoolSeqFlow-*/ ``` - This is the recommended route. The archive is the pipeline only — no documentation sources or CI config — and it extracts into a versioned directory, so you always know which release a working copy came from. + This is the recommended route. The archive is the pipeline, the analysis frame and this manual — no site sources, no CI config, no test suite, and **no analysis modules**. Every module is published on its own timetable and installed from the catalogue, so a new installation starts with an empty module store and you choose what goes into it. It extracts into a versioned directory, so you always know which release a working copy came from. Verify it if you like: download `SHA256SUMS` from the same release and run `sha256sum -c SHA256SUMS`. @@ -273,37 +273,82 @@ Analyzing one set of reads under several parameter sets — two reference genome ### 4. Verify it any time { #check } +**There are two checks and they answer different questions**, so `check` takes a word and refuses without one. An installation is a tool; a project is your configuration and your data. Neither answers the other, and a bare `check` would have to guess which you meant — leaving the other unchecked without saying so. + ```bash -./PoolSeqFlow check +./PoolSeqFlow check install # the tools and helpers this release is built to run +./PoolSeqFlow check project # the configuration and the commands this project names ``` +Both end by reporting how many checks passed, and **fail loudly if any did not** rather than summarizing. + +#### `check install` { #check-install } + ```text Tools + from ~/.local/opt/miniconda3/envs/PoolSeqFlow-3.0.0 nextflow nextflow OK 26.04.6 build 12646 + cutadapt cutadapt OK 5.2 + … samtools samtools OK samtools 1.24 bcftools bcftools OK bcftools 1.24 … - tool list from: params.software in parameters.config Pipeline helpers atomic_mv.sh OK depth2freq.awk OK … +``` + +**Every command the release is built to run**, with the version each reports. The list is the canonical one — what this release expects its environment to provide. + +**And each has to come from that environment.** Every tool on the list is pinned in `install/environment.yml`, so one that resolves from anywhere else means the environment is missing a package and your system's copy is standing in — at whatever version it happens to be, and only on this machine. That is reported, not passed over: + +```text + samtools samtools OUTSIDE THE ENVIRONMENT /usr/bin/samtools +``` + +It is the failure worth catching, because it is the quiet one: the pipeline runs, the results look fine, and nothing reproduces anywhere else. Reinstalling is the fix — `PoolSeqFlow install`. + +**Every helper in `bin/`**, present and executable. `nextflow.config` puts that directory on `PATH` and the process scripts call the helpers by bare name, so a lost executable bit fails mid-run rather than at startup. `bin/` is enumerated rather than listed, so a helper added to a release is checked without anyone remembering to say so. + +The three `check_*.sh` scripts live there too — `bin/` is where everything that is *run* rather than sourced lives — and are the one thing skipped: they are run by the command line, never by a process script, so a run does not depend on them. `lib/` is not enumerated at all, because what is in there is sourced rather than run, which is the whole reason the two directories are separate. + +**It reads no `parameters.config` and needs no project.** Run it from anywhere, including straight after installing and before you have a project at all. + +#### `check project` { #check-project } +Run it from your project directory. + +```text Configuration + parameters.config WRITTEN FOR THIS RELEASE parameters.config PARSES + metadata.csv PARSES + runs.csv not in use + +Tools, as this project configures them + + nextflow nextflow OK 26.04.6 build 12646 + samtools /usr/bin/samtools OK samtools 1.19 + bcftools bcftools OK bcftools 1.24 + … + + tool list from: params.software in parameters.config ``` -It ends by reporting how many checks passed, and **fails loudly if any did not** rather than summarizing. It covers three things: +**Your files parse** — `parameters.config` through Nextflow itself, and `metadata.csv` and the run table through the same parsers step 0 uses, so what you are told here is what a run would tell you. It also says whether the config was written for this release, which is the one thing that stops a run before anything else is read. + +**Every command, as *you* configured it.** The list comes from `params.software`, so a command [repointed at a system binary](#using-system-tools) is checked the way the run will call it — and the second column shows what will actually be invoked. That override is the setting most likely to be wrong and least likely to announce itself. -**Every command the pipeline invokes**, with the version each reports. Once you have a `parameters.config`, the list is read from `params.software` through `nextflow config` rather than assumed — so a command [repointed at a system binary](#using-system-tools) is checked as *you* configured it. That override is the setting most likely to be wrong and least likely to announce itself. +**A path outside the environment is not a finding here.** `check install` treats one as a fault, because the installation is supposed to provide its own tools; `check project` does the opposite, because repointing one is a thing you are allowed to do and this is where you see the result. It reports what will be invoked and whether it runs, and leaves the judgment to you. -**Every helper in `bin/`**, present and executable. `nextflow.config` puts that directory on `PATH` and the process scripts call the helpers by bare name, so a lost executable bit fails mid-run rather than at startup. `lib/` is checked for presence only — what is in there is sourced by another script rather than run, which is the whole reason the two directories are separate. +A project that repoints nothing gets the same tool answers from both checks. **That is the point**: the difference between them is exactly your own configuration. -**That `parameters.config` parses**, once it exists. +If `params.software` cannot be read, `check project` says so and checks no tool. It never falls back to the canonical list — the whole reason the section exists is that it reads yours. --- @@ -465,7 +510,8 @@ How to read the tables: [Interpreting Results](#interpreting-results). | `PoolSeqFlow install` | Create this release's conda environment, install the pipeline, then verify both | | `PoolSeqFlow init` | Populate the current directory as a project ([what it writes](#3-make-your-project)) | | `PoolSeqFlow init_multi` | The same, for a project running several parameter sets over one set of reads | -| `PoolSeqFlow check` | Verify an existing installation ([what it covers](#check)) | +| `PoolSeqFlow check install` | Verify an installation — the tools and helpers it is built to run ([what it covers](#check-install)) | +| `PoolSeqFlow check project` | Verify a project — its configuration, and the commands it names ([what it covers](#check-project)) | | `PoolSeqFlow run` | Start — or resume — the pipeline | | `PoolSeqFlow dryrun` | Create the directory tree the run would write, empty, so the layout can be approved before any compute is spent. Records nothing and changes none of your files | | `PoolSeqFlow dryclean` | Remove the preview `dryrun` made | @@ -532,6 +578,15 @@ Download and install it exactly as you did the first time. Nothing is replaced: Your projects are untouched by this. `parameters.config` lives in your project directory, not in the installation, so nothing an install or an uninstall does can reach it. +**The new release's module store starts empty.** No module ships inside a release, and the store belongs to the installation that runs it — so the old release keeps everything you installed into it and the new one has nothing. Ask the old one what to reinstall, then install each into the new one: + +```bash +PoolSeqFlow-3.0.0 analysis modules list # what the old release has +PoolSeqFlow analysis modules install mds # and again, into the new one +``` + +Nothing is lost by this. A module installs the libraries it needs along with it, and the analyses it has already produced are results — no install or uninstall reaches them. + ### A project belongs to one release This is the part that changes how upgrading works. Completed steps are skipped by looking for output files, not by checking what produced them, so continuing an existing project under a new release would leave one set of results built by two versions of the pipeline, with nothing on disk to say which is which. @@ -771,7 +826,7 @@ Standard, optional, and unchanged in approach. It runs only when you ask for it, ### The analysis layer -The modules that read these tables — what each one estimates, what it assumes and what it cannot tell you — are documented with the modules themselves in [Shipped Modules](#shipped-modules). This section will grow as they do. +The modules that read these tables — what each one estimates, what it assumes and what it cannot tell you — are documented with the modules themselves in [Modules](#modules). This section will grow as they do. ## Design Decisions @@ -1749,9 +1804,12 @@ There are three directories, and keeping them apart is most of understanding the ```text ~/.local/opt/PoolSeqFlow-/ -├── bin/ # Run by the pipeline; all executable, all on PATH +├── bin/ # Run, never sourced; all executable, all on PATH │ ├── atomic_mv.sh # Cross-filesystem moves, staged and renamed │ ├── cap_depth.awk # Truncate a BAM to a depth ceiling +│ ├── check_install.sh # Verifies an installation (PoolSeqFlow check install) +│ ├── check_project.sh # Verifies a project (PoolSeqFlow check project) +│ ├── check_analysis_install.sh # The analysis layer, R packages included │ ├── classify_manifest.sh # Sorts a parameter change into added/changed/removed │ ├── config_migrate.sh # Backs migrate_config │ ├── createDepthFile.sh # Extract AD/DP columns from a VCF @@ -1765,12 +1823,13 @@ There are three directories, and keeping them apart is most of understanding the │ └── write_citations.py # Writes CITATIONS.md and references.bib per run ├── lib/ # Sourced by another script, never run; not on PATH │ ├── tool_version.sh # Asks each tool its version, one way per tool -│ └── wrapper_lib.sh # Machinery shared by the wrapper and the install checks -├── install/ -│ ├── environment.yml # Pinned conda environment -│ ├── environment-analysis.yml # Pinned conda environment for the analysis layer -│ ├── check_install.sh # Verifies an installation (PoolSeqFlow check) -│ └── check_analysis_install.sh # The same for the analysis layer, R packages included +│ └── wrapper_lib.sh # Machinery shared by the wrapper and the checks +├── install/ # The pinned environments, and nothing else +│ ├── environment.yml # The pipeline's +│ └── environment-analysis.yml # The analysis layer's +├── citations/ # What the pipeline itself cites +│ ├── references.bib # Authored; edit this one +│ └── citations.json # Generated from it, and what a run reads ├── scripts/ │ ├── 0_verify_environment.nf # The nine checks that gate everything else │ ├── 1_build_dictionaries.nf # BWA, SAMtools and SnpEff indices from your reference @@ -1786,18 +1845,33 @@ There are three directories, and keeping them apart is most of understanding the │ ├── metadata.nf # Reading metadata.csv, and the projections from it │ ├── resolve_parameters.nf # Computed parameters, and one parameter set per run │ └── variants.nf # Which runs share which work, and where it goes +├── analysis/ # The analysis frame. No module is part of it +│ ├── lib/nf/ # The workflow library a module imports +│ ├── lib/rmd/report.Rmd # The PDF report every analysis carries +│ ├── modules.nf # Finds and dispatches an installed module +│ ├── 0_verify_analysis.nf # The builtin verify module +│ ├── complete.nf # Promotion for analysis results +│ ├── frame.config +│ ├── frame.version # What a module declares it needs +│ ├── analysis.config.template +│ ├── citations.json +│ ├── references.bib +│ └── modules/ # THE MODULE STORE — created empty, and yours to fill ├── manual/ # This manual ├── nextflow.config ├── parameters.config.template ├── metadata.csv.template ├── multi-run.csv.example ├── poolseqflow.nf # Workflow entry point +├── analysis.nf # Entry point for the analysis layer ├── dryrun.nf # Entry point for the layout preview └── PoolSeqFlow # CLI wrapper, pipeline and analysis layer alike ``` One copy serves any number of projects, and it is replaced wholesale when you upgrade — which is why nothing of yours belongs in it. `bin/` is prepended to `PATH` by `nextflow.config`, which is how the helper scripts are callable by bare name inside process scripts. +**`analysis/modules/` is the one part of the installation you add to.** No module arrives with a release, so a fresh copy has an empty store; `analysis modules install ` puts a module there along with the libraries it declares, which live together under `analysis/modules/lib/`. It is still part of the installation and not part of a project — replaced wholesale on upgrade, like everything else here, which is why [upgrading](#upgrading) means installing your modules again. + ### What you provide, on `mainDir` ```text @@ -3086,7 +3160,9 @@ cd /path/to/project PoolSeqFlow analysis verify ``` -Four ship with the release — `verify`, `basicstats`, `association` and `mds` — and each has a page of its own under [Shipped Modules](#shipped-modules), which is where what they compute and what they assume is written down. Every other module is installed separately and published on its own timetable. +**Only `verify` comes with the pipeline.** It is part of the analysis frame itself, reports what the layer can see and produces nothing. Every other module — `basicstats`, `association` and `mds` among them — is published on its own timetable and installed from the catalogue, so a fresh installation has an empty store and you choose what goes into it. Each has a page of its own under [Modules](#modules), which is where what it computes and what it assumes is written down. + +A module is installed with the **libraries** it declares: the shared arithmetic more than one module wants, each one published and versioned like a module and installed into `analysis/modules/lib/`. You never ask for a library by name; it arrives with whatever needs it, and leaves when nothing installed still declares it. ### The modules installed here { #analysis-modules } @@ -3116,7 +3192,7 @@ All of them read the installation rather than your project, so they work from an #### What installing a module does to your environment { #module-packages } -**There is one analysis environment per release and every module shares it.** A module that needs an R package the release does not ship names it in its manifest, pinned to an exact version, and `modules install` puts it in that shared environment. No module shipped with this release names one — they run on base R plus what the environment already carries — so this is what you will see when you install one of the modules published separately: +**There is one analysis environment per release and every module shares it.** A module names every R package it needs in its manifest, pinned to an exact version, and `modules install` puts them in that shared environment. It names them whether or not the release's own environment already carries them: the manifest is a statement of what the module needs, not of what one release happens to provide, and removing a module never takes a package the release itself is built on. This is what you see when you install one: ``` Installing what fst runs on, into 'PoolSeqFlow-3.0.0-analysis': @@ -3136,7 +3212,7 @@ Either way it is a compatibility question between a release and a module, settle **Uninstalling takes back only what nothing else asks for.** If two modules both name `r-poolfstat=3.0.0`, removing one leaves it installed for the other. And because `conda remove` takes everything that depends on what it is given, the removal is planned before it is run: if taking a package out would take something else with it, nothing is removed and the module stays installed. -**Two things follow from the store living inside the installation.** Reinstalling the pipeline over itself wipes the store, so the packages its modules added are taken out of the environment first, while the manifests declaring them still exist — afterwards both are back to what the release ships. And `analysis uninstall` removes the environment while leaving the store, so `analysis install` puts back what the modules still there need. Neither is something to manage by hand. +**Two things follow from the store living inside the installation.** Reinstalling the pipeline over itself wipes the store, so the packages its modules added are taken out of the environment first, while the manifests declaring them still exist — afterwards both are back to what a fresh installation is, which is empty, and the modules you want are installed again. And `analysis uninstall` removes the environment while leaving the store, so `analysis install` puts back what the modules still there need. Neither is something to manage by hand. `list` is worth knowing about before you need it. Modules live inside the release's own installation, so each release has its own set and a module installed for one is never picked up by another — reinstalling the same version wipes them, and one command puts each back. More usefully: **a module directory that has lost its pipeline stops every analysis run, not only its own**, and `list` is what names the one at fault. It also tells you where the store is, which is the directory a module is installed into. @@ -3626,7 +3702,7 @@ The kinds are the phenotype's — `quantitative`, `binary`, `ordinal`, `nominal` #### What adjusting for one costs, and why it is a decision { #covariates-not-adjusted } -**The frame never adjusts for anything.** It resolves each covariate, says which are [part of the design](#design-covariates), and publishes both. Whether a module puts a covariate in its model is that module's business, declared in its own section — and no module ships without saying. +**The frame never adjusts for anything.** It resolves each covariate, says which are [part of the design](#design-covariates), and publishes both. Whether a module puts a covariate in its model is that module's business, declared in its own section — and no module is published without saying. **The arithmetic is why it has to be your decision.** *n* is the number of **pools**, typically six to twenty, so every covariate a model fits is a degree of freedom the effect you came for does not get. At six pools a comparison starts with four; a single covariate makes it three; a repeated-measures design of three units has one left before any covariate at all. @@ -3721,7 +3797,7 @@ Everything the analysis layer produces goes under `Analysis/` — on `mainDir` w **Every published analysis carries the script that produced it.** That is a guarantee of the layer rather than a convention module authors are asked to follow: an analysis handed over without one is refused, nothing is published, and the folder is left exactly as it was. So a result you find in `Analysis/Results` can always be regenerated, and the folder also holds the verification record that cleared it — which names the module, its version, the runs it covered and the configuration it was assembled from. -**It carries the shared library it used, folded in.** A module sources functions the analysis layer provides — effective sample size, gene diversity, the harmonic means — and what is published is those functions and the module's own code in one file, stamped with the frame version that defined them. A driver that merely `source()`s code the reader does not have would satisfy the letter of the guarantee and none of its point. +**It carries the libraries it used, folded in.** A module declares the libraries it needs — effective sample size, gene diversity, the harmonic means — and each is installed alongside it; what is published is those functions and the module's own code in one file, headed by the frame version it ran against. A driver that merely `source()`s code the reader does not have would satisfy the letter of the guarantee and none of its point — and a library lives in the module store, which you can uninstall from. The file in your results is the copy that ran, whatever the store holds afterward. **And a `README.md` that says how to read what is in the folder.** { #analysis-readme } Every module declares, for each file it publishes, the section of this manual that explains it; the frame renders one table from those declarations and from what every analysis carries anyway: @@ -3793,10 +3869,10 @@ The cost of a working cycle is therefore one transfer, not two — which matters **Do not run it while a module is running.** The two would be moving the same folders in opposite directions, and neither checks for the other. -# Shipped Modules +# Modules -Most modules are installed separately. These are the ones a release carries, so they are available the moment the analysis environment exists, and their versions move on the pipeline's timetable rather than a catalogue's. +**Every module is installed separately** — none comes with the pipeline, and a fresh installation has an empty store. These are the ones published alongside this release, each on its own timetable and at its own version, installed with `PoolSeqFlow analysis modules install ` and documented here so you can read what one computes before deciding to install it. | Module | What it computes | Needs | |---|---|---| @@ -4527,7 +4603,7 @@ Some things cannot be asserted from inside the suite, so they are separate gates - **`nextflow lint`** over every workflow file, at zero errors *and* zero warnings. The strict parser rejects a good deal of ordinary Groovy, and it reports a parse failure in one file as "not defined" at every call site in *other* files — so a clean lint is worth more than it sounds. - **The citation check.** Every reference is authored once in BibTeX and compiled to the JSON the pipeline reads; the gate regenerates it and fails if the two disagree, so the file a run cites from cannot drift from the file I edit. - **The manual check.** This manual is one file, and every page of the site is generated from it. The gate re-parses it, resolves every cross-reference, and fails on a link to a heading that no longer exists or two headings that would collide — which is what stops the documentation rotting quietly as things are renamed. -- **The version check**, which fails when a shared library has changed without its version moving, and the **archive check**, which builds the release tarball and asserts that everything a user needs is in it and everything they do not need is out. +- **The version check**, which fails when the analysis frame, a module or a library has changed without its own version moving, and the **archive check**, which builds the release tarball and asserts that everything a user needs is in it and everything they do not need is out — modules included, since none of them ships. ### What this does not do @@ -4823,7 +4899,7 @@ The diversity statistic itself is [Nei 1973](#ref-nei1973diversity), the correct #### Nei 1972 { #ref-nei1972distance } **Nei, M.** (1972). Genetic Distance between Populations. *The American Naturalist* 106(949), 283–292. [10.1086/282771](https://doi.org/10.1086/282771) -: The distance the analysis layer places pools by, D_m = (J_X + J_Y)/2 - J_XY, where J is the probability that two chromosomes carry the same allele. It is the MINIMUM distance of this paper and not the standard distance D, which is defined in the same one and is a log of a ratio; the minimum distance is linear in the J terms, so averaging over loci and averaging over sites are the same operation and no ratio-of-averages question arises. What is applied here beyond the paper is the sampling correction: each J is replaced by its unbiased estimator from a sample of n_eff chromosomes, and J_XY takes none because the two pools are sequenced independently. It is HERE rather than in a module because analysis/lib/R/nei_distance.R is library code. +: The distance the analysis layer places pools by, D_m = (J_X + J_Y)/2 - J_XY, where J is the probability that two chromosomes carry the same allele. It is the MINIMUM distance of this paper and not the standard distance D, which is defined in the same one and is a log of a ratio; the minimum distance is linear in the J terms, so averaging over loci and averaging over sites are the same operation and no ratio-of-averages question arises. What is applied here beyond the paper is the sampling correction: each J is replaced by its unbiased estimator from a sample of n_eff chromosomes, and J_XY takes none because the two pools are sequenced independently. It is HERE rather than in a module because it is the nei_distance library, installed with whatever module declares it. #### Nei 1973 { #ref-nei1973diversity } @@ -4848,7 +4924,7 @@ The diversity statistic itself is [Nei 1973](#ref-nei1973diversity), the correct #### Hivert et al. 2018 { #ref-hivert2018poolseq } **Hivert, V., Leblois, R., Petit, E. J., Gautier, M. & Vitalis, R.** (2018). Measuring Genetic Differentiation from Pool-seq Data. *Genetics* 210(1), 315–330. [10.1534/genetics.118.300900](https://doi.org/10.1534/genetics.118.300900) -: The effective sample size the analysis layer weights by, n_eff = n*d / (n + d - 1) for a pool of n chromosomes read to depth d. The paper does not write it in that form - it defines D2 as the sum of (d + n - 1)/n, which is the sum of d / n_eff under this form and under no other. Its pools are parameterized by HAPLOID size, which is what leaves n_eff, and everything weighted by it, general over ploidy. It is HERE rather than in a module because analysis/lib/R/n_eff.R is library code and every module that weights anything calls it. +: The effective sample size the analysis layer weights by, n_eff = n*d / (n + d - 1) for a pool of n chromosomes read to depth d. The paper does not write it in that form - it defines D2 as the sum of (d + n - 1)/n, which is the sum of d / n_eff under this form and under no other. Its pools are parameterized by HAPLOID size, which is what leaves n_eff, and everything weighted by it, general over ploidy. It is HERE rather than in a module because it is the n_eff library, installed with whatever module declares it, and every module that weights anything calls it. ### Estimating from pooled reads diff --git a/analysis/modules/README.md b/modules/README.md similarity index 81% rename from analysis/modules/README.md rename to modules/README.md index bc4a3a7..18b1117 100644 --- a/analysis/modules/README.md +++ b/modules/README.md @@ -44,12 +44,12 @@ A module ships no configuration. `PoolSeqFlow analysis` assembles it — the ins ## Rules -Follow these and a module written by anyone runs beside the ones shipped here. Most of them exist because of a failure that has already happened once. +Follow these and a module written by anyone runs beside the ones published here. Most of them exist because of a failure that has already happened once. ### The shape 1. **`main.nf` is required, and the directory name must equal the manifest's `name`.** A directory holding a manifest and no `main.nf` is refused while the DAG is built — before the results folder is cleared — and it stops **every** invocation, not only yours. A directory holding no manifest at all is passed over in silence, so a half-finished install is harmless. -2. **Import the library by a literal relative path**, `'../../lib/nf/plan.nf'`. An interpolated include path is rejected both by `nextflow lint` and at runtime, and the literal resolves in a checkout and in an installation alike because a module sits at `analysis/modules//` in both. +2. **Import the library by a literal relative path**, `'../../lib/nf/plan.nf'`. An interpolated include path is rejected both by `nextflow lint` and at runtime, and the literal resolves because a module RUNS from the store, at `analysis/modules//`, beside the frame it imports. Its source lives at `modules//` here, where that path does not resolve — which is why `00_static` lints a module in an assembled store layout rather than where git keeps it. 3. **Import only the workflows you call.** An unused import still couples you: a library workflow that is renamed or removed breaks every module naming it, called or not. Import breadth is coupling breadth. 4. **Report your own version.** `manifest.json`'s `version` is the module's, and it moves on the module's timetable — never the pipeline release's. That separation is the reason modules are installed separately at all. The modules published with PoolSeqFlow use **`YYYYMMDD.NNN`** — the day the manifest changed, then a counter from `001` for that day — bumped whenever the manifest is touched and committed with the change. Any version string works as far as the frame is concerned, but `install` picks the newest with `sort -V`, so whatever you choose has to order correctly under it. @@ -76,16 +76,26 @@ A name no class answers to is refused while the DAG is built. Without that it wo 6b. **`license` is an SPDX identifier and it is required.** A result your module produces is produced under your terms, not the pipeline's — PoolSeqFlow is Apache-2.0 and imposes nothing on you, but a module that loads a GPL package into its R session is GPL itself. The frame prints it in the verification report so a user can see what an analysis was produced under before it runs, which is the whole reason it is not optional. -Every module shipped here is **GPL-3.0-or-later**, and the check that decides it is worth copying: each offers a compiled hot path through `Rcpp`, which is GPL, and each has it on **by default** — `nocpp` turns it off for one run, but the module ships expecting it and stops rather than falling back when `Rcpp` is absent. A dependency that only the default configuration reaches is still a dependency. Run the same test over what you load: `packageDescription("")$License` for each one, and take the strictest. +Every module published here is **GPL-3.0-or-later**, and the check that decides it is worth copying: each offers a compiled hot path through `Rcpp`, which is GPL, and each has it on **by default** — `nocpp` turns it off for one run, but the module ships expecting it and stops rather than falling back when `Rcpp` is absent. A dependency that only the default configuration reaches is still a dependency. Run the same test over what you load: `packageDescription("")$License` for each one, and take the strictest. 6c. **`frame` is the oldest `analysis/frame.version` your module runs on**, written the same way — `YYYYMMDD.NNN`. It is the axis `contract` does not cover: `contract` is about the shape of the published tables, while `frame` is about the library you import. You call `analysisPlan`, `designJson`, `moduleSettings` and `PublishResults` by name, and a frame that predates one of them is not a table problem. Set it to the frame you developed against and move it when you start calling something newer. 6d. **`environment` is the oldest PoolSeqFlow release whose analysis environment holds what you need**, written as a release is — `3.0.0`. There is one analysis environment per release and every module shares it, so its package set is a property of the release rather than something negotiated at install time. A module built against a later release's environment is refused here rather than installed and left to fail in R. -6e. **`packages` names what has to be added to that environment, pinned.** Each entry is `=` and nothing else: one `=`, an exact version, no build string, no range, no channel prefix. A build string names one platform's build, so a manifest carrying one cannot install anywhere else; the channel is the release's decision and not yours. Leave the field out when the release's own environment suffices — which is true of every module shipped here. +6e. **`packages` names what has to be added to that environment, pinned.** Each entry is `=` and nothing else: one `=`, an exact version, no build string, no range, no channel prefix. A build string names one platform's build, so a manifest carrying one cannot install anywhere else; the channel is the release's decision and not yours. **Name everything you load, including packages the release's own environment already carries.** The manifest states what YOUR module needs, not what one release happens to provide, and a release that slims its environment must not silently break you. Removing a module never takes a package the release itself is built on: the uninstall subtracts the baseline as well as what other installed modules declare. **You cannot pin a package the environment already holds at another version.** Installing your module refuses outright in that case, naming both versions, and installs nothing. That covers the release's own packages and every other module's alike: one version of one package serves everyone in the shared environment, so a disagreement is a release-time incompatibility rather than something to resolve on a user's machine. The same pin at the same version is fine and is the ordinary case for two modules built on one library. +### Declaring the libraries you use + +6f. **`libraries` names the shared arithmetic your module calls, and each one is installed with you.** A library is the same shape as a module — a folder with its own `manifest.json`, its `.R`, and its `.cpp` if it offers a compiled form — published and versioned on its own timetable and installed into `analysis/modules/lib//`. Declare it and it arrives when you are installed; it leaves when no installed module still declares it, so two modules sharing one never take it from each other. + +**You do not list the files.** The frame resolves them: `moduleLibraryFiles('')` returns the `.R` to source, in the order you declared the libraries, and `moduleCompiledFiles('')` returns the `.cpp` beside them. Both read your manifest, so the list exists once. A module that kept its own copy of that list had a second list to keep equal, and a disagreement between them was silent. + +**A library declares what it needs too** — its own `packages`, and `libraries` of its own if it calls another. It declares `contract` only if it reads a published table: `allele_frequencies` and `site_diversity` parse depth-table cells and are bound to `freq-1`, while `n_eff` and `chunk_ranges` take numbers and know nothing about the tables. + +Write a library when more than one module wants the same decision-free arithmetic. A derivation that carries a DECISION does not belong in one: it takes the decision as a required argument with no default, so the module passes the value, declares it in its `gates`, and the report prints it. + ### Reading a published result 7. **Never write into the results tree.** Copy what you need out of it; a module that moves a published artifact damages the run that produced it. The pipeline's results are inputs and nothing else. @@ -104,7 +114,7 @@ Every module shipped here is **GPL-3.0-or-later**, and the check that decides it 14. **What you hand it must be real files.** `PublishResults` dereferences what Nextflow staged for exactly this reason: `cleanup = true` is set for analysis runs, so the work directory goes on success and a symlinked result becomes a dangling link. Anything you write anywhere else is yours to get right the same way. 15. **Emit the script that produced each result** into the folder beside it, in the same set of files you hand `PublishResults`. A result nobody can regenerate is the reproducibility gap this whole layer exists to close, so this one is **enforced, not asked**: publishing an analysis that carries no `*.R`, `*.r`, `*.Rmd`, `*.rmd`, `*.sh`, `*.py` or `*.jl` fails the run, names what you did produce, and publishes nothing. The folder is left exactly as the verification cleared it, so the retry is not refused. -15b. **Carry your own `citations.json`. It is required, like `manifest.json` and `main.nf`.** The modules shipped here do not write it by hand: they carry a `references.bib` and compile it with `dev/scripts/bib2citations.py`, so an entry can be pasted from a publisher and the fields that are not bibliographic — `r_package`, `note`, `id` — ride along as extra BibTeX fields. The frame reads only the JSON, so a module built any other way is fine; what it must not be is edited in both places. Every published analysis gets `CITATIONS.md` and `references.bib` written into its folder, in the same rename as the result. The frame contributes PoolSeqFlow, Nextflow and R; **the method your module implements and the R packages it uses are yours to declare**, because you are published separately and the frame cannot hold a list of citations for modules that do not exist yet. At the very least, name the libraries you call — a user cannot credit a package they never learn they depended on. The shape is `analysis/citations.json`'s, one entry per reference, keyed however you like; an entry naming `"r_package": "vegan"` has its version asked of R at run time rather than taken from what the environment pins, so it stays correct if someone repointed it. A directory without the file is refused at DAG-build time, before the verification clears a results folder, exactly as a missing `main.nf` is, and `modules install` refuses an archive that lacks one. +15b. **Carry your own `citations.json`. It is required, like `manifest.json` and `main.nf`.** The modules published here do not write it by hand: they carry a `references.bib` and compile it with `dev/scripts/bib2citations.py`, so an entry can be pasted from a publisher and the fields that are not bibliographic — `r_package`, `note`, `id` — ride along as extra BibTeX fields. The frame reads only the JSON, so a module built any other way is fine; what it must not be is edited in both places. Every published analysis gets `CITATIONS.md` and `references.bib` written into its folder, in the same rename as the result. The frame contributes PoolSeqFlow, Nextflow and R; **the method your module implements and the R packages it uses are yours to declare**, because you are published separately and the frame cannot hold a list of citations for modules that do not exist yet. At the very least, name the libraries you call — a user cannot credit a package they never learn they depended on. The shape is `analysis/citations.json`'s, one entry per reference, keyed however you like; an entry naming `"r_package": "vegan"` has its version asked of R at run time rather than taken from what the environment pins, so it stays correct if someone repointed it. A directory without the file is refused at DAG-build time, before the verification clears a results folder, exactly as a missing `main.nf` is, and `modules install` refuses an archive that lacks one. 15c. **Declare an `outputs` entry for every file you publish, and say where it is explained.** Each is `{"file": "design.tsv", "summary": "one row per pool", "anchor": "the-experimental-design"}` — `file` may be a glob, `summary` is the one-line description the reader gets, and `anchor` names a heading of the manual this release ships. A module published separately has nowhere in that manual to point and gives `"url"` instead, in full. The frame renders one `README.md` from every module's declarations plus what every analysis carries anyway, so a folder found months later says how to read itself and no module invents its own way of saying it. Two things are checked for you: an `anchor` no heading answers to is refused before your module starts, and a `file` you declared and did not publish fails the publish with nothing written. **This is where a number that is not a measurement gets said to be one** — a bound, an estimate with an assumption behind it, a figure whose meaning changes with a setting. A table cell cannot carry that sentence and a `summary` plus a link can. @@ -135,7 +145,7 @@ Every module shipped here is **GPL-3.0-or-later**, and the check that decides it ## Where they come from -A fresh installation holds the modules this release ships and nothing else. Everything else is installed separately from the pipeline, into this release's own installation, so a module installed for one release is never picked up by another. +**A fresh installation holds no modules at all.** Every module is installed separately, into this release's own installation, so a module installed for one release is never picked up by another. Only `verify` is always there, and it belongs to the frame rather than the store. ```bash PoolSeqFlow analysis modules available # what is published @@ -180,7 +190,7 @@ dev/scripts/publish-module.sh [ref] It builds the tarball into `modules-repo/`, reads `contract`, `frame`, `environment` and `summary` out of the module's own manifest at that ref, appends the row, and bumps `#!index-version`. It refuses to overwrite a version that is already published — somebody may have installed it, and its checksum is in the catalogue — so a change means bumping the module's version and publishing that. -**The tarball is built from the extracted tree rather than piped straight out of `git archive`, and that is not fussiness.** `git archive :analysis/modules/` reads a subtree, and `.gitattributes` patterns are anchored at the repository root — so `analysis/modules/*/test/ export-ignore` does not match `test/` inside that subtree, and the module's own cases would ship where a release tarball excludes them. Archiving a tree also stamps `mtime` as *now*, so two builds of one ref would not match. The script drops `test/` and repacks with the commit's timestamp, which makes republishing the same ref produce the same bytes. +**The tarball is built from the extracted tree rather than piped straight out of `git archive`, and that is not fussiness.** `git archive :modules/` reads a subtree, and `.gitattributes` patterns are anchored at the repository root — so a rule written for `modules/` does not match `test/` inside that subtree, and the module's own cases would ship where a release tarball excludes them. Archiving a tree also stamps `mtime` as *now*, so two builds of one ref would not match. The script drops `test/` and repacks with the commit's timestamp, which makes republishing the same ref produce the same bytes. The tarballs are committed and served from the site alongside the catalogue, so **a published row and the file it names go out in one commit** — otherwise the row advertises a download that 404s until the next deploy. diff --git a/analysis/modules/association/association.R b/modules/association/association.R similarity index 99% rename from analysis/modules/association/association.R rename to modules/association/association.R index fdf6802..9eb2ead 100644 --- a/analysis/modules/association/association.R +++ b/modules/association/association.R @@ -1,4 +1,4 @@ -# The module's own analysis. The shared library is above this line in the published copy. +# The module's own analysis. The libraries it declares are above this line in the published copy. # # association.R --design design.json --pools pools.json --options options.json # --cpp allele_frequencies.cpp --depths a.tsv,b.tsv --out published diff --git a/analysis/modules/association/citations.json b/modules/association/citations.json similarity index 100% rename from analysis/modules/association/citations.json rename to modules/association/citations.json diff --git a/analysis/modules/association/main.nf b/modules/association/main.nf similarity index 88% rename from analysis/modules/association/main.nf rename to modules/association/main.nf index adc5f8f..fc10310 100644 --- a/analysis/modules/association/main.nf +++ b/modules/association/main.nf @@ -9,16 +9,11 @@ nextflow.enable.dsl=2 include { analysisPlan } from '../../lib/nf/plan.nf' +include { moduleLibraryFiles; moduleCompiledFiles } from '../../lib/nf/modules.nf' include { installDir; frameVersion; moduleSettings } from '../../lib/nf/paths.nf' include { designJson } from '../../lib/nf/design.nf' include { PublishResults } from '../../lib/nf/results.nf' -// The shared library files this module CALLS, in the order they are concatenated. Exactly this -// list is folded into the script published beside the result, so a function the module does -// not call does not travel with a result it did not compute. -def libraryFiles() { - return ['n_eff.R', 'allele_frequencies.R', 'chunk_ranges.R'] -} // This module's settings, with the value each takes when the project does not set it. // moduleSettings() refuses a key that is not here and names the ones that are. @@ -80,8 +75,8 @@ process Analyze { pools = groovy.json.JsonOutput.toJson(target.pools).replace("'", "'\\''") // Rendered here rather than read from the environment inside R: installDir() validates and // refuses with a message, where an unset variable at task time is a file-not-found. - library = libraryFiles().collect { name -> "${installDir()}/analysis/lib/R/${name}" } - compiled = "${installDir()}/analysis/lib/cpp/allele_frequencies.cpp" + library = moduleLibraryFiles('association') + compiled = moduleCompiledFiles('association') // 0 means the cores Nextflow gave this task. Anything else oversubscribes them. workers = settings.workers > 0 ? settings.workers : task.cpus options = groovy.json.JsonOutput.toJson([ phenotypes : settings.phenotypes, @@ -95,7 +90,7 @@ process Analyze { workers : workers, usecpp : useCompiled(settings) ]) .replace("'", "'\\''") - // The published script's header: the frame version that defined the library, which + // The published script's header: the frame version this ran against, which // implementation of the parse ran, and the settings that shaped the work. The permutation // budget is here because it sets the smallest p a run can report. header = ["# association, PoolSeqFlow analysis frame ${frameVersion()}", @@ -110,15 +105,15 @@ process Analyze { printf '%s' '${pools}' > pools.json printf '%s' '${options}' > options.json - # The script published beside the result, and the one that runs: the shared library first, - # then this module's own. + # The script published beside the result, and the one that runs: the declared libraries + # first, then this module's own. { printf '%s\\n' '${header}' - echo '# The shared library follows, then this module.' + echo '# The libraries this module declares follow, then the module itself.' cat ${library.join(' ')} cat ${moduleDir}/association.R } > published/association.R - cp ${compiled} published/allele_frequencies.cpp + cp ${compiled.join(' ')} published/ Rscript --vanilla published/association.R --design design.json --pools pools.json \\ --options options.json --cpp published/allele_frequencies.cpp \\ diff --git a/analysis/modules/association/manifest.json b/modules/association/manifest.json similarity index 95% rename from analysis/modules/association/manifest.json rename to modules/association/manifest.json index c381064..2386a44 100644 --- a/analysis/modules/association/manifest.json +++ b/modules/association/manifest.json @@ -1,14 +1,19 @@ { "name": "association", - "version": "20260910.001", + "version": "20260910.004", "contract": "freq-1", "license": "GPL-3.0-or-later", - "frame": "20260908.003", + "frame": "20260910.001", "environment": "3.0.0", "summary": "each allele's frequency regressed on a phenotype measured per pool, with a permutation p", "needs": [ "depths" ], + "libraries": [ + "n_eff", + "allele_frequencies", + "chunk_ranges" + ], "gates": [ "THIS IS A MODEL-BASED TEST AND NOT AN ASSUMPTION-LIGHT ONE. It asserts that the variance of a pool's frequency is p(1-p) times (dispersion + 1/n_eff), that units are independent, and that a site's alleles are several views of one comparison. Where those do not describe your experiment the numbers will still be produced and will still be wrong; the diagnostics in permutations.tsv are what say so.", "The fit is on UNITS, not pools. Each unit's pools are collapsed onto one value first, weighted by their effective sizes, and the degrees of freedom are the units carrying data at that site minus two. Where a unit holds one pool the collapse changes nothing.", @@ -65,5 +70,13 @@ "summary": "the compiled form of the per-site parse, shipped whether or not this run used it; the header of association.R says which path produced the numbers", "anchor": "association-compiled" } + ], + "packages": [ + "r-dofuture=1.3.0", + "r-foreach=1.5.2", + "r-future=1.75.0", + "r-ggplot2=4.0.3", + "r-jsonlite=2.0.0", + "r-rcpp=1.1.2" ] } diff --git a/analysis/modules/association/references.bib b/modules/association/references.bib similarity index 100% rename from analysis/modules/association/references.bib rename to modules/association/references.bib diff --git a/analysis/modules/association/test/association.sh b/modules/association/test/association.sh similarity index 74% rename from analysis/modules/association/test/association.sh rename to modules/association/test/association.sh index 072f3bb..d79a529 100644 --- a/analysis/modules/association/test/association.sh +++ b/modules/association/test/association.sh @@ -1,9 +1,9 @@ #!/bin/bash # association, against the analytic corpus its own tools build. # cost: jvm -# covers: analysis/modules/association/ analysis/lib/R/ analysis/lib/cpp/ +# covers: modules/association/ modules/lib/ # covers: test/tools/freq_corpus.py -# covers: analysis.nf analysis/modules/association/main.nf +# covers: analysis.nf modules/association/main.nf # # The fixtures and helpers every analysis suite shares are in test/lib/analysis.sh. # @@ -25,20 +25,21 @@ association_corpus() { # Run the module's R directly over the corpus, under one set of options, into $1. # -# Every .R in the shared library rather than the list main.nf names: they are standalone function +# Every library's .R rather than the list the manifest names: they are standalone function # definitions, so a superset is harmless, and the case then cannot go stale when that list -# changes. The Nextflow case is what proves main.nf assembles the same thing. +# changes. The Nextflow case is what proves main.nf assembles the same thing, and 00_static is +# what proves the manifest declares exactly what the module calls. association_direct() { local dest="$1" options="$2" design="${3:-}" mkdir -p "$dest" [ -n "$design" ] || design="$CORPUS_DIR/design.json" - cat "$REPO_ROOT"/analysis/lib/R/*.R \ - "$REPO_ROOT/analysis/modules/association/association.R" > "$dest/association.R" + cat "$REPO_ROOT"/modules/lib/*/*.R \ + "$REPO_ROOT/modules/association/association.R" > "$dest/association.R" printf '%s' "$options" > "$dest/options.json" ( cd "$CORPUS_DIR/Frequencies" && Rscript --vanilla "$dest/association.R" \ --design "$design" --pools "$CORPUS_DIR/pools.json" \ --options "$dest/options.json" \ - --cpp "$REPO_ROOT/analysis/lib/cpp/allele_frequencies.cpp" \ + --cpp "$REPO_ROOT/modules/lib/allele_frequencies/allele_frequencies.cpp" \ --depths 'Test_snp_depth.tsv' --out "$dest" ) > "$dest/out.txt" 2>&1 } @@ -228,7 +229,7 @@ test_both_paths_through_the_parse_agree() { # The module credits the statistics it computes and not the family they belong to. A reader who # follows an entry and finds nothing of it in the output is worse served than by no entry. test_association_cites_the_statistics_it_computes() { - local citations="$REPO_ROOT/analysis/modules/association/citations.json" + local citations="$REPO_ROOT/modules/association/citations.json" assert_file "$citations" "association ships a citations.json" local id for id in phipson2010 benjamini1995 long2026; do @@ -241,3 +242,68 @@ test_association_cites_the_statistics_it_computes() { grep -q '"hivert2018"' "$citations" \ && fail_case "association must not redefine hivert2018: a BibTeX key is defined once" } + +# ONE CASE THROUGH NEXTFLOW, and it is what every other case here cannot do. The rest call the +# module's R directly, which proves the arithmetic and says nothing about main.nf - so a fault +# in how the PROCESS assembles its command survives every one of them. +# +# It survived exactly that way. `cp ${compiled} published/allele_frequencies.cpp` interpolated +# the Groovy LIST moduleCompiledFiles() returns, so the process ran `cp [/path/to/x.cpp]` and +# died on the brackets. basicstats has this case and failed on it in a full run; association did +# not have one and passed the same run with the identical line. +# +# The fixture needs a PHENOTYPE, which the shared baseline has no column for: every other +# analysis suite runs on exp_ variables alone. pt_wingspan is added here over the six pools the +# planted results were produced from. +# +# ONE VALUE PER UNIT, and the baseline's units are the three populations followed through two +# timepoints - so both rows of a population carry the same wingspan. association fits on units +# and a unit takes one value; two values on one unit is repeated measures, which it refuses by +# design and says so. Giving each row its own value is the obvious thing to write here and the +# module is right to reject it. +test_association_runs_through_the_frame() { + analysis_ready single || return + if ! have_r; then skip_case "no Rscript"; return; fi + analysis_write_metadata "$ANALYSIS_SB" 'SampleID,RG_Sample,RG_Library,RG_Platform,RG_PlatformUnit,exp_population,exp_time,pt_wingspan +TestSample1,TestSample1,Lib1,ILLUMINA,Unit1,Pop1,T1,10.5 +TestSample2,TestSample2,Lib1,ILLUMINA,Unit1,Pop1,T2,10.5 +TestSample3,TestSample3,Lib1,ILLUMINA,Unit1,Pop2,T1,13.8 +TestSample4,TestSample4,Lib1,ILLUMINA,Unit1,Pop2,T2,13.8 +TestSample5,TestSample5,Lib1,ILLUMINA,Unit1,Pop3,T1,16.4 +TestSample6,TestSample6,Lib1,ILLUMINA,Unit1,Pop3,T2,16.4' + # A pt_ column is RECORDED by default and becomes a phenotype only once declared with a + # measurement scale - which is the layer working as designed, and is what a module author + # writing this case for the first time will trip over. + # $ANALYSIS_TIME_BLOCK is carried along because this REPLACES main/analysis.config rather + # than adding to it, and the baseline put the timeVar declaration there - the fixture has an + # exp_time column, and the layer refuses one it has not been told how to read. + analysis_write_metadata_config "$ANALYSIS_SB" \ + "$ANALYSIS_TIME_BLOCK + phenotypes { pt_wingspan { kind = 'quantitative' } }" + analysis_plant_results "$ANALYSIS_SB/store/Output" + cat > "$ANALYSIS_SB/main/association.config" <<'CFG' +params { + analysis { + modules { + association { + phenotypes = ['pt_wingspan'] + } + } + } +} +CFG + + local status; status=$(analysis_run_module association) + assert_status 0 "$status" "association should run; see $ANALYSIS_SB/run.out" + + local dir="$ANALYSIS_SB/main/Analysis/Results/association" + assert_file "$dir/association.tsv" "the site table" + assert_file "$dir/permutations.tsv" "the diagnostics that say what it assumed" + assert_file "$dir/phenotype.tsv" "the phenotype as it was fitted" + assert_file "$dir/association.R" "the script that produced them" + # The one the bug above destroyed: a compiled source is published whether or not the run used + # it, so its absence is a broken process rather than a choice about the hot path. + assert_file "$dir/allele_frequencies.cpp" "the compiled parse, published either way" + assert_contains "$(cat "$dir/association.R")" "allele_frequencies <- function" \ + "the libraries it declares must be folded into the published script" +} diff --git a/analysis/modules/basicstats/basicstats.R b/modules/basicstats/basicstats.R similarity index 99% rename from analysis/modules/basicstats/basicstats.R rename to modules/basicstats/basicstats.R index 40948cf..e221d19 100644 --- a/analysis/modules/basicstats/basicstats.R +++ b/modules/basicstats/basicstats.R @@ -1,4 +1,4 @@ -# The module's own analysis. The shared library is above this line in the published copy. +# The module's own analysis. The libraries it declares are above this line in the published copy. # # basicstats.R --design design.json --pools pools.json --options options.json # --cpp site_diversity.cpp --depths a.tsv,b.tsv --out published diff --git a/analysis/modules/basicstats/citations.json b/modules/basicstats/citations.json similarity index 100% rename from analysis/modules/basicstats/citations.json rename to modules/basicstats/citations.json diff --git a/analysis/modules/basicstats/main.nf b/modules/basicstats/main.nf similarity index 86% rename from analysis/modules/basicstats/main.nf rename to modules/basicstats/main.nf index ccdaa19..db6f9c1 100644 --- a/analysis/modules/basicstats/main.nf +++ b/modules/basicstats/main.nf @@ -9,17 +9,11 @@ nextflow.enable.dsl=2 include { analysisPlan } from '../../lib/nf/plan.nf' +include { moduleLibraryFiles; moduleCompiledFiles } from '../../lib/nf/modules.nf' include { installDir; frameVersion; moduleSettings } from '../../lib/nf/paths.nf' include { designJson } from '../../lib/nf/design.nf' include { PublishResults } from '../../lib/nf/results.nf' -// The shared library files this module CALLS, in the order they are concatenated. Exactly this -// list is folded into the script published beside the result, so a function the module does -// not call does not travel with a result it did not compute. -def libraryFiles() { - return ['harmonic_mean.R', 'n_eff.R', 'pool_n_eff.R', - 'site_diversity.R', 'chunk_ranges.R'] -} // This module's settings, with the value each takes when the project does not set it. // moduleSettings() refuses a key that is not here and names the ones that are. @@ -82,8 +76,8 @@ process Analyze { pools = groovy.json.JsonOutput.toJson(target.pools).replace("'", "'\\''") // Rendered here rather than read from the environment inside R: installDir() validates and // refuses with a message, where an unset variable at task time is a file-not-found. - library = libraryFiles().collect { name -> "${installDir()}/analysis/lib/R/${name}" } - compiled = "${installDir()}/analysis/lib/cpp/site_diversity.cpp" + library = moduleLibraryFiles('basicstats') + compiled = moduleCompiledFiles('basicstats') // 0 means the cores Nextflow gave this task. Anything else oversubscribes them. workers = settings.workers > 0 ? settings.workers : task.cpus options = groovy.json.JsonOutput.toJson([ minReads : settings.minReads, @@ -92,7 +86,7 @@ process Analyze { usecpp : useCompiled(settings), chromosomes: settings.chromosomes ]) .replace("'", "'\\''") - // The published script's header: the frame version that defined the library, which + // The published script's header: the frame version this ran against, which // implementation of the per-site loop ran, and the settings that shaped the work. header = ["# basicstats, PoolSeqFlow analysis frame ${frameVersion()}", "# ${useCompiled(settings) ? 'site_diversity.cpp, compiled at run time' : 'site_diversity(), vectorized R'}" + @@ -105,15 +99,15 @@ process Analyze { printf '%s' '${pools}' > pools.json printf '%s' '${options}' > options.json - # The script published beside the result, and the one that runs: the shared library first, - # then this module's own. + # The script published beside the result, and the one that runs: the declared libraries + # first, then this module's own. { printf '%s\\n' '${header}' - echo '# The shared library follows, then this module.' + echo '# The libraries this module declares follow, then the module itself.' cat ${library.join(' ')} cat ${moduleDir}/basicstats.R } > published/basicstats.R - cp ${compiled} published/site_diversity.cpp + cp ${compiled.join(' ')} published/ Rscript --vanilla published/basicstats.R --design design.json --pools pools.json \\ --options options.json --cpp published/site_diversity.cpp \\ diff --git a/analysis/modules/basicstats/manifest.json b/modules/basicstats/manifest.json similarity index 91% rename from analysis/modules/basicstats/manifest.json rename to modules/basicstats/manifest.json index fe866f2..a31616e 100644 --- a/analysis/modules/basicstats/manifest.json +++ b/modules/basicstats/manifest.json @@ -1,15 +1,20 @@ { "name": "basicstats", - "version": "20260910.001", + "version": "20260910.004", "contract": "freq-1", "license": "GPL-3.0-or-later", - "frame": "20260908.003", + "frame": "20260910.001", "environment": "3.0.0", "summary": "site counts, depth, effective pool size and gene diversity, per pool", "needs": [ "frequencies", "depths" ], + "libraries": [ + "n_eff", + "site_diversity", + "chunk_ranges" + ], "gates": [ "Depth is the sum of a pool's comma-separated cell in the depth table: reads supporting any allele at that site, after step 7's depth, quality and false-positive filters.", "Every statistic is computed over CALLED SITES only. Sites the pipeline did not call are absent from the tables and are not counted as invariant.", @@ -63,5 +68,14 @@ "summary": "the compiled form of the per-site hot path, shipped whether or not this run used it; the header of basicstats.R says which path produced the numbers", "anchor": "basicstats-compiled" } + ], + "packages": [ + "r-data.table=1.18.6.1", + "r-dofuture=1.3.0", + "r-foreach=1.5.2", + "r-future=1.75.0", + "r-ggplot2=4.0.3", + "r-jsonlite=2.0.0", + "r-rcpp=1.1.2" ] } diff --git a/analysis/modules/basicstats/references.bib b/modules/basicstats/references.bib similarity index 100% rename from analysis/modules/basicstats/references.bib rename to modules/basicstats/references.bib diff --git a/analysis/modules/basicstats/test/basicstats.sh b/modules/basicstats/test/basicstats.sh similarity index 97% rename from analysis/modules/basicstats/test/basicstats.sh rename to modules/basicstats/test/basicstats.sh index 59dbe45..69ad850 100644 --- a/analysis/modules/basicstats/test/basicstats.sh +++ b/modules/basicstats/test/basicstats.sh @@ -1,9 +1,9 @@ #!/bin/bash # basicstats, against the analytic corpus its own tools build. # cost: jvm -# covers: analysis/modules/basicstats/ analysis/lib/R/ analysis/lib/cpp/ +# covers: modules/basicstats/ modules/lib/ # covers: test/tools/freq_corpus.py -# covers: analysis.nf analysis/modules/basicstats/main.nf +# covers: analysis.nf modules/basicstats/main.nf # # The fixtures and helpers every analysis suite shares are in test/lib/analysis.sh. # @@ -12,13 +12,14 @@ # Run the module's R directly over the corpus, under one set of options, into $1. # -# Every .R in the shared library rather than the list main.nf names: they are standalone -# function definitions, so a superset is harmless, and the case then cannot go stale when that -# list changes. The Nextflow case above is what proves main.nf assembles the same thing. +# Every library's .R rather than the list the manifest names: they are standalone function +# definitions, so a superset is harmless, and the case then cannot go stale when that list +# changes. The Nextflow case above is what proves main.nf assembles the same thing, and +# 00_static is what proves the manifest declares exactly what the module calls. basicstats_direct() { local dest="$1" options="$2" corpus="$3" rscript="${4:-Rscript}" bin="" mkdir -p "$dest" - cat "$REPO_ROOT"/analysis/lib/R/*.R "$REPO_ROOT/analysis/modules/basicstats/basicstats.R" \ + cat "$REPO_ROOT"/modules/lib/*/*.R "$REPO_ROOT/modules/basicstats/basicstats.R" \ > "$dest/basicstats.R" printf '%s' "$options" > "$dest/options.json" # An environment's R drives an environment's compiler - conda's is @@ -29,7 +30,7 @@ basicstats_direct() { ( cd "$corpus/Frequencies" && PATH="${bin}$PATH" "$rscript" --vanilla "$dest/basicstats.R" \ --design "$corpus/design.json" --pools "$corpus/pools.json" \ --options "$dest/options.json" \ - --cpp "$REPO_ROOT/analysis/lib/cpp/site_diversity.cpp" \ + --cpp "$REPO_ROOT/modules/lib/site_diversity/site_diversity.cpp" \ --depths 'Test_indel_depth.tsv,Test_snp_depth.tsv' \ --histograms "$(find "$corpus/Reports/Depth" -name '*_depth_histogram.tsv' | sort | paste -sd,)" \ --out "$dest" ) > "$dest/out.txt" 2>&1 diff --git a/analysis/lib/R/README.md b/modules/lib/README.md similarity index 100% rename from analysis/lib/R/README.md rename to modules/lib/README.md diff --git a/analysis/lib/R/allele_frequencies.R b/modules/lib/allele_frequencies/allele_frequencies.R similarity index 100% rename from analysis/lib/R/allele_frequencies.R rename to modules/lib/allele_frequencies/allele_frequencies.R diff --git a/analysis/lib/cpp/allele_frequencies.cpp b/modules/lib/allele_frequencies/allele_frequencies.cpp similarity index 98% rename from analysis/lib/cpp/allele_frequencies.cpp rename to modules/lib/allele_frequencies/allele_frequencies.cpp index 8cf5ab7..885c9ce 100644 --- a/analysis/lib/cpp/allele_frequencies.cpp +++ b/modules/lib/allele_frequencies/allele_frequencies.cpp @@ -1,4 +1,4 @@ -// The compiled form of analysis/lib/R/allele_frequencies.R, for projects large enough to want it. +// The compiled form of allele_frequencies.R beside it, for projects large enough to want it. // // Same seam: a list of depth-table columns in, a list of site, alleles, depth and freq out. One // pass over the strings, parsing digits as it goes, so the split lists the vectorized R spends diff --git a/modules/lib/allele_frequencies/manifest.json b/modules/lib/allele_frequencies/manifest.json new file mode 100644 index 0000000..1b6aba0 --- /dev/null +++ b/modules/lib/allele_frequencies/manifest.json @@ -0,0 +1,14 @@ +{ + "name": "allele_frequencies", + "version": "20260910.002", + "kind": "library", + "contract": "freq-1", + "license": "GPL-3.0-or-later", + "frame": "20260910.001", + "environment": "3.0.0", + "summary": "per-site allele frequencies parsed from the depth table, vectorized R with a compiled path", + "libraries": [], + "packages": [ + "r-rcpp=1.1.2" + ] +} diff --git a/analysis/lib/R/chunk_ranges.R b/modules/lib/chunk_ranges/chunk_ranges.R similarity index 100% rename from analysis/lib/R/chunk_ranges.R rename to modules/lib/chunk_ranges/chunk_ranges.R diff --git a/modules/lib/chunk_ranges/manifest.json b/modules/lib/chunk_ranges/manifest.json new file mode 100644 index 0000000..bfb8076 --- /dev/null +++ b/modules/lib/chunk_ranges/manifest.json @@ -0,0 +1,11 @@ +{ + "name": "chunk_ranges", + "version": "20260910.002", + "kind": "library", + "license": "GPL-3.0-or-later", + "frame": "20260910.001", + "environment": "3.0.0", + "summary": "a row count split into contiguous bins for chunked or parallel reading", + "libraries": [], + "packages": [] +} diff --git a/analysis/lib/R/harmonic_mean.R b/modules/lib/n_eff/harmonic_mean.R similarity index 100% rename from analysis/lib/R/harmonic_mean.R rename to modules/lib/n_eff/harmonic_mean.R diff --git a/modules/lib/n_eff/manifest.json b/modules/lib/n_eff/manifest.json new file mode 100644 index 0000000..2d8e2e4 --- /dev/null +++ b/modules/lib/n_eff/manifest.json @@ -0,0 +1,11 @@ +{ + "name": "n_eff", + "version": "20260910.002", + "kind": "library", + "license": "GPL-3.0-or-later", + "frame": "20260910.001", + "environment": "3.0.0", + "summary": "effective pool size at a site and over a pool, and the harmonic mean it needs", + "libraries": [], + "packages": [] +} diff --git a/analysis/lib/R/n_eff.R b/modules/lib/n_eff/n_eff.R similarity index 100% rename from analysis/lib/R/n_eff.R rename to modules/lib/n_eff/n_eff.R diff --git a/analysis/lib/R/pool_n_eff.R b/modules/lib/n_eff/pool_n_eff.R similarity index 100% rename from analysis/lib/R/pool_n_eff.R rename to modules/lib/n_eff/pool_n_eff.R diff --git a/analysis/lib/R/add_distance.R b/modules/lib/nei_distance/add_distance.R similarity index 100% rename from analysis/lib/R/add_distance.R rename to modules/lib/nei_distance/add_distance.R diff --git a/modules/lib/nei_distance/manifest.json b/modules/lib/nei_distance/manifest.json new file mode 100644 index 0000000..c4c0de6 --- /dev/null +++ b/modules/lib/nei_distance/manifest.json @@ -0,0 +1,14 @@ +{ + "name": "nei_distance", + "version": "20260910.002", + "kind": "library", + "contract": "freq-1", + "license": "GPL-3.0-or-later", + "frame": "20260910.001", + "environment": "3.0.0", + "summary": "Nei's minimum distance accumulated over sites, combined and averaged, with a compiled path", + "libraries": [], + "packages": [ + "r-rcpp=1.1.2" + ] +} diff --git a/analysis/lib/R/mean_distance.R b/modules/lib/nei_distance/mean_distance.R similarity index 100% rename from analysis/lib/R/mean_distance.R rename to modules/lib/nei_distance/mean_distance.R diff --git a/analysis/lib/R/nei_distance.R b/modules/lib/nei_distance/nei_distance.R similarity index 100% rename from analysis/lib/R/nei_distance.R rename to modules/lib/nei_distance/nei_distance.R diff --git a/analysis/lib/cpp/nei_distance.cpp b/modules/lib/nei_distance/nei_distance.cpp similarity index 98% rename from analysis/lib/cpp/nei_distance.cpp rename to modules/lib/nei_distance/nei_distance.cpp index 1e130c1..82a572d 100644 --- a/analysis/lib/cpp/nei_distance.cpp +++ b/modules/lib/nei_distance/nei_distance.cpp @@ -1,4 +1,4 @@ -// The compiled form of analysis/lib/R/nei_distance.R, for projects large enough to want it. +// The compiled form of nei_distance.R beside it, for projects large enough to want it. // // Same seam: frequencies by allele, the site index that groups them, and each site's effective // sample sizes in; the raw and corrected pool-by-pool sums and the site counts out. One pass diff --git a/modules/lib/site_diversity/manifest.json b/modules/lib/site_diversity/manifest.json new file mode 100644 index 0000000..e147bee --- /dev/null +++ b/modules/lib/site_diversity/manifest.json @@ -0,0 +1,14 @@ +{ + "name": "site_diversity", + "version": "20260910.002", + "kind": "library", + "contract": "freq-1", + "license": "GPL-3.0-or-later", + "frame": "20260910.001", + "environment": "3.0.0", + "summary": "gene diversity per site, 1 - sum(p^2), vectorized R with a compiled path", + "libraries": [], + "packages": [ + "r-rcpp=1.1.2" + ] +} diff --git a/analysis/lib/R/site_diversity.R b/modules/lib/site_diversity/site_diversity.R similarity index 100% rename from analysis/lib/R/site_diversity.R rename to modules/lib/site_diversity/site_diversity.R diff --git a/analysis/lib/cpp/site_diversity.cpp b/modules/lib/site_diversity/site_diversity.cpp similarity index 96% rename from analysis/lib/cpp/site_diversity.cpp rename to modules/lib/site_diversity/site_diversity.cpp index b64ac3d..132b2d7 100644 --- a/analysis/lib/cpp/site_diversity.cpp +++ b/modules/lib/site_diversity/site_diversity.cpp @@ -1,4 +1,4 @@ -// The compiled form of analysis/lib/R/site_diversity.R, for projects large enough to want it. +// The compiled form of site_diversity.R beside it, for projects large enough to want it. // // Same seam: a character vector of depth-table cells in, a list of two numeric vectors out. One // pass over the strings, parsing digits as it goes, so nothing between the input and the two diff --git a/analysis/modules/mds/citations.json b/modules/mds/citations.json similarity index 100% rename from analysis/modules/mds/citations.json rename to modules/mds/citations.json diff --git a/analysis/modules/mds/main.nf b/modules/mds/main.nf similarity index 84% rename from analysis/modules/mds/main.nf rename to modules/mds/main.nf index 963a3db..709bddf 100644 --- a/analysis/modules/mds/main.nf +++ b/modules/mds/main.nf @@ -9,23 +9,12 @@ nextflow.enable.dsl=2 include { analysisPlan } from '../../lib/nf/plan.nf' +include { moduleLibraryFiles; moduleCompiledFiles } from '../../lib/nf/modules.nf' include { installDir; frameVersion; moduleSettings } from '../../lib/nf/paths.nf' include { designJson } from '../../lib/nf/design.nf' include { PublishResults } from '../../lib/nf/results.nf' -// The shared library files this module CALLS, in the order they are concatenated. Exactly this -// list is folded into the script published beside the result, so a function the module does -// not call does not travel with a result it did not compute. -def libraryFiles() { - return ['n_eff.R', 'allele_frequencies.R', 'nei_distance.R', 'add_distance.R', - 'mean_distance.R', 'chunk_ranges.R'] -} -// The compiled forms this module offers, published beside the result whether or not the run -// used them. -def compiledFiles() { - return ['allele_frequencies.cpp', 'nei_distance.cpp'] -} // This module's settings, with the value each takes when the project does not set it. // moduleSettings() refuses a key that is not here and names the ones that are. @@ -85,8 +74,8 @@ process Analyze { pools = groovy.json.JsonOutput.toJson(target.pools).replace("'", "'\\''") // Rendered here rather than read from the environment inside R: installDir() validates and // refuses with a message, where an unset variable at task time is a file-not-found. - library = libraryFiles().collect { name -> "${installDir()}/analysis/lib/R/${name}" } - compiled = compiledFiles().collect { name -> "${installDir()}/analysis/lib/cpp/${name}" } + library = moduleLibraryFiles('mds') + compiled = moduleCompiledFiles('mds') // 0 means the cores Nextflow gave this task. Anything else oversubscribes them. workers = settings.workers > 0 ? settings.workers : task.cpus options = groovy.json.JsonOutput.toJson([ dimensions : settings.dimensions, @@ -98,7 +87,7 @@ process Analyze { workers : workers, usecpp : useCompiled(settings) ]) .replace("'", "'\\''") - // The published script's header: the frame version that defined the library, which + // The published script's header: the frame version this ran against, which // implementation ran, and the settings that shaped the work. header = ["# mds, PoolSeqFlow analysis frame ${frameVersion()}", "# ${useCompiled(settings) ? 'allele_frequencies.cpp and nei_distance.cpp, compiled at run time' : 'allele_frequencies() and nei_distance(), vectorized R'}" + @@ -114,11 +103,11 @@ process Analyze { printf '%s' '${pools}' > pools.json printf '%s' '${options}' > options.json - # The script published beside the result, and the one that runs: the shared library first, - # then this module's own. + # The script published beside the result, and the one that runs: the declared libraries + # first, then this module's own. { printf '%s\\n' '${header}' - echo '# The shared library follows, then this module.' + echo '# The libraries this module declares follow, then the module itself.' cat ${library.join(' ')} cat ${moduleDir}/mds.R } > published/mds.R diff --git a/analysis/modules/mds/manifest.json b/modules/mds/manifest.json similarity index 94% rename from analysis/modules/mds/manifest.json rename to modules/mds/manifest.json index 6abd289..907bfbb 100644 --- a/analysis/modules/mds/manifest.json +++ b/modules/mds/manifest.json @@ -1,14 +1,20 @@ { "name": "mds", - "version": "20260910.001", + "version": "20260910.003", "contract": "freq-1", "license": "GPL-3.0-or-later", - "frame": "20260908.003", + "frame": "20260910.001", "environment": "3.0.0", "summary": "the pools placed by Nei's minimum distance, on a classical MDS (principal coordinates analysis)", "needs": [ "depths" ], + "libraries": [ + "n_eff", + "allele_frequencies", + "nei_distance", + "chunk_ranges" + ], "gates": [ "THE DISTANCE IS NEI'S MINIMUM DISTANCE, (J_A + J_B) / 2 - J_AB, corrected for the sampling in each pool and averaged over sites. J is the probability that two chromosomes drawn from one pool carry the same allele; the sum runs over EVERY allele of a site including the reference, so a triallelic site contributes three terms and no allele is privileged.", "IT IS NOT FLOORED AT ZERO AND SMALL NEGATIVE DISTANCES ARE CORRECT. The correction subtracts each pool's own sampling noise, so two pools drawn from one population estimate a true distance of zero and land either side of it. A negative entry in distance.tsv means the pools are indistinguishable at this depth, not that something failed.", @@ -62,5 +68,13 @@ "summary": "the compiled form of the pairwise distance accumulation, shipped whether or not this run used it", "anchor": "mds-compiled" } + ], + "packages": [ + "r-dofuture=1.3.0", + "r-foreach=1.5.2", + "r-future=1.75.0", + "r-ggplot2=4.0.3", + "r-jsonlite=2.0.0", + "r-rcpp=1.1.2" ] } diff --git a/analysis/modules/mds/mds.R b/modules/mds/mds.R similarity index 99% rename from analysis/modules/mds/mds.R rename to modules/mds/mds.R index 71b43bd..b224d0c 100644 --- a/analysis/modules/mds/mds.R +++ b/modules/mds/mds.R @@ -1,4 +1,4 @@ -# The module's own analysis. The shared library is above this line in the published copy. +# The module's own analysis. The libraries it declares are above this line in the published copy. # # mds.R --design design.json --pools pools.json --options options.json # --cpp-frequencies allele_frequencies.cpp --cpp-distance nei_distance.cpp diff --git a/analysis/modules/mds/references.bib b/modules/mds/references.bib similarity index 100% rename from analysis/modules/mds/references.bib rename to modules/mds/references.bib diff --git a/analysis/modules/mds/test/mds.sh b/modules/mds/test/mds.sh similarity index 95% rename from analysis/modules/mds/test/mds.sh rename to modules/mds/test/mds.sh index 80e4534..80cee9c 100644 --- a/analysis/modules/mds/test/mds.sh +++ b/modules/mds/test/mds.sh @@ -1,9 +1,9 @@ #!/bin/bash # mds, against the analytic corpus its own tools build. # cost: jvm -# covers: analysis/modules/mds/ analysis/lib/R/ analysis/lib/cpp/ +# covers: modules/mds/ modules/lib/ # covers: test/tools/freq_corpus.py -# covers: analysis.nf analysis/modules/mds/main.nf +# covers: analysis.nf modules/mds/main.nf # # The fixtures and helpers every analysis suite shares are in test/lib/analysis.sh. # @@ -25,20 +25,21 @@ mds_corpus() { # Run the module's R directly over the corpus, under one set of options, into $1. # -# Every .R in the shared library rather than the list main.nf names: they are standalone function +# Every library's .R rather than the list the manifest names: they are standalone function # definitions, so a superset is harmless, and the case then cannot go stale when that list -# changes. The Nextflow case is what proves main.nf assembles the same thing. +# changes. The Nextflow case is what proves main.nf assembles the same thing, and 00_static is +# what proves the manifest declares exactly what the module calls. mds_direct() { local dest="$1" options="$2" design="${3:-}" mkdir -p "$dest" [ -n "$design" ] || design="$CORPUS_DIR/design.json" - cat "$REPO_ROOT"/analysis/lib/R/*.R "$REPO_ROOT/analysis/modules/mds/mds.R" > "$dest/mds.R" + cat "$REPO_ROOT"/modules/lib/*/*.R "$REPO_ROOT/modules/mds/mds.R" > "$dest/mds.R" printf '%s' "$options" > "$dest/options.json" ( cd "$CORPUS_DIR/Frequencies" && Rscript --vanilla "$dest/mds.R" \ --design "$design" --pools "$CORPUS_DIR/pools.json" \ --options "$dest/options.json" \ - --cpp-frequencies "$REPO_ROOT/analysis/lib/cpp/allele_frequencies.cpp" \ - --cpp-distance "$REPO_ROOT/analysis/lib/cpp/nei_distance.cpp" \ + --cpp-frequencies "$REPO_ROOT/modules/lib/allele_frequencies/allele_frequencies.cpp" \ + --cpp-distance "$REPO_ROOT/modules/lib/nei_distance/nei_distance.cpp" \ --depths 'Test_snp_depth.tsv' --out "$dest" ) > "$dest/out.txt" 2>&1 } @@ -126,12 +127,12 @@ PY # The module's R over a cohort built by mds_wide_cohort in $1, options $2, output into $3. mds_on_cohort() { mkdir -p "$3" - cat "$REPO_ROOT"/analysis/lib/R/*.R "$REPO_ROOT/analysis/modules/mds/mds.R" > "$3/mds.R" + cat "$REPO_ROOT"/modules/lib/*/*.R "$REPO_ROOT/modules/mds/mds.R" > "$3/mds.R" printf '%s' "$2" > "$3/options.json" ( cd "$1/Frequencies" && Rscript --vanilla "$3/mds.R" \ --design "$1/design.json" --pools "$1/pools.json" --options "$3/options.json" \ - --cpp-frequencies "$REPO_ROOT/analysis/lib/cpp/allele_frequencies.cpp" \ - --cpp-distance "$REPO_ROOT/analysis/lib/cpp/nei_distance.cpp" \ + --cpp-frequencies "$REPO_ROOT/modules/lib/allele_frequencies/allele_frequencies.cpp" \ + --cpp-distance "$REPO_ROOT/modules/lib/nei_distance/nei_distance.cpp" \ --depths 'Test_snp_depth.tsv' --out "$3" ) > "$3/out.txt" 2>&1 } @@ -321,13 +322,13 @@ test_a_single_haploid_genome_is_refused() { sed 's/"nChrom": *[0-9]*/"nChrom": 1/; s/"ploidy": *[0-9]*/"ploidy": 1/; s/"size": *[0-9]*/"size": 1/' \ "$CORPUS_DIR/pools.json" > "$sb/one.json" mkdir -p "$sb/run" - cat "$REPO_ROOT"/analysis/lib/R/*.R "$REPO_ROOT/analysis/modules/mds/mds.R" > "$sb/run/mds.R" + cat "$REPO_ROOT"/modules/lib/*/*.R "$REPO_ROOT/modules/mds/mds.R" > "$sb/run/mds.R" printf '%s' "$MDS_OPTIONS" > "$sb/run/options.json" ( cd "$CORPUS_DIR/Frequencies" && Rscript --vanilla "$sb/run/mds.R" \ --design "$CORPUS_DIR/design.json" --pools "$sb/one.json" \ --options "$sb/run/options.json" \ - --cpp-frequencies "$REPO_ROOT/analysis/lib/cpp/allele_frequencies.cpp" \ - --cpp-distance "$REPO_ROOT/analysis/lib/cpp/nei_distance.cpp" \ + --cpp-frequencies "$REPO_ROOT/modules/lib/allele_frequencies/allele_frequencies.cpp" \ + --cpp-distance "$REPO_ROOT/modules/lib/nei_distance/nei_distance.cpp" \ --depths 'Test_snp_depth.tsv' --out "$sb/run" ) > "$sb/run/out.txt" 2>&1 assert_contains "$(cat "$sb/run/out.txt")" "one chromosome" "should say what is wrong" diff --git a/modules-repo/association-20260910.001.tar.gz b/modules/repo/association-20260910.001.tar.gz similarity index 100% rename from modules-repo/association-20260910.001.tar.gz rename to modules/repo/association-20260910.001.tar.gz diff --git a/modules-repo/basicstats-20260910.001.tar.gz b/modules/repo/basicstats-20260910.001.tar.gz similarity index 100% rename from modules-repo/basicstats-20260910.001.tar.gz rename to modules/repo/basicstats-20260910.001.tar.gz diff --git a/modules-repo/index.tsv b/modules/repo/index.tsv similarity index 78% rename from modules-repo/index.tsv rename to modules/repo/index.tsv index eb02dfa..022d93f 100644 --- a/modules-repo/index.tsv +++ b/modules/repo/index.tsv @@ -53,7 +53,7 @@ #!index-format: 1 #!index-version: 20260910.003 -name version contract frame environment url sha256 summary -basicstats 20260910.001 freq-1 20260908.003 3.0.0 https://ozankiratli.github.io/PoolSeqFlow/modules-repo/basicstats-20260910.001.tar.gz f154f07cbcc66abab0b330fb1a4d0ce55dfd7db0625d37f9840e6d083039883b site counts, depth, effective pool size and gene diversity, per pool -association 20260910.001 freq-1 20260908.003 3.0.0 https://ozankiratli.github.io/PoolSeqFlow/modules-repo/association-20260910.001.tar.gz 19b68feb0ca7879143e88b2a6e206b663e7d1bb9935d1e98fa42dcbdc7704913 each allele's frequency regressed on a phenotype measured per pool, with a permutation p -mds 20260910.001 freq-1 20260908.003 3.0.0 https://ozankiratli.github.io/PoolSeqFlow/modules-repo/mds-20260910.001.tar.gz 96797656267316ff5971981c8d940ab6ddc1fa34800eed6312a1c7ded1d5facf the pools placed by Nei's minimum distance, on a classical MDS (principal coordinates analysis) +name kind version contract frame environment url sha256 summary +basicstats module 20260910.001 freq-1 20260908.003 3.0.0 https://ozankiratli.github.io/PoolSeqFlow/modules-repo/basicstats-20260910.001.tar.gz f154f07cbcc66abab0b330fb1a4d0ce55dfd7db0625d37f9840e6d083039883b site counts, depth, effective pool size and gene diversity, per pool +association module 20260910.001 freq-1 20260908.003 3.0.0 https://ozankiratli.github.io/PoolSeqFlow/modules-repo/association-20260910.001.tar.gz 19b68feb0ca7879143e88b2a6e206b663e7d1bb9935d1e98fa42dcbdc7704913 each allele's frequency regressed on a phenotype measured per pool, with a permutation p +mds module 20260910.001 freq-1 20260908.003 3.0.0 https://ozankiratli.github.io/PoolSeqFlow/modules-repo/mds-20260910.001.tar.gz 96797656267316ff5971981c8d940ab6ddc1fa34800eed6312a1c7ded1d5facf the pools placed by Nei's minimum distance, on a classical MDS (principal coordinates analysis) diff --git a/modules-repo/mds-20260910.001.tar.gz b/modules/repo/mds-20260910.001.tar.gz similarity index 100% rename from modules-repo/mds-20260910.001.tar.gz rename to modules/repo/mds-20260910.001.tar.gz diff --git a/poolseqflow.nf b/poolseqflow.nf index 5b319e2..ccfae9a 100644 --- a/poolseqflow.nf +++ b/poolseqflow.nf @@ -217,6 +217,6 @@ workflow { software : params.software, dir : params.dir, annotate : run_defs.any { r -> r.annotate }, - citationsData: "${projectDir}/install/citations.json".toString() ] + citationsData: "${projectDir}/citations/citations.json".toString() ] Citations(citations_run, VCF2Frequencies.out.collect()) } diff --git a/test/lib/analysis.sh b/test/lib/analysis.sh index 867855f..3b21fa9 100644 --- a/test/lib/analysis.sh +++ b/test/lib/analysis.sh @@ -239,7 +239,7 @@ R_LIB_OUTPUT="" r_lib_section() { local out status=0 out=$(Rscript --vanilla "$REPO_ROOT/test/tools/r_lib_tests.R" \ - "$REPO_ROOT/analysis/lib/R" "$1" 2>&1) || status=$? + "$REPO_ROOT/modules/lib" "$1" 2>&1) || status=$? R_LIB_OUTPUT="$out" printf '%s' "$status" } @@ -493,8 +493,52 @@ analysis_archive_main() { } # One analysis, start to finish: verify, then the module itself. +# The real module out of modules// into the sandbox store, the way `modules install` does +# it: the module's own directory without its cases, then every library its manifest declares, +# under the store's library directory. +# +# NO MODULE SHIPS INSIDE A RELEASE. A sandbox built from the payload therefore has an EMPTY +# store, and a module named without this is simply not installed - which is what the frame says, +# correctly, before anything runs. Nothing to do for a fixture module planted by +# analysis_install_module: it has no directory under modules/ and is left where it was put. +# +# The `libraries` list is read with python3 rather than through the wrapper's own +# module_libraries(), so the harness and the shipped reader are two implementations of one +# thing. A bug in the shipped one shows up here as a missing library instead of being copied. +analysis_install_from_source() { + local name="$1" store="$ANALYSIS_SB/install/analysis/modules" + [ -d "$REPO_ROOT/modules/$name" ] || return 0 + [ -d "$store/$name" ] && return 0 + mkdir -p "$store" + cp -r "$REPO_ROOT/modules/$name" "$store/$name" + rm -rf "$store/$name/test" + + # Transitively, because a library may declare libraries of its own. None does today and the + # design does not forbid it, so the worklist costs nothing and cannot be the thing that + # breaks when one does. + local pending; pending=$(_analysis_declared_libraries "$store/$name/manifest.json") + local lib next + while [ -n "$pending" ]; do + next="" + for lib in $pending; do + [ -d "$REPO_ROOT/modules/lib/$lib" ] || continue + [ -d "$store/lib/$lib" ] && continue + mkdir -p "$store/lib" + cp -r "$REPO_ROOT/modules/lib/$lib" "$store/lib/$lib" + next="$next $(_analysis_declared_libraries "$store/lib/$lib/manifest.json")" + done + pending="$next" + done +} + +_analysis_declared_libraries() { + [ -f "$1" ] || return 0 + python3 -c 'import json,sys; print(" ".join(json.load(open(sys.argv[1])).get("libraries", [])))' "$1" +} + analysis_run_module() { local module="$1" status + analysis_install_from_source "$module" status=$(run_analysis "$ANALYSIS_SB" "$module") [ "$status" = "0" ] || { printf 'verify:%s\n' "$status"; return 0; } run_module "$ANALYSIS_SB" "$module" diff --git a/test/lib/sandbox.sh b/test/lib/sandbox.sh index 5a245fe..ecde2b8 100644 --- a/test/lib/sandbox.sh +++ b/test/lib/sandbox.sh @@ -55,12 +55,12 @@ make_pipeline_sandbox() { sb=$(guard_path "$TEST_TMPDIR/$name") rm -rf "$sb" mkdir -p "$sb/install" "$sb/main" "$sb/store" - # install/ too: it carries citations.json, which a run reads at the end. Copied whole - # rather than by file, so the next thing added there is present without a change here. - # manual/ is in PAYLOAD_ITEMS as well: a published analysis links each file it holds to a - # section of it, and the frame checks those anchors against the installed copy. + # citations/ too: it carries citations.json, which a run reads at the end. Each directory + # is copied whole rather than by file, so the next thing added to one is present without a + # change here. manual/ is in PAYLOAD_ITEMS as well: a published analysis links each file it + # holds to a section of it, and the frame checks those anchors against the installed copy. cp -r "$REPO_ROOT"/scripts "$REPO_ROOT"/bin "$REPO_ROOT"/lib "$REPO_ROOT"/analysis \ - "$REPO_ROOT"/install "$REPO_ROOT"/manual "$sb/install"/ + "$REPO_ROOT"/install "$REPO_ROOT"/citations "$REPO_ROOT"/manual "$sb/install"/ cp "$REPO_ROOT"/poolseqflow.nf "$REPO_ROOT"/dryrun.nf "$REPO_ROOT"/analysis.nf \ "$REPO_ROOT"/nextflow.config "$sb/install"/ # The wrapper, so cases can exercise clean/reset against a real project instead of @@ -712,12 +712,14 @@ run_launcher_with_envs() { rm -rf "$sb" mkdir -p "$sb/install" "$sb/bin" "$sb/scripts" cp "$REPO_ROOT/PoolSeqFlow" "$sb/" - # check_install.sh does real work against a real environment; these tests are about - # environment selection, so it is stubbed to a success. - printf '#!/bin/bash\necho "STUB check_install ran"\n' > "$sb/install/check_install.sh" + # Both check scripts do real work against a real environment - check_project.sh runs + # Nextflow as well - and these tests are about which script the wrapper reaches and with + # what activated, so each is stubbed to a success that names itself. + printf '#!/bin/bash\necho "STUB check_install ran"\n' > "$sb/bin/check_install.sh" + printf '#!/bin/bash\necho "STUB check_project ran in $PWD"\n' > "$sb/bin/check_project.sh" printf 'name: stub\n' > "$sb/install/environment.yml" printf 'name: stub\n' > "$sb/install/environment-analysis.yml" - chmod +x "$sb/install/check_install.sh" + chmod +x "$sb/bin/check_install.sh" "$sb/bin/check_project.sh" # A complete payload, because `install` refuses to deploy an incomplete copy - it would # otherwise produce an installation missing a file, which is worse than failing. Empty # placeholders are enough: nothing here ever runs Nextflow. @@ -745,6 +747,15 @@ run_launcher_with_envs() { # lib/ makes every launcher case fail before it reaches what it is testing. cp "$REPO_ROOT/lib/wrapper_lib.sh" "$sb/lib/" + # A project to stand in, for the arms that read one. Its content is whatever the case set: + # `storageDir` is the key require_migrated_config turns on, so a case chooses between a + # current config and an older one by writing that line or not. + if [ -n "${LAUNCHER_PROJECT_CONFIG:-}" ]; then + printf '%s\n' "$LAUNCHER_PROJECT_CONFIG" > "$sb/parameters.config" + else + rm -f "$sb/parameters.config" + fi + # The stub conda goes in its own directory rather than $sb/bin, which belongs to the # pipeline and is part of what `install` deploys - a fake conda inside the payload would # be copied into every test installation. @@ -814,7 +825,7 @@ run_analysis_launcher_with_envs() { local sb sb=$(guard_path "$TEST_TMPDIR/analysis-launcher") rm -rf "$sb" - mkdir -p "$sb/install" "$sb/lib" "$sb/analysis" + mkdir -p "$sb/install" "$sb/lib" "$sb/bin" "$sb/analysis" cp "$REPO_ROOT/PoolSeqFlow" "$sb/" cp "$REPO_ROOT/lib/wrapper_lib.sh" "$sb/lib/" : > "$sb/analysis.nf" @@ -830,11 +841,11 @@ run_analysis_launcher_with_envs() { mkdir -p "$sb/analysis/lib/nf" cp "$REPO_ROOT/analysis/lib/nf/modules.nf" "$sb/analysis/lib/nf/" printf 'name: stub\n' > "$sb/install/environment-analysis.yml" - # Stubbed for the same reason install/check_install.sh is: it needs a real R environment, + # Stubbed for the same reason bin/check_install.sh is: it needs a real R environment, # and these tests are about which environment is chosen. printf '#!/bin/bash\necho "STUB check_analysis_install ran"\n' \ - > "$sb/install/check_analysis_install.sh" - chmod +x "$sb/install/check_analysis_install.sh" + > "$sb/bin/check_analysis_install.sh" + chmod +x "$sb/bin/check_analysis_install.sh" if [ -n "${LAUNCHER_STORE_MODULE:-}" ]; then local store="$sb/analysis/modules/$LAUNCHER_STORE_MODULE" diff --git a/test/run_tests.sh b/test/run_tests.sh index 9aa28eb..a4b1f0b 100755 --- a/test/run_tests.sh +++ b/test/run_tests.sh @@ -256,7 +256,7 @@ for suite in "$SCRIPT_DIR"/suites/*.sh; do [ -f "$suite" ] || continue SUITES+=("$suite") done -for suite in "$REPO_ROOT"/analysis/modules/*/test/*.sh; do +for suite in "$REPO_ROOT"/modules/*/test/*.sh; do [ -f "$suite" ] || continue SUITES+=("$suite") done diff --git a/test/suites/00_static.sh b/test/suites/00_static.sh index c0d81c4..0766571 100644 --- a/test/suites/00_static.sh +++ b/test/suites/00_static.sh @@ -1,8 +1,8 @@ #!/bin/bash # Checks that need no data: syntax, release packaging, version consistency. # cost: static -# covers: PoolSeqFlow install/ dev/scripts/ modules-repo/index.tsv .gitattributes -# covers: analysis/citations.json install/citations.json install/references.bib +# covers: PoolSeqFlow install/ dev/scripts/ modules/repo/index.tsv .gitattributes +# covers: analysis/citations.json citations/citations.json citations/references.bib # covers: analysis/references.bib manual/references.bib # `nextflow lint` was brought to zero warnings during the post-2.2.0 audit. Held there @@ -10,13 +10,49 @@ test_nextflow_lint_is_clean() { have_tools || { skip_case "no conda environment"; return; } local out + # Everything but modules/, which is linted below in the layout it runs in. out=$(cd "$REPO_ROOT" && PATH="$TEST_CONDA_ENV/bin:$PATH" \ JAVA_HOME="$TEST_CONDA_ENV" JAVA_CMD="$TEST_CONDA_ENV/bin/java" \ - nextflow lint . 2>&1) + nextflow lint analysis scripts lib bin install \ + poolseqflow.nf analysis.nf dryrun.nf nextflow.config 2>&1) assert_contains "$out" "had no errors" "lint should report no errors" assert_not_contains "$out" "warning" "lint should report no warnings" } +# A MODULE IS LINTED WHERE IT RUNS, NOT WHERE GIT KEEPS IT. +# +# A module's source lives in modules// and is installed into analysis/modules//, and +# its `include` of the frame is written '../../lib/nf/...' - correct from the STORE and +# meaningless from the source directory. That is the cost of the store and the sources being +# different places, which is deliberate: while they were one directory the modules shipped inside +# every release because they were sources sitting in the install path. +# +# So the check is not "skip the modules" but "assemble the layout a module actually sees". Linting +# them in modules/ would report errors that say nothing, and linting nothing at all would let a +# real one through. +test_every_module_lints_in_the_store_layout() { + have_tools || { skip_case "no conda environment"; return; } + local sb; sb=$(guard_path "$TEST_TMPDIR/module-lint") + rm -rf "$sb"; mkdir -p "$sb/analysis/modules" + cp -r "$REPO_ROOT/analysis/lib" "$sb/analysis/" + cp -r "$REPO_ROOT/scripts" "$sb/" + cp "$REPO_ROOT/nextflow.config" "$sb/" + local found=0 dir + for dir in "$REPO_ROOT"/modules/*/; do + [ -f "$dir/main.nf" ] || continue # modules/lib holds libraries, not modules + cp -r "$dir" "$sb/analysis/modules/" + found=$((found + 1)) + done + [ "$found" -gt 0 ] || { fail_case "no module sources found under modules/"; return; } + + local out + out=$(cd "$sb" && PATH="$TEST_CONDA_ENV/bin:$PATH" \ + JAVA_HOME="$TEST_CONDA_ENV" JAVA_CMD="$TEST_CONDA_ENV/bin/java" \ + nextflow lint "$sb/analysis/modules" 2>&1) + assert_contains "$out" "had no errors" "modules should lint in the store layout:"$'\n'"$out" + assert_not_contains "$out" "had errors" "no module should report an error:"$'\n'"$out" +} + test_shell_scripts_parse() { local script bad=0 while read -r script; do @@ -133,14 +169,14 @@ test_the_release_archive_gate_passes() { # published after a release is installable into it. A copy inside the tarball would be a second # answer to what can be installed, frozen on the day the release was built. test_the_module_catalogue_never_reaches_a_release() { - local index="modules-repo/index.tsv" + local index="modules/repo/index.tsv" [ -f "$REPO_ROOT/$index" ] || { fail_case "$index is missing"; return; } # The archive itself, not `git check-attr`: the catalogue is covered by a directory pattern # now, and check-attr reports `unspecified` for a file inside one even though git archive # excludes it - `test/run_tests.sh` answers the same way. What matters is the tarball. local listing; listing=$(working_tree_archive) [ -n "$listing" ] || { skip_case "git archive produced nothing"; return; } - assert_not_contains "$listing" "modules-repo" \ + assert_not_contains "$listing" "modules/repo" \ "the catalogue and the tarballs beside it must not reach a release" # And the release must not be able to fall back to a copy of its own: a frozen catalogue # inside a tarball would be a second answer to what can be installed. @@ -157,17 +193,17 @@ test_the_module_catalogue_never_reaches_a_release() { # Only rows pointing into this repository are checked. A third-party row would name a host # nothing here can see, and asserting on that would fail for a reason that is not ours. test_every_catalogue_row_has_the_tarball_it_advertises() { - local index="$REPO_ROOT/modules-repo/index.tsv" - [ -f "$index" ] || { fail_case "modules-repo/index.tsv is missing"; return; } + local index="$REPO_ROOT/modules/repo/index.tsv" + [ -f "$index" ] || { fail_case "modules/repo/index.tsv is missing"; return; } command -v sha256sum > /dev/null 2>&1 || { skip_case "no sha256sum"; return; } local rows=0 name version url sha file - while IFS=$'\t' read -r name version _contract _frame _env url sha _summary; do + while IFS=$'\t' read -r name _kind version _contract _frame _env url sha _summary; do case "$name" in ''|'#'*|name) continue ;; esac case "$url" in *"/modules-repo/"*) ;; *) continue ;; esac rows=$((rows + 1)) - file="$REPO_ROOT/modules-repo/${url##*/}" - [ -f "$file" ] || { fail_case "$name $version: the catalogue names ${url##*/}, which is not in modules-repo/" + file="$REPO_ROOT/modules/repo/${url##*/}" + [ -f "$file" ] || { fail_case "$name $version: the catalogue names ${url##*/}, which is not in modules/repo/" continue; } local actual; actual=$(sha256sum "$file" | awk '{print $1}') assert_eq "$sha" "$actual" "$name $version: the row's sha256 is not ${url##*/}'s" @@ -182,7 +218,7 @@ test_every_catalogue_row_has_the_tarball_it_advertises() { # reads as an empty field in every row rather than as an error. test_the_module_catalogue_header_is_the_one_the_wrapper_reads() { local header wanted column - header=$(grep -v '^[[:space:]]*#' "$REPO_ROOT/modules-repo/index.tsv" \ + header=$(grep -v '^[[:space:]]*#' "$REPO_ROOT/modules/repo/index.tsv" \ | grep -v '^[[:space:]]*$' | head -1) # Taken from the wrapper rather than written out here: the columns are matched by name, so # the coupling to assert is that every name it looks for is in the header - not that the @@ -202,7 +238,7 @@ test_the_module_catalogue_header_is_the_one_the_wrapper_reads() { # instead of reading the wrong field out of each row; the test above cannot protect a wrapper # that has already shipped. test_the_module_catalogue_declares_its_layout_and_its_version() { - local index="$REPO_ROOT/modules-repo/index.tsv" + local index="$REPO_ROOT/modules/repo/index.tsv" local format version supported format=$(sed -n 's|^#![[:space:]]*index-format:[[:space:]]*\(.*\)$|\1|p' "$index" | head -1 | tr -d ' ') version=$(sed -n 's|^#![[:space:]]*index-version:[[:space:]]*\(.*\)$|\1|p' "$index" | head -1 | tr -d ' ') @@ -241,16 +277,23 @@ test_the_analysis_version_scripts_are_there_and_runnable() { # answer depends on whether the tree it reads is dirty - which this one's is not, most days. test_the_frame_version_moves_with_a_change_and_not_with_the_calendar() { local sb; sb=$(guard_path "$TEST_TMPDIR/version-rule") - rm -rf "$sb"; mkdir -p "$sb/dev/scripts" "$sb/analysis/lib/R" \ - "$sb/analysis/modules/demo/test" "$sb/modules-repo" + rm -rf "$sb"; mkdir -p "$sb/dev/scripts" "$sb/analysis/lib/nf" \ + "$sb/modules/demo/test" "$sb/modules/repo" \ + "$sb/analysis/modules/ghost" cp "$REPO_ROOT/dev/scripts/check-analysis-versions.sh" "$sb/dev/scripts/" printf 'frame {}\n' > "$sb/analysis/frame.config" printf '20260101.001\n' > "$sb/analysis/frame.version" - printf 'f <- function() 1\n' > "$sb/analysis/lib/R/thing.R" - printf '#!index-format: 1\n#!index-version: 20260101.001\n' > "$sb/modules-repo/index.tsv" - printf '{"name": "demo", "version": "20260101.001"}\n' > "$sb/analysis/modules/demo/manifest.json" - printf 'workflow {}\n' > "$sb/analysis/modules/demo/main.nf" - printf 'echo case\n' > "$sb/analysis/modules/demo/test/demo.sh" + printf 'def one() { 1 }\n' > "$sb/analysis/lib/nf/thing.nf" + printf '#!index-format: 1\n#!index-version: 20260101.001\n' > "$sb/modules/repo/index.tsv" + printf '{"name": "demo", "version": "20260101.001"}\n' > "$sb/modules/demo/manifest.json" + printf 'workflow {}\n' > "$sb/modules/demo/main.nf" + printf 'echo case\n' > "$sb/modules/demo/test/demo.sh" + # `ghost` is planted in the INSTALL STORE, which the gate must not read. The store is + # gitignored and empty in a checkout, so a loop pointed at it iterates nothing and reports + # every module fine - which is exactly what happened and went undetected, because this + # fixture used to plant `demo` there too and so agreed with the bug. + printf '{"name": "ghost", "version": "20260101.001"}\n' > "$sb/analysis/modules/ghost/manifest.json" + printf 'workflow {}\n' > "$sb/analysis/modules/ghost/main.nf" # Committed AS OF the day the version names, because a clean tree is compared against the # commit date and this fixture would otherwise say the frame changed today. (cd "$sb" && git init -q . && git add -A \ @@ -263,31 +306,39 @@ test_the_frame_version_moves_with_a_change_and_not_with_the_calendar() { out=$(cd "$sb" && bash dev/scripts/check-analysis-versions.sh 2>&1) assert_contains "$out" "up to date" "an untouched frame needs no new version:"$'\n'"$out" - printf 'g <- function() 2\n' >> "$sb/analysis/lib/R/thing.R" + printf 'def two() { 2 }\n' >> "$sb/analysis/lib/nf/thing.nf" out=$(cd "$sb" && bash dev/scripts/check-analysis-versions.sh 2>&1 || true) assert_contains "$out" "BEHIND" "a changed frame with a stale version:"$'\n'"$out" # The version now names THE DAY THE CHANGE WAS MADE, which is January and not today. This # is the assertion that fails if the dirty answer goes back to being today's date. - touch -d '2026-01-02T00:00:00Z' "$sb/analysis/lib/R/thing.R" + touch -d '2026-01-02T00:00:00Z' "$sb/analysis/lib/nf/thing.nf" printf '20260102.001\n' > "$sb/analysis/frame.version" out=$(cd "$sb" && bash dev/scripts/check-analysis-versions.sh 2>&1) assert_contains "$out" "up to date" \ "a bump dated to the change stays good however long it sits:"$'\n'"$out" - # A MODULE'S OWN CASES ARE NOT THE MODULE. analysis/modules/*/test/ carries export-ignore, + # A MODULE'S OWN CASES ARE NOT THE MODULE. publish-module.sh drops test/ from the tarball, # so nothing there reaches a published module - and the manifest version is what an # installation and every published result record the module by. Fixing a case must not move # it; touching what the module computes must. - printf 'echo another case\n' >> "$sb/analysis/modules/demo/test/demo.sh" + printf 'echo another case\n' >> "$sb/modules/demo/test/demo.sh" out=$(cd "$sb" && bash dev/scripts/check-analysis-versions.sh 2>&1) assert_not_contains "$out" "module 'demo'" \ "a module's test changing is not the module changing:"$'\n'"$out" - printf 'process P {}\n' >> "$sb/analysis/modules/demo/main.nf" + printf 'process P {}\n' >> "$sb/modules/demo/main.nf" out=$(cd "$sb" && bash dev/scripts/check-analysis-versions.sh 2>&1 || true) assert_contains "$out" "module 'demo'" \ "but its main.nf changing is:"$'\n'"$out" + + # And the store is still not a source, however stale what sits in it looks. `ghost` has + # been changed and never bumped for the whole of this case; a gate reading the store would + # have named it by now. + printf 'process Q {}\n' >> "$sb/analysis/modules/ghost/main.nf" + out=$(cd "$sb" && bash dev/scripts/check-analysis-versions.sh 2>&1 || true) + assert_not_contains "$out" "ghost" \ + "the install store is not checked for version bumps:"$'\n'"$out" } # THE RELEASE GATE REFUSES TO ANSWER RATHER THAN ANSWERING WRONGLY. @@ -302,18 +353,18 @@ test_the_frame_version_moves_with_a_change_and_not_with_the_calendar() { test_the_release_gate_refuses_what_it_cannot_check() { local sb out sb=$(guard_path "$TEST_TMPDIR/release-gate") - rm -rf "$sb"; mkdir -p "$sb/origin/dev/scripts" "$sb/origin/analysis/lib/R" \ - "$sb/origin/modules-repo" + rm -rf "$sb"; mkdir -p "$sb/origin/dev/scripts" "$sb/origin/analysis/lib/nf" \ + "$sb/origin/modules/repo" cp "$REPO_ROOT/dev/scripts/check-analysis-versions.sh" "$sb/origin/dev/scripts/" printf 'frame {}\n' > "$sb/origin/analysis/frame.config" printf '20260101.001\n' > "$sb/origin/analysis/frame.version" - printf 'f <- function() 1\n' > "$sb/origin/analysis/lib/R/thing.R" - printf '#!index-format: 1\n#!index-version: 20260101.001\n' > "$sb/origin/modules-repo/index.tsv" + printf 'def one() { 1 }\n' > "$sb/origin/analysis/lib/nf/thing.nf" + printf '#!index-format: 1\n#!index-version: 20260101.001\n' > "$sb/origin/modules/repo/index.tsv" # Two commits, because a shallow clone of a one-commit repository is not shallow. (cd "$sb/origin" && git init -q . && git add -A \ && GIT_COMMITTER_DATE='2026-01-01T00:00:00Z' \ git -c user.email=t@t -c user.name=t commit -qm base --date='2026-01-01T00:00:00Z' \ - && printf 'g <- function() 2\n' >> analysis/lib/R/thing.R \ + && printf 'def two() { 2 }\n' >> analysis/lib/nf/thing.nf \ && printf '20260102.001\n' > analysis/frame.version \ && git add -A \ && GIT_COMMITTER_DATE='2026-01-02T00:00:00Z' \ @@ -340,10 +391,10 @@ test_the_release_gate_refuses_what_it_cannot_check() { # An uncommitted change under analysis/ is dated by mtime, which on a fresh checkout is # checkout time and says nothing about when the work was done. - printf 'h <- function() 3\n' >> "$sb/origin/analysis/lib/R/thing.R" + printf 'def three() { 3 }\n' >> "$sb/origin/analysis/lib/nf/thing.nf" out=$(cd "$sb/origin" && bash dev/scripts/check-analysis-versions.sh --release 2>&1 || true) assert_contains "$out" "REFUSED" "a dirty analysis/ must be refused at a release:"$'\n'"$out" - assert_contains "$out" "thing.R" "naming what is uncommitted" + assert_contains "$out" "thing.nf" "naming what is uncommitted" } # THE MODULE CHECK MUST BITE ON A CLEAN TREE, WHICH IS THE ONLY STATE A RELEASE IS EVER IN. @@ -351,16 +402,21 @@ test_the_release_gate_refuses_what_it_cannot_check() { test_a_committed_module_change_without_a_version_bump_is_caught() { local sb out sb=$(guard_path "$TEST_TMPDIR/module-version-committed") - rm -rf "$sb"; mkdir -p "$sb/dev/scripts" "$sb/analysis/lib/R" \ - "$sb/analysis/modules/demo/test" "$sb/modules-repo" + rm -rf "$sb"; mkdir -p "$sb/dev/scripts" "$sb/analysis/lib/nf" \ + "$sb/modules/demo/test" "$sb/modules/lib/helper" "$sb/modules/repo" cp "$REPO_ROOT/dev/scripts/check-analysis-versions.sh" "$sb/dev/scripts/" printf 'frame {}\n' > "$sb/analysis/frame.config" printf '20260101.001\n' > "$sb/analysis/frame.version" - printf 'f <- function() 1\n' > "$sb/analysis/lib/R/thing.R" - printf '#!index-format: 1\n#!index-version: 20260101.001\n' > "$sb/modules-repo/index.tsv" - printf '{"name": "demo", "version": "20260101.001"}\n' > "$sb/analysis/modules/demo/manifest.json" - printf 'workflow {}\n' > "$sb/analysis/modules/demo/main.nf" - printf 'echo case\n' > "$sb/analysis/modules/demo/test/demo.sh" + printf 'def one() { 1 }\n' > "$sb/analysis/lib/nf/thing.nf" + printf '#!index-format: 1\n#!index-version: 20260101.001\n' > "$sb/modules/repo/index.tsv" + printf '{"name": "demo", "version": "20260101.001"}\n' > "$sb/modules/demo/manifest.json" + printf 'workflow {}\n' > "$sb/modules/demo/main.nf" + printf 'echo case\n' > "$sb/modules/demo/test/demo.sh" + # A library carries a manifest and a version exactly like a module and is published the + # same way, so it is owed the same check under the other half of the glob. + printf '{"name": "helper", "kind": "library", "version": "20260101.001"}\n' \ + > "$sb/modules/lib/helper/manifest.json" + printf 'h <- function() 1\n' > "$sb/modules/lib/helper/helper.R" (cd "$sb" && git init -q . && git add -A \ && GIT_COMMITTER_DATE='2026-01-01T00:00:00Z' \ git -c user.email=t@t -c user.name=t commit -qm base --date='2026-01-01T00:00:00Z') \ @@ -368,7 +424,7 @@ test_a_committed_module_change_without_a_version_bump_is_caught() { || { skip_case "could not build a repository to check in"; return; } # Commit a change to what the module computes, and do not move its version. - (cd "$sb" && printf 'process P {}\n' >> analysis/modules/demo/main.nf && git add -A \ + (cd "$sb" && printf 'process P {}\n' >> modules/demo/main.nf && git add -A \ && GIT_COMMITTER_DATE='2026-01-02T00:00:00Z' \ git -c user.email=t@t -c user.name=t commit -qm 'change demo' \ --date='2026-01-02T00:00:00Z') > /dev/null 2>&1 @@ -377,8 +433,8 @@ test_a_committed_module_change_without_a_version_bump_is_caught() { "a committed module change with a stale version must be caught:"$'\n'"$out" # And a commit that moves the version along with the change is not reported. - (cd "$sb" && printf 'process Q {}\n' >> analysis/modules/demo/main.nf \ - && printf '{"name": "demo", "version": "20260103.001"}\n' > analysis/modules/demo/manifest.json \ + (cd "$sb" && printf 'process Q {}\n' >> modules/demo/main.nf \ + && printf '{"name": "demo", "version": "20260103.001"}\n' > modules/demo/manifest.json \ && git add -A \ && GIT_COMMITTER_DATE='2026-01-03T00:00:00Z' \ git -c user.email=t@t -c user.name=t commit -qm 'change demo and bump' \ @@ -387,15 +443,24 @@ test_a_committed_module_change_without_a_version_bump_is_caught() { assert_not_contains "$out" "module 'demo'" \ "while a change committed with its bump is not:"$'\n'"$out" - # A module's own cases are not the module: analysis/modules/*/test/ is export-ignored, so - # nothing there reaches a published module. - (cd "$sb" && printf 'echo more\n' >> analysis/modules/demo/test/demo.sh && git add -A \ + # A module's own cases are not the module: publish-module.sh drops test/ from the tarball, + # so nothing there reaches a published module. + (cd "$sb" && printf 'echo more\n' >> modules/demo/test/demo.sh && git add -A \ && GIT_COMMITTER_DATE='2026-01-04T00:00:00Z' \ git -c user.email=t@t -c user.name=t commit -qm 'a case only' \ --date='2026-01-04T00:00:00Z') > /dev/null 2>&1 out=$(cd "$sb" && bash dev/scripts/check-analysis-versions.sh 2>&1 || true) assert_not_contains "$out" "module 'demo'" \ "and a commit touching only its cases is not the module changing:"$'\n'"$out" + + # The same question of a LIBRARY, which is the half of the glob a module never exercises. + (cd "$sb" && printf 'i <- function() 2\n' >> modules/lib/helper/helper.R && git add -A \ + && GIT_COMMITTER_DATE='2026-01-05T00:00:00Z' \ + git -c user.email=t@t -c user.name=t commit -qm 'change helper' \ + --date='2026-01-05T00:00:00Z') > /dev/null 2>&1 + out=$(cd "$sb" && bash dev/scripts/check-analysis-versions.sh 2>&1 || true) + assert_contains "$out" "helper" \ + "a committed library change with a stale version must be caught too:"$'\n'"$out" } test_release_archive_carries_the_runtime() { @@ -405,7 +470,13 @@ test_release_archive_carries_the_runtime() { local needed for needed in "poolseqflow.nf" "nextflow.config" "parameters.config.template" \ "PoolSeqFlow" "analysis.nf" "metadata.csv.template" \ - "scripts/" "bin/" "lib/" "analysis/" "install/"; do + "scripts/" "bin/" "lib/" "analysis/" "install/" "citations/"; do + assert_contains "$listing" "$needed" "release tarball must carry $needed" + done + # Named files and not only their directories: a directory traveling empty would satisfy + # every line above, and each of these is read at run time by something that does not check. + for needed in "citations/citations.json" "bin/check_install.sh" "bin/check_project.sh" \ + "install/environment.yml"; do assert_contains "$listing" "$needed" "release tarball must carry $needed" done } @@ -542,7 +613,7 @@ test_check_install_hint_uses_the_versioned_environment() { # a branch it never entered, so it passed where the install was broken and failed where it # worked. /usr/bin:/bin because an empty PATH kills the script at `dirname` long before the # summary. - out=$(cd "$REPO_ROOT" && env -u ENV_NAME PATH=/usr/bin:/bin bash install/check_install.sh 2>&1) + out=$(cd "$REPO_ROOT" && env -u ENV_NAME PATH=/usr/bin:/bin bash bin/check_install.sh 2>&1) status=$? # The precondition, asserted rather than assumed: a machine carrying every tool on the bare # PATH would otherwise fail below on an empty string, which reads like a broken epilogue. @@ -572,6 +643,39 @@ test_prep_version_rejects_a_malformed_version() { # Release-prep logs are one machine's package solve on one day. They must not become # history, and the broad `!test/**` style re-inclusions elsewhere make that worth asserting. +# THE RELEASE BODY IS THIS VERSION'S CHANGELOG SECTION, and release.yml publishes whatever the +# extractor prints. A tag build is the expensive place to discover the section is missing, so the +# same extractor is the gate in release.yml's version step - this checks it answers here first. +# +# The executable bit is checked because release.yml calls the script bare. A file committed +# 100644 passes every other check in this suite and dies with "Permission denied" in the first +# step of a tag build. +test_the_release_body_extracts_for_this_version() { + local script="$REPO_ROOT/dev/scripts/changelog-section.sh" + [ -x "$script" ] || { fail_case "dev/scripts/changelog-section.sh is not executable"; return; } + + local version; version=$(sed -n 's/^VERSION="\(.*\)"$/\1/p' "$REPO_ROOT/PoolSeqFlow" | head -1) + local out status + out=$(cd "$REPO_ROOT" && "$script" "$version" 2>&1) && status=0 || status=$? + assert_status 0 "$status" "the changelog has no section for $version:"$'\n'"$out" + assert_contains "$out" "## [$version]" "the section should start with its own heading" + + # A version the changelog does not describe must fail, not print an empty body. + out=$(cd "$REPO_ROOT" && "$script" 99.99.99 2>&1) && status=0 || status=$? + assert_status 1 "$status" "an absent section should be refused" + assert_contains "$out" "no '## [99.99.99]' section" "and should say what is missing" + + # The standing tail release.yml appends, and its authoring comment, which must not reach a + # reader: the sed that strips it is anchored on a line that has to stay a line of its own. + local tail_file="$REPO_ROOT/dev/release-notes-tail.md" + [ -f "$tail_file" ] || { fail_case "dev/release-notes-tail.md is missing"; return; } + assert_contains "$(sed -n '/^-->$/p' "$tail_file")" "-->" \ + "the tail's comment must close on a line of its own, or the sed strips the whole file" + local rendered; rendered=$(sed "1,/^-->$/d" "$tail_file") + assert_not_contains "$rendered" "