docs: scope reproducibility claims to x86_64 + fix false reproduction-path claims - #4
Open
Permea-lab-admin wants to merge 1 commit into
Open
Permea-lab-admin wants to merge 1 commit into
Permea-lab-admin wants to merge 1 commit into
Conversation
The README claimed reproduction "on a laptop" / "on commodity hardware" and
that every number reproduces on CPU. That is unscoped and untrue on arm64:
PR-AUC and MCC deviate ~0.001 from float summation order (the F1-max operating
point can jump a grid point). ROC-AUC reproduces exactly everywhere. State the
truth: bit-exact on x86_64 with the pinned environment; ~0.001 on arm64,
expected numerical behaviour.
Also corrects two inaccurate reproduction-path claims:
- "The MMseqs2 binary ships under tools/mmseqs/bin" — no such directory or
binary exists in the repo. Replaced with real install instructions and a
pointer to the deposited canonical cluster artifacts (results/clusters/),
so numbers reproduce without re-clustering (cluster assignments are
build-sensitive: mmseqs 18-8cc5c gives 2918 clusters, not the published
2913).
- Dataset source cited as "permanent public source (Zenodo 20093418)". The
regeneration script actually fetches the B3Pred FASTAs from a live web
server (webs.iiitd.edu.in/raghava/b3pred/); the Zenodo record is a software
zip whose FASTAs are not verified byte-identical. State the real source and
that it is a live server, not an archive.
Depends on the canonical cluster deposit (PR #3): the results/clusters/
reference resolves once that merges.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Corrects the README's reproducibility section, which overstates what reproduces and where, and asserts two files/sources that are not what it says.
The unscoped claim (the main fix)
The README said reproduction happens "on a laptop" / "on commodity hardware" and that "a laptop reproduces every reported number." This is untrue on arm64. Measured, with the paper's exact pinned environment and the canonical 2913-cluster artifact:
~0.001 deviation on PR-AUC/MCC from floating-point summation order — the F1-maximising operating point (argmax over a discrete 19-point grid) can land on an adjacent grid point. ROC-AUC is rank-based and reproduces exactly. This is expected numerical behaviour, not a bug — but the claim must be scoped to where it holds: bit-exact on x86_64 with the pinned environment.
Two false reproduction-path claims
tools/mmseqs/bin" — no such directory or binary exists in the repo. Replaced with real install instructions and a pointer to the deposited canonical cluster artifacts. Cluster assignments are build-sensitive (mmseqs18-8cc5c→ 2918 clusters, not the published 2913), so reproducing the numbers should use the deposited artifacts, not re-clustering.regenerate.pyactually fetches the B3Pred FASTAs from a live web server (webs.iiitd.edu.in/raghava/b3pred/). The Zenodo record is a software zip whose FASTAs are not verified byte-identical. Corrected to name the real source and to state it is a live server, not an archive.Dependency
The
results/clusters/reference resolves once PR #3 (canonical cluster deposit) merges. Merge #3 first.Scope
Documentation only. No code or result changes.
🤖 Generated with Claude Code