Reconcile main with dev: adata-cli 0.5.0 - #10
Merged
Merged
Conversation
…empty groups are included in the subset output
…ve clarity and consistency
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Root attributes
The CLI could not read any file written by anndata >= 0.11. Since pandas' future.infer_string became the default, anndata writes obs/_index as a nullable-string-array *group* rather than a dataset, and axis_len required a dataset, so view, export dataframe and subset all failed outright on every recent file. HDF5 -> Zarr conversion was also broken on any store with string columns, and subset silently dropped raw/. Rename the project to adata-cli (package src/adata, command `adata`) and rename `info` to `view`; both old names remain as deprecating aliases through 0.x. Add src/adata/elements/ as the single home for on-disk format knowledge -- spec (encoding constants), strings (backend dtype reconciliation), read (every historical layout) and write (current spec only). The import, export and subset paths now share it instead of each reimplementing the format. Fixes: - resolve_index/element_len accept a group-valued index, so modern files read - create_dataset resolves text to variable-length UTF-8 per backend, so all four HDF5/Zarr pairings convert with categoricals and nullable dtypes intact - subset_axis_group narrows group-valued columns instead of copying them whole, which had been producing stores with mismatched column lengths - raw/ is subset against its own var axis; unrecognised top-level keys are copied with a warning rather than dropped - every written element is tagged; None round-trips via anndata's `null` encoding instead of an invented marker attribute - categoricals survive a CSV round-trip - sparse subsetting streams in blocks (byte-exact against scipy, CSR and CSC) - get_entry_type dispatches on encoding-type before falling back to structure - column-order is honoured on export New: `ls` for any HDF5/Zarr store including .loom; dataframe export from any path; results on stdout with status on stderr; npy export to stdout. tests/test_anndata_roundtrip.py has anndata write the fixtures and read the results back, with OldFormatWarning promoted to an error. The CI matrix runs tests/ on 3.12 and 3.13 rather than naming files individually, which is why test_storage_root_attrs.py had never run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four commands, all built on the elements layer added in the previous change rather than each reimplementing the format. `create` writes a valid empty store -- root attributes, obs/var frames with real indices, tagged mapping groups -- so `import` has something to fill in. `import` loses its obs/var restriction and gains an `image` subcommand, so a whole object can be assembled from the command line. `split` writes one store per distinct value of an annotation column, with label sanitisation, collision suffixes and a CSV manifest following the conventions of cellgeni/scraft's split_h5ad, but streaming rather than reading the source into memory. Closes #2. `subset` gains --obs-query/--var-query over a small predicate language (==, !=, <, <=, >, >=, in, not in, and, or, not, parentheses) evaluated chunk-wise. Only the columns a query mentions are read. This deliberately is not SQL: anything more involved is better served by exporting to CSV and using duckdb. A var query propagates to raw/ by name, so raw stays consistent. `concat` joins along obs with inner/outer alignment, --label/--keys/ --index-unique, and merge strategies for var and uns. Verified against anndata.concat: var order, obs order and X values agree exactly for both joins. obs columns keep their dtypes -- categoricals union their category sets and remap codes, nullable columns keep their masks, and a column missing from one input is padded rather than dropped. obsp/varp and raw are not carried over, as in anndata's own implementation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Prepares the first PyPI release and gives the project documentation that lives somewhere other than the repo root. Packaging: `Documentation` was "README.md", which is not a URL and PyPI rejects it. Adds license-files, Operating System and Typing classifiers, Python 3.13, py.typed, and a Changelog URL. `__version__` reads from installed metadata rather than being duplicated, and `adata --version` reports it. publish.yml publishes on a tag using PyPI trusted publishing, so there is no API token to hold. It reuses tests.yml via workflow_call, so a release cannot skip the suite, and refuses to publish when the tag does not match the version in pyproject.toml -- nothing enforced that before, and 0.3.2 was tagged against a tree declaring 0.3.1. A workflow_dispatch path targets TestPyPI first. Docs become a GitHub Pages site served from docs/ with plain Jekyll, so the files stay readable as markdown on github.com. Adds index.md, a full COMMANDS.md reference, and to each ELEMENTS doc a section on what this tool actually does with each element -- they were pure spec transcriptions saying nothing about the CLI. Documents the `null` encoding (absent from the upstream prose spec but written by anndata 0.12+) and, for Zarr, the v2/v3 differences that matter when copying between stores. csvkit is replaced by duckdb in the tutorial and the image. The tutorial now leads with `--obs-query`, which covers the example it used csvsql for, and keeps duckdb for filters that genuinely need SQL. The image drops the csvkit side-venv for a single static duckdb binary, and passes the --locked flag its comment already claimed. Verified: the wheel installs into a clean venv and runs; the image builds and the documented duckdb workflow runs inside it end to end. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The action posts its summary as a PR comment, which needs pull-requests: write. Without it the step 403s and fails the job even when every test passed, which is what happened on #6. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A PR stacked on another feature branch matched neither branch filter, so it reported no checks at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts: # .github/workflows/tests.yml
CSC matrices were silently dropped by concat. `kinds == {"csc_matrix"}` matched
no branch, so the code warned -- misreporting agreeing inputs as disagreeing --
and returned successfully, writing an output with no X at all. CSC is now
concatenated as CSC: for each target column, each input's row indices are
appended with that input's row offset added, which stays sorted without a
re-sort. Encodings are also checked before the output store is created, so a
genuine mismatch fails loudly instead of leaving a partial store.
Default concat keys came from filename stems, so inputs in different
directories sharing a name produced duplicate keys. --label writes those as
categorical categories, which must be unique, so the command reported success
while producing an output pandas could not read. Duplicates are now rejected,
for explicit --keys too.
`import image` replaced its destination unconditionally, so
`import image data.h5ad obs img.png` deleted the obs dataframe. Images are now
refused for paths that must hold a dataframe, and validated against the axis
for everything else.
`raw/var` was not recognised by validate_dimensions, so it could be replaced at
any length while raw/X kept its width. raw/var, raw/X and raw/varm/* are now
validated against raw's own var axis, taken from raw/X.
Under --uns-merge same/unique, non-dict groups were compared by their child key
names alone, so two dataframes with the same columns but different values
looked equal and the first input's was kept. Comparison now recurses into
contents and attributes, with a size cap above which an element is treated as
equal to nothing. Dict groups already recursed, which is why this only showed
up for dataframes, sparse matrices and categoricals.
A nullable var column surviving a merge was rendered as text, turning missing
values into empty strings and losing the numeric dtype. It now keeps its
values, mask, encoding and na-value.
Each fix has a regression test that fails without it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…#6 A nested string list such as [["a","b"],["c","d"]] was flattened into a length-4 vector on import, because the string branch of _write_json_list reshaped before writing. The shaped array is now passed through, and write_string_array no longer list()s an ndarray, which would flatten it again. Exporting a `null` element emitted its storage placeholder rather than None: the string "Empty(dtype=...)" on HDF5, and false on Zarr. _dataset_to_jsonable now recognises the encoding, so JSON payloads containing null round-trip exactly. Chasing those two through Zarr surfaced a larger one. anndata writes a consolidated metadata index at the root of a .zarr store. New members do land on disk, but every reader honouring that index -- anndata included -- keeps reading the stale snapshot, so an `--inplace` import into an anndata-written store reported success and then appeared to do nothing. Writable stores are now opened with use_consolidated=False and the index is rewritten on close. Only stores written by anndata were affected, which is why the existing Zarr tests, which build their own stores, never caught it. That in turn exposed subset_raw_group assuming `raw` is a group. anndata writes a null-encoded placeholder array there when an object has no raw, so on Zarr the subset crashed with "'Array' object has no attribute 'keys'". A non-group `raw` is now copied verbatim. Each fix has a regression test that fails without it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The suite checked this tool mostly against stores it built itself, which cannot catch what actually broke it twice: anndata changing how it writes a file. tests/test_anndata_versions.py now builds a reference store with each of anndata 0.8, 0.9, 0.10, 0.11, 0.12 and 0.13 -- each in its own uv-assembled environment with era-appropriate pins -- and checks that the CLI reads it and that what the CLI writes reopens in that same release. 204 cases across both formats, marked `integration` and run as a separate CI job. The pins are load-bearing: a modern pandas makes string columns a type the older releases cannot write, and pandas 1.x has no Python 3.12 wheels, so 0.8 and 0.9 build against 3.11. New unit modules cover the element layer against HDF5, Zarr v2 and Zarr v3 (test_elements.py), the storage layer including cross-version copying (test_storage.py), the file-format paths and their failure modes (test_formats.py), dimension validation for every axis-bearing path (test_validate.py), the command surfaces reached only indirectly (test_commands_coverage.py), and invariants that must hold for any data -- subsetting everything is the identity, split partitions exactly, concat undoes split (test_invariants.py). Fixed while writing them: Zarr v3 -> v2 copies failed because v3 codec objects were forwarded into a v2 array, which only accepts numcodecs -- the mirror of the filters bug fixed earlier. Codecs now only travel between stores of the same Zarr version. Also silenced zarr's warning about consolidated metadata at our own call site, since writing it is deliberate. Adds a shared multi-backend store fixture, replacing the hand-rolled builders new tests would otherwise have copied, and docs/TESTING.md. 741 unit tests plus 204 compatibility tests, up from 244. Coverage 85.5% -> 92.4%, with a 90% floor enforced in CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
publish-pypi depended on check-version, which only runs for tags. GitHub skips a job whose dependency was skipped, so a manual dispatch targeting PyPI would never have published anything. Both publish jobs now use always() with the success of build asserted explicitly, accepting check-version as either passed or not applicable. The Dockerfile sets ENTRYPOINT ["adata"], so the README and index examples were running `adata adata view ...`, which Typer rejects. Dropped the repeated executable; verified against the built image. COMMANDS.md listed `--zarr-format` among split's options, but the flag did not exist -- `adata split ... --zarr-format 2` failed with an unknown option. Implemented it rather than deleting the claim: split already inherited the source store's Zarr version, so only the override was missing. Adds tests/test_docs_are_accurate.py, which extracts every `adata ...` invocation from the README and docs and checks each command and option against the real --help. That alone would not have caught this one, since the claim lived in prose rather than a fenced example, so sentences naming an option and the commands offering it are checked too. Both checks fail if the flag is removed again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The docs test scraped `--help` output, which Rich wraps to the terminal width. It passed on a wide local terminal and failed in CI, where the width is narrower and long option names break across lines -- an assertion about rendered UI rather than about the CLI. Options now come from the Click command objects behind the Typer app, so the check is exact and width-independent. Verified passing at COLUMNS 40 through 200, and still failing at both widths when --zarr-format is removed from split again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ilt fixtures The cross-version codec fix was incomplete. _target_zarr_format defaulted an unspecified target to v3, and the subset paths never passed one, so `adata subset src.zarr -o out.zarr --zarr-format 2` on a v3 source forwarded v3-only options into a v2 array. It surfaced first through sharding rather than compressors: "Zarr format 2 arrays can only be created with shard_shape set to None". dataset_create_kwargs now takes the destination group and reads the version from it, so a call site that forgets cannot be wrong; an unknown target carries nothing version-specific rather than guessing. Fixing that exposed a second problem in the same area. Zarr requires a shard to be a whole number of chunks, so clamping a chunk to the subset size invalidates the source's shard geometry -- v3 to v3 then failed too. Chunk clamping is now one helper covering every dimension, and it drops sharding when it changes a chunk, since sharding is storage layout rather than data. The dense path had its own inline clamp that missed this; it now shares the helper. Separately, the compatibility suite turned any failed fixture build into a skip, so a broken release pin could have left that CI job green having checked nothing. A build failure now fails wherever the fixtures are required -- CI, or ADATA_REQUIRE_VERSION_FIXTURES=1 -- and stays a skip locally, where a broken environment should not block unrelated work. The job also asserts that a plausible number of cases were collected, so a marker or collection mistake cannot pass as success either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The guard added in the previous commit counted lines containing "::" in `pytest -q --collect-only` output, which prints "<Function ...>" instead -- so it always found zero and failed the job it was meant to protect. That is the same mistake as the help-scraping docs test: asserting on formatted output rather than on data. The check now parses the JUnit XML the run already produces and requires at least 150 executed cases, which is what "the job actually ran something" means. Verified both ways: 204 ran against a real run, and 0 when every case is skipped via ADATA_SKIP_VERSION_FIXTURES. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The short name was free when this was planned but has since been claimed by an unrelated "AData CLI" project, uploaded 2026-04-04. Publishing would have failed against a project we do not own, so the distribution is now `pyadata-cli`, matching the pending publisher configured on PyPI. Only the distribution name changes. The repository, the import package, the command and the documentation all stay `adata` / `adata-cli`, so the sole user-visible difference is `pip install pyadata-cli`. That mismatch is explained where the install is documented rather than left to surprise anyone. __version__ reads its metadata under the new name, so `adata --version` keeps working; verified by installing the built wheel into a clean environment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A CI run hung for the full 20-minute job timeout and was cancelled, reporting only that an orphan pytest process had been killed. The last test to complete was the one before the new split --zarr-format cases; py3.13 passed the same commit, and it does not reproduce locally, so it is timing-dependent. subset_h5ad wrapped its whole body in `console.status(...)` and then opened a `Progress` on the same console inside it. Rich allows one live display per console, and `split` repeats that pattern once per output group. The spinner is now stopped explicitly once the selection is known, before the progress bar starts, with a finally to guarantee it stops on any path. Whether or not that was the cause, a hang should not cost twenty minutes and leave no evidence. pytest-timeout is now a dev dependency with a 300s suite-wide limit, so a stuck test fails with a traceback pointing at it; the compatibility tests get 1800s, since assembling six environments on a cold uv cache is legitimately slow. Verified the timeout fires. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rename to adata-cli; restore AnnData format compatibility (0.4.0)
Add create, split, concat and query-based subsetting (0.5.0)
Docs site, PyPI packaging, duckdb (1.0.0 prep)
Test against six real anndata releases; coverage 85% → 92%
…with-dev # Conflicts: # Dockerfile
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Test Results (py3.13)825 tests 825 ✅ 1m 7s ⏱️ Results for commit 9baac85. |
Test Results (py3.12)825 tests 825 ✅ 1m 28s ⏱️ Results for commit 9baac85. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9baac85490
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+951
to
+952
| if name not in group or (where < 0).any(): | ||
| aligned.append(_MISSING) |
There was a problem hiding this comment.
When --join outer combines inputs with different var indices, every where containing an absent variable is treated as if the entire column were missing. For example, inputs indexed by [g1,g2] and [g2,g3] both produce a mapping containing -1, so even --merge first marks both copies as _MISSING and silently drops the var column. Reindex each present column onto the union with an appropriate missing representation instead of rejecting the whole column.
Useful? React with 👍 / 👎.
| for col in columns: | ||
| if col is None: | ||
| continue | ||
| for category in read_categories(col): |
There was a problem hiding this comment.
When concatenating a legacy Zarr dataframe whose categorical column is a code array and whose labels live under obs/__categories, this call lacks the parent dataframe group. read_categories can resolve that legacy layout only when parent_group is provided, so concat raises KeyError for these otherwise supported stores instead of combining the column.
Useful? React with 👍 / 👎.
| # half-written store behind. | ||
| check_matrix_encodings(roots, console) | ||
|
|
||
| with open_store(output, "w", zarr_format=zarr_format) as dst_store: |
There was a problem hiding this comment.
For .zarr output with no explicit --zarr-format, this passes None, causing zarr-python to create its default v3 store even when every input is v2. This silently upgrades v2 data and can make the result unreadable by older consumers, contrary to the source-version behavior documented in docs/COMMANDS.md:232-235; infer the format from the opened input stores before creating the destination.
Useful? React with 👍 / 👎.
Comment on lines
+912
to
+915
| if label: | ||
| if label in written: | ||
| raise ValueError( | ||
| f"--label {label!r} collides with an existing obs column." |
There was a problem hiding this comment.
When --label matches an obs column retained by the selected join, this error is raised only after the output store and its partial obs dataframe have already been written. The failed command therefore leaves an incomplete path behind, and the next corrected concat refuses to run because the output exists; perform this collision check before opening the destination or remove the partial output on failure.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings
mainup to date with everything merged intodev(#6, #7, #8, #9), and settles the one commitmainhad thatdevdid not.The divergence
maincarried32520d3— "Install duckdb-cli alongside csvkit in Dockerfile" — committed directly tomainand never merged back.devnever had it, so a plain merge would have reverted it.That commit's intent is preserved, by better means. It added
duckdb-clito a side virtualenv that existed to holdcsvkit. The Dockerfile ondevdropscsvkitentirely and installs duckdb as a single static binary with no venv, so it cannot conflict with the project's dependencies. The conflict was resolved in favour ofdev's version.Verified on the merged result:
What lands on main
The whole 0.5.0 line — see CHANGELOG.md. In short:
adata-cli; command isadata;infois nowview.h5adandinforemain as deprecating aliases until 1.0.0.ls,create,split(implementing a obs.col based splitting #2),concat, andsubset --obs-query.ls.Two things this unblocks
GitHub Pages. It is configured correctly (
main//docs, statusbuilt) but currently 404s, becausemain'sdocs/has noindex.md. Merging this brings https://cellgeni.github.io/adata-cli/ up.Releasing.
publish.ymlfires on a tag, runs the full suite, and refuses to publish unless the tag matchesversioninpyproject.toml— currently0.5.0. Note the distribution ispyadata-cli, notadata-cli: the short name was claimed on PyPI on 2026-04-04 by an unrelated project. The import package and command are unaffected.Before tagging
0.3.2currently points at a commit ondev, not onmain. Worth settling, sincepublish.ymlnow triggers on tags.pypienvironment is configured so far.workflow_dispatchwithtarget=testpypiis the rehearsal path.🤖 Generated with Claude Code