Transcript-flow alluvial + post-hoc endpoints view (neighboring-cell class) - #5
Open
atuldeshpande wants to merge 38 commits into
Open
Transcript-flow alluvial + post-hoc endpoints view (neighboring-cell class)#5atuldeshpande wants to merge 38 commits into
atuldeshpande wants to merge 38 commits into
Conversation
Adds the design for a Sankey/alluvial plot of per-transcript assignment flow through SEG (14 phases) and NOSEG (8 phases) pipelines. Specifies the 5-class vocabulary (collapsible to 3 for the headline view), the snapshot_phase logging hook (one int8 column per phase, ~140 MB at 10M tx), hook insertion sites per pipeline, conservation invariants, and the plot_transcript_flow API with plotly + matplotlib backends. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the all-14-phases default with three nested tiers: - collapsed (Tier A, 5 SEG nodes): phase1+rescue, group+rescue, stitch+demote+rescue, finalize. Display-time only; no new snapshots. - default (Tier B, 9 SEG / 8 NOSEG): phase1 collapsed, group + demote visible. Hooks run here unless overridden. - verbose (Tier C, 14 SEG): expands Phase-1 sub-mutations (prune, reassign_1c, split_p1, rerank, qc_p1, maha_remerge) — opt-in via snapshot_level="verbose". Updates §5.1 (phase-key constants + collapse maps), §5.2 (hook table with tier markers), §5.4 (cost: ~90 MB default / ~140 MB verbose at 10M tx), §6 (adds view parameter to plot_transcript_flow), §9 (scope recap), §10 (open Q on runner kwarg name). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add PHASE_DISPLAY_LABELS table mapping internal keys to user-facing Sankey node labels. Tier B 'phase1' renders as "Prune" since pruning is the headline Phase-1 operation. Tier C verbose 'prune' sub-step also renders as "Prune" — no collision since they never appear in the same view. All other phases title-cased for consistency. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Spec fix: remove references to a fictitious _pipeline_runner.py — this codebase has no central runner; each tutorial composes phases by hand. Hook sites now point at tutorials/lung_cancer/run_lung_cancer.py as the canonical SEG demo driver. Plan: 10 TDD tasks with frequent commits. 1. Classifier + display labels (sankey_log.py) 2. Snapshot round-trip + conservation tests 3. Tidy transition DataFrame data-prep (flow_plot.py) 4. View resolution + auto-pipeline detect 5. plot_transcript_flow + matplotlib backend 6. Plotly backend (lazy import) 7. Wire exports in tracer.__init__ 8. Snapshot calls in lung_cancer SEG driver 9. Integration smoke + plot artifact 10. plotly as optional [viz] extra in pyproject.toml Self-review: spec coverage verified, no placeholders, types consistent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds src/tracer/sankey_log.py with:
- _classify_etype_vec (int8 codes: main/partial/component/unassigned/dropped)
- snapshot_phase(df, phase, *, id_col) hook
- PHASE_KEYS_{SEG,NOSEG}_{DEFAULT,VERBOSE,COLLAPSED} tier tables
- COLLAPSE_{SEG,NOSEG} display-time group maps
- PHASE_DISPLAY_LABELS (phase1 → 'Prune')
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds src/tracer/flow_plot.py with _prepare_flow_data and _collapse_classes. Tidy schema: (phase_from, phase_to, class_from, class_to, n). Supports 5↔3 class grouping, min_flow_frac filtering, strict conservation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
_resolve_view picks a phase list given: - pipeline (seg/noseg/auto — auto sniffs which etype_at_* cols are present) - view (default/collapsed/verbose) collapsed view maps the display-label group to its end-of-group source column. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the public plot_transcript_flow function. Matplotlib backend draws flow polygons via PathPatch — no interactivity but works headless and in any matplotlib env. Default palette covers 5 + 3 class groupings. drop_unchanged collapses identity-only phase boundaries. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
backend='plotly' (default) renders an interactive go.Sankey with per-link hover (count + percent). plotly imported lazily inside _render_plotly; flow_plot module import has no plotly requirement. HTML output via fig.write_html. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds snapshot_phase calls after each Stage 1/2/3/5 boundary in the canonical SEG demo driver, with id_col tracking the active assignment column at each stage. Drives the transcript-flow Sankey artifact. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Synthetic 9-phase SEG snapshot df → both backends render without error, artifacts save to disk, plotly file contains a Sankey trace, class grouping three/five produce sensible node counts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PDAC 500µm ROI verification (2026-06-09) showed the modern SEG pipeline does not emit 'component' (absorbed by Stitch + Final-Rescue) and does not emit 'dropped' at finalize (leftover -1 routed into nearest entity, not DROP sentinel). The five-class vocabulary is retained for backward- compat with saved runs and NOSEG cascade; empty buckets render as zero-height bands. The default 3-class grouping makes this visually invisible. Documented in §4. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
UX changes:
- Tier A node labels renamed from compound names ('Prune + Rescue',
'Group + Rescue', 'Stitch + Demote + Rescue', 'Cascade + Rescue') to
the principal earlier-stage name ('Prune', 'Group', 'Stitch',
'Cascade'). The post-stage rescues/demotes are folded silently.
- New display_label_for(phase_key, *, pipeline, view) helper inverts
COLLAPSE_{SEG,NOSEG} so Tier A renders the right collapsed label
when reading a source-column snapshot.
- Phase labels moved from in-situ text annotations to proper axis labels
(matplotlib xticks rotated 20°; plotly annotations along the top).
- Class legend added below the X axis (matplotlib: Patch handles,
ncol=len(classes); plotly: invisible scatter traces with horizontal
legend orientation).
- Plotly Sankey gets explicit node.x/node.y pinning + arrangement='snap'
so column-header annotations align with the actual node positions.
Layout robustness fix:
- _layout_nodes_mpl now computes per-phase totals from MAX(incoming,
outgoing) edges. Caught on the real PDAC 500µm ROI: when
drop_unchanged removes an identity-only boundary, the downstream
phase had only outgoing edges in tidy; the old layout looked at
incoming-only and produced an empty node set, then _draw_ribbons_mpl
raised 'ribbon source node not in layout' (the defensive raise from
the Task-5 review). Fix: layout uses max per-class so a phase with
flow on either side renders correctly.
All 44 tests pass + 1 skip; validated end-to-end on PDAC 500µm
(204,931 transcripts).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
By default, plot_transcript_flow stacks classes top-to-bottom by ascending class code — for the canonical 5 codes this matches the semantic order (main → partial → component → unassigned → dropped). Extended palettes that add new codes (e.g. main_neighbor=5) used to land at the bottom, visually adjacent to dropped instead of their semantic sibling main. Adds class_order: Optional[Sequence[int]] (None = current behavior). User supplies a top-to-bottom order; palette codes not listed are appended at the bottom in sorted order so they're never silently dropped. Unknown codes raise ValueError. Threaded through both _render_matplotlib and _render_plotly via a single _resolve_class_order helper. 4 new tests in TestResolveClassOrder. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The default top-to-bottom visual order is now driven by the explicit CLASS_SEMANTIC_ORDER list in sankey_log (main → partial → component → unassigned → dropped), not by ascending int-code coincidence. Identical behavior for the canonical 5 codes; extended palette codes still get appended at the bottom by default (callers slot them next to their semantic sibling via class_order). Renames test_default_is_sorted_by_code → test_default_follows_canonical_ semantic_order, adds test_default_matches_semantic_order_for_full_5. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both backends previously rendered a legend entry per palette code, which produced dead swatches for codes the data never emitted (e.g. component and dropped on modern SEG runs). Now the legend filters classes to those that actually appear in the tidy transition df. Zero-count classes remain in the palette (so an explicit class_order keeps working) but don't clutter the legend. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The plotly backend pinned both node.x and node.y for every (phase, class) cell in the grid — including zero-count cells. With arrangement='snap', that forced plotly to allocate vertical slots for empty bands and broke the link-width allocator's ability to honor flow conservation: large ribbons (e.g. main → unassigned at Phase 1) would visually disappear because the source band's apparent height didn't match its actual mass. Fix: keep node.x (so columns stay aligned with the column-header annotations) but drop node.y. Plotly auto-balances y within each column based on incoming + outgoing flow. The matplotlib backend already handled this correctly via _layout_nodes_mpl's max(in, out) totals. Caught on real PDAC 500µm data — input column's 176k main mass wasn't visually flowing into the Phase 1 unassigned band. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Within each source node, outgoing ribbons were iterated in ascending target-class CODE order — so a ribbon going to a top-of-column target (e.g. neighboring_cell, code 5 in an extended palette ranked first in class_order) would emerge from the bottom of the source. The Sankey convention is to order outgoing ribbons by the target's vertical-position rank, so top-of-column targets emerge from the top of their source. Fix: in _draw_ribbons_mpl, sort the per-boundary ribbons by (class_order rank of class_from, class_order rank of class_to) before the stacking loop. This preserves vertical-order at both source and target ends. Also reverts the plotly node.y mass-weighted pinning (broke HTML on real data) — plotly stays at x-pinned + y-auto-balanced for now. matplotlib backend respects class_order; plotly auto-layout may not honor it. Tradeoff documented in the source comment. Caught on real PDAC 500µm: ribbon unassigned → neighboring cell at post-rescue boundaries was emerging from the bottom of unassigned and bending across the column to reach neighboring cell at the top. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The classifier's _UNASSIGNED_SENTINELS was just {"-1"}, missing the
canonical post-finalize tokens. After finalize_unassigned normalizes
the leftover -1 tx to "UNASSIGNED" (modern SEG default), those rows
fell through to the default CLASS_PARTIAL fill — so the Sankey showed
0 unassigned at finalize and an erroneous bump in the partial count.
Fix: align _UNASSIGNED_SENTINELS with the canonical
tracer.spatial.UNASSIGNED_LABELS set so the classifier recognizes:
- "-1" (pre-finalize sentinel)
- "UNASSIGNED" (post-finalize canonical token)
- "nan" (legacy null token)
- "prune_rejected" (mid-pipeline stage rejection)
- "group_rejected" (mid-pipeline stage rejection)
DROP and demote_rejected stay in _DROPPED_SENTINELS for backward
compat with older runs that emitted them.
Also adds label_target='ribbons'|'both' kwarg to plot_transcript_flow
(matplotlib): puts stage labels above the ribbons at midpoints between
columns, matching the Sankey semantic (columns = states, ribbons =
stages). Default 'columns' preserves existing behavior.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The runner's Finalize step is just defensive normalization (collapses
lingering -1 / DROP / demote_rejected / UNASSIGNED_* into the canonical
"UNASSIGNED" token). After the classifier fix to recognize "UNASSIGNED"
as unassigned, the final_rescue and finalize snapshots are bit-identical
at the classifier level on modern SEG. The redundant column adds no
information and elongates the X axis.
Changes:
- PHASE_KEYS_{SEG,NOSEG}_DEFAULT drop "finalize" — default tier ends
at "final_rescue", which is the effective pipeline output
- PHASE_KEYS_{SEG,NOSEG}_COLLAPSED drop "finalize" — Tier A also ends
at final_rescue
- PHASE_KEYS_NOSEG_VERBOSE = NOSEG_DEFAULT + ["finalize"] — verbose tier
keeps finalize for explicit inspection of the no-op step
- New _PHASE_COLUMN_OVERRIDES dict + display_label_for honors it: the
"final_rescue" column reads "Finalize" (state name), while the ribbon
entering it still reads "Final Rescue" (the action stage name)
Updated 4 tests (default-tier lengths, collapsed-tier source columns,
new test for verbose preserving finalize, new test for column override).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hover tooltip percentages were computed as 100 * n / tidy['n'].sum(), but tidy['n'].sum() sums ribbon mass across every phase boundary — since transcripts are conserved across boundaries, a tx is counted n_boundaries times. On the PDAC 500µm run (8 phases / 7 boundaries), percentages came out ~7x too small (a true 50% ribbon read as ~7%). Fix: use the total mass in any single column as the denominator (tidy[phase_from == phases[0]]['n'].sum()) — this is the actual transcript count, conserved across columns. Matches what the matplotlib backend already does via _layout_nodes_mpl. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-locates two end-to-end drivers in tutorials/pdac_io/ next to the existing bootstrap-PMI CSVs in data/: - run_pdac_full_pipeline.py: vanilla SEG sequential run on the full sample (adapted from benchmarks/bench_pdac_full_seq.py). Writes partition_sequential.parquet + summary.json to output/full_seq/. - run_pdac_full_sankey.py: same pipeline + monkey-patched _record_stage for per-phase snapshot_phase calls + Tier A/B Sankey rendering. Writes partition (incl. all etype_at_<phase> snapshot cols) + summary.json + phase_counts.csv + Tier A/B HTML+PNG to output/full_sankey/. Both default to the locally-saved pmi_bs_pdac_io_C_5_95.csv panel (override via PANEL_CSV env var). Configurable runner knobs via PMI_THR / RESCUE_MEAN_ADMIT / RESCUE_AGGREGATOR_PERCENTILE. Expected scale: 3-5M tx, 30 min - 2 h wall, 20+ GB RSS peak — launch with nohup. Output dir auto-created. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two coupled pieces on top of feature/transcript-flow-sankey: - view="endpoints": plot only initial/final proportions ([first,last] phases) - promote the runner's neighboring-cell prototype into core (sankey_log), classifying cell-etype tx whose assigned cell_id differs from origin as a first-class 'neighboring cell' (code 5) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r class Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
plot_endpoints_flow computes initial/final classes from any TRACER partition's origin cell_id + final label; neighboring cell = whole-cell final base cell_id != origin. Drops the snapshot_phase wiring and runner monkey-patch dependency. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t class_order contract)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the transcript-flow alluvial/Sankey visualization plus a new post-hoc endpoints view that plots only initial→final transcript proportions, and surfaces a first-class "neighboring cell" entity class.
This branch bundles the underlying
feature/transcript-flow-sankeyinfrastructure (not previously on upstream) with the endpoints feature on top.Endpoints feature (the new work, on top of the Sankey base)
sankey_log.classify_endpoints(df, *, orig_id_col, label_col, etype_col=None)— computes(initial, final)entity-class codes from any TRACER partition's originalcell_id+ final assignment label. Pure post-hoc — no pipeline snapshots required.flow_plot.plot_endpoints_flow(df, *, orig_id_col="cell_id", label_col=None, ...)— public two-column alluvial (Initial → Final) with tracked crossing ribbons.CLASS_MAIN_NEIGHBOR(code 5, "neighboring cell") — a transcript assigned to a different basecell_idthan its origin. Rules: "etype wins" (-tr-partials stay partial); originally-unassigned tx are not neighbors; the 3-class grouping folds neighbor back into original.phase_labelsoverride, lone-boundarydrop_unchangedguard, colorblind-safe (Okabe-Ito) neighbor color.Design + plan:
docs/superpowers/specs/2026-06-17-endpoints-alluvial-view-design.md,docs/superpowers/plans/2026-06-17-endpoints-alluvial-view.md.Test Plan
pytest tests/test_flow_plot.py tests/test_sankey_log.py tests/test_sankey_integration.py→ 76 passed, 1 skippedpdac_full_endpoints.htmlon real outputtests/test_etype.pyhas 9 pre-existing failures from missingtorch_geometric/ unbuilt_cy_prune(environmental, unrelated to this PR)🤖 Generated with Claude Code