epic: Allele centric mapping, storage, and service#791
Draft
bencap wants to merge 93 commits into
Draft
Conversation
- add shared seqrepo and seqrepo data proxy providers in services - reuse shared seqrepo provider in deps to remove duplicate config - align vrs sequence proxy behavior with dcd-mapping expectations
- add shared helpers to translate hgvs, normalize alleles, and compute ids - clear cached merkle digests before identification so mutated alleles do not keep stale ga4gh identifiers - document the invariant that all allele identification must route through the helper to prevent digest correctness regressions
Introduce a reusable declarative mixin implementing transaction-time SCD Type 2 versioning for rows that change over time, used by either versioned entities or link/association rows. - A row is live while valid_to is NULL; current/as_of express the half-open [valid_from, valid_to) point-in-time predicate so call sites never hand-roll it - supersede_with (single row) and supersede_live_where (bulk) stamp the retired valid_to and successor valid_from with one timestamp for a gap-free handoff regardless of transaction boundaries - retire/retire_live_where are withdrawal primitives; retire cascades to live child links named in __retire_cascade__ - bulk supersede refuses to run on cascade-bearing classes since it cannot fire the cascade - consumers must add a partial unique index over their natural key WHERE valid_to IS NULL as a backstop against duplicate live rows Cover the mixin with unit tests against purpose-built models, using explicit timestamps far from the transaction clock to prove the gap-free handoff and verify the partial unique index backstop.
Add a lib module with reusable helpers for parsing and rewriting HGVS strings ahead of VRS translation. - extract_accession returns the reference accession (substring before the first colon), tolerant of whitespace and missing separators - split_cis_phased_hgvs expands a bracketed multivariant expression into fully-qualified components carrying the original accession and coordinate prefix, since ga4gh's AlleleTranslator only yields a single Allele per call and each component must translate alone - join_cis_phased_hgvs is the inverse, recombining components into one bracketed block and returning None when they do not share a single accession and coordinate prefix Cover the helpers with unit tests including the split/join round trip and the mixed-accession and mixed-prefix rejection cases.
Add VRS translation support for cis-phased multivariant HGVS, building on the new hgvs split helpers. - translate_hgvs_to_variation translates each component HGVS to an Allele independently, returning a bare Allele for a single component and wrapping two or more in a CisPhasedBlock, mirroring dcd_mapping's vrs_map._construct_vrs_allele; the reverse-translation job emits bracketed genomic forms the AlleleTranslator cannot translate directly - identify_variation generalizes identify_allele to blocks, clearing every member and location digest plus the block's own before identifying so a stale member digest never propagates into the block id; the block digest is order-independent so a set dedups to one row Cover both with unit tests, including the order-independent block digest and the stale-digest clearing.
get_hgvs_from_post_mapped can now join the members of a multi-variant block (Haplotype/CisPhasedBlock) into a single bracketed cis-phased expression via the new combine_cis flag, replacing the previous behavior of returning None for any multi-variant block. - combine_cis defaults off because some consumers cannot yet handle a bracketed expression — notably ClinGen submission, which has no single CAID for a multi-variant cis block (#764) - the CSV export fallback opts in, so g./p. HGVS columns are populated from cis-phased post-mapped output instead of left empty - drop the stale commented-out error branches and the resolved TODO
Introduce parallel mapping tables for the Better Reverse Translation epic (#746). The existing mapped_variants table is left untouched (frozen serving) while the new schema is built out separately. - add alleles, mapping_records, and mapping_record_alleles tables with their ORM models; alleles are content-addressed by vrs_digest and shared across mapping records via the link table - mapping_record_alleles.is_authoritative distinguishes the assay's actual measurement from translator-derived links, since the same VRS allele can be authoritative for one record and derived for another - put mapping_records and mapping_record_alleles on valid-time versioning (ValidTime mixin): a re-map retires the prior live row by closing valid_to instead of deleting, so history is retained and point-in-time queries are a single predicate; partial unique indexes promote "one live row per key" to the database - derive Allele.transcript and MappingRecord.transcript as hybrid properties from the HGVS columns rather than storing them, so they cannot drift; drop the stored alleles.transcript column - add the cross_level_translation annotation type, written once per variant to record whether filling unmapped levels succeeded, was skipped, or failed - annotate Variant and TargetGeneMapping with mapping_records relationships and typed primary keys
Add type stubs for the VRS and hgvs APIs used by the reverse translation work so callers can be type-checked without casts. - ga4gh.core: type ga4gh_identify and re-export PrevVrsVersion - ga4gh.vrs: stub the data proxy, AlleleTranslator, and normalize; translate_from returns Any so callers annotate the concrete VRS subtype they expect without a redundant cast - hgvs.assemblymapper: stub AssemblyMapper with the g_to_t/c_to_g/c_to_p conversions used for cross-level translation
Wire the variant-annotation package into the server extra as a local editable (PEP 660) dependency so the API can drive the reverse translation pipeline. - declare variant-annotation as a develop path dependency in pyproject.toml and the server extra; lock pulls in its transitive deps (variant-translation, biopython, openpyxl, et-xmlfile) - mount ../variant-annotation into the dev and worker containers and prepend it to PYTHONPATH so the editable install resolves - pass --no-directory to poetry install in the Dockerfile so the build does not try to install the not-yet-copied editable sibling - ignore_missing_imports for variant_annotation.* in mypy: its editable import hook is unfollowable and it ships no py.typed, so treat it as untyped rather than maintaining drift-prone local stubs - add a Makefile with dev/test/lint/format targets
Rework the variant mapping job to persist its results into the new MappingRecord / Allele / MappingRecordAllele schema instead of MappedVariant, building on the closure tables for the Better Reverse Translation epic. - create one MappingRecord per variant (including failed variants, with null VRS data); successfully post-mapped variants additionally get-or-create an authoritative Allele and link it via an is_authoritative MappingRecordAllele - supersede a variant's prior live record through ValidTime supersede_with — retiring it and its allele links and inserting the new record under one timestamp — instead of flipping a current flag, so the old/new handoff is gap-free - require a TargetGeneMapping for every mapped score, raising on a miss rather than tolerating a null link, since dcd-mapping guarantees one - combine cis-phased members when deriving hgvs_assay_level - update the mapping job tests to assert against MappingRecord and the authoritative-allele link, including the gap-free supersession and cascade-retire of the prior link TODO#765: mapping is not idempotent, so each run always creates a new authoritative link.
Add a seqrepo_data_proxy to the worker context in both the standalone context and the on_job_start hook, so jobs performing VRS translation have a SeqRepo-backed data proxy available alongside the cdot HGVS data provider. Mirror it in the mock worker context used by tests.
Add a reverse_translate_variants_for_score_set worker job that builds the cross-level HGVS equivalence class for every mapped variant in a score set, persisting the candidates as non-authoritative alleles. - for each current authoritative MappingRecord, collapse the assay-level HGVS to its protein consequence and expand to all coding/genomic candidates via a single batched construct_equivalent_variants call from the variant-annotation library, run off-loop in a thread - resolve each record's coding (NM_) transcript from its target gene's cdna alignment, falling back to a batched UTA NP_→NM_ lookup for protein-level mappings; records with no coding transcript are skipped - translate each candidate to VRS and write it as a get-or-created Allele linked non-authoritatively to the record, deduping by vrs_digest and never relinking the record's authoritative allele - supersede the prior live derived links with the new set in one gap-free operation via ValidTime.supersede_live_where - record per-variant cross_level_translation annotation status (success / failed / skipped), retaining per-candidate translation errors as metadata; the job fails only when every variant fails - add WorkerCoordinateTranslator and NullTranscriptSource adapters for the library's translation ports, deferring AssemblyMapper init so its network calls do not fire under mocked unit tests - get_or_create_allele helper, job registration, and a pipeline dependency placing reverse translation after mapping and before CAR submission TODO#765: a re-run supersedes the whole derived set wholesale because re-mapping re-mints the records; idempotent records would let unchanged derived links stay live.
…n date Previously the cdna-transcript lookup was keyed by target_gene_id alone, which meant a re-mapped score set could bind a stale NM_ transcript from an earlier run rather than the one the current run emitted. - Key cdna_transcript_by_run on (target_gene_id, mapped_date); within a key the highest-id row wins, so a same-run replacement takes precedence. - Carry mapped_date through the MappingRecord query so each record anchors to its own run's cdna row. - Add _TranscriptResolutionSkipReason to distinguish recoverable skips (protein-coding target, transcript unresolved) from correct skips (non-coding/regulatory target, no protein consequence). Emit skip_category in annotation_metadata. - Add target_gene_id to _TranscriptResolution so skip classification can look up the target's TargetCategory. - Add Mapped[] type annotations to TargetGene.id and .category columns. - New tests: genomic-accession coding target RT, latest-cdna-row-within- run selection, stale-cdna-row isolation, skip classification (coding recoverable vs. regulatory correct), and cdna TGM persistence in the mapping job. Refs mavedb-api#763
Forward translation emits predicted protein consequences in parentheses (e.g. p.(Ala222Val)). The parens denote inference, not a distinct variant form, so normalize to the bare p. form before storing. Strings without prediction parens are returned unchanged. Includes parametrized unit tests.
…st collisions - Re-identify each component via normalize_and_identify after translate_from so the stored vrs_digest reflects current content, not a stale value cached by the reused AlleleTranslator's Merkle tree. Without this, distinct biological variants at the same position (e.g. A>C and A>T) share one digest and are merged by the digest-keyed get_or_create_allele. - Use ga4gh_identify with in_place="always" in identify_allele and identify_variation so an allele that already carries a translator-stamped id has it recomputed, not retained. - Coerce ReferenceLengthExpression -> LiteralSequenceExpression in normalize_and_identify, mirroring dcd_mapping's _rle_to_lse, so RT-built alleles hash identically to the mapper's authoritative alleles and dedup correctly across sources. - Add regression tests: stale-id overwrite, distinct-alt digest isolation, cis-phased ordering canonicalization, and RLE->LSE coercion.
…tighten comments - Emit the protein consequence (result.hgvs_p) as a protein-level member of the equivalence set alongside the coding/genomic candidates. Prediction parens (p.(Ala222Val)) are stripped via strip_protein_prediction_parens before translation and storage. Protein-assay inputs are excluded (the protein is already the authoritative allele). - Trim all inline and docstring comments to essential why; remove redundant prose throughout reverse_translation.py. - Add tests: protein allele persistence, skip-classification for all three skip categories (no_assay_level_hgvs, transcript_unresolved, no_coding_transcript).
…rom failures Previously, variant mapping success was determined solely by the presence of pre/post-mapped alleles. This conflated genuinely failed mappings with benign absences (intronic variants, no-protein-consequence variants) that legitimately produce no allele. - Add `MappingOutcome` enum to `lib/mapping/schema.py` mirroring `dcd_mapping.schemas.MappingOutcome`, with an `is_benign_absence` helper to classify INTRONIC and NO_PROTEIN_CONSEQUENCE outcomes - Rewrite the per-record outcome logic in `map_variants_for_score_set` to branch on the typed outcome rather than allele presence: MAPPED -> SUCCESS, benign absence -> SKIPPED, FAILED -> FAILED - Replace the `successful_mapped_variants` scalar with a typed `Counter[MappingOutcome]` that feeds `mapped_count`, `failed_count`, and `skipped_count` tallies in logs and final state decision - Derive `mapping_state` from genuine failures only; all-benign result sets are treated as complete, not failed - Raise `NonexistentMappingResultsError` when a score annotation has no `outcome` field (malformed/older payload) - Expand test coverage for intronic and no-protein-consequence scenarios, verifying correct status and failure-category assignment
… UTA-backed source WtCodonMode.ALL reads the reference codon via TranscriptSource.codon_at, which requires a real UTA connection. The previous NullTranscriptSource always returned None, silently breaking WT-codon resolution. - Extract `uta_transcript_source()` context manager into `translation_ports.py`; removes the ad-hoc UTA connection setup that was duplicated in the job and the now-deleted NullTranscriptSource - Scope the live UTA client around the full `run_in_executor` call so the connection outlives the synchronous executor block - Remove NullTranscriptSource; callers that only need transcript_for_protein (already resolved by the job) also benefit from the real client without extra cost - Add a TODO noting that non-substitution consequences (del/ins/delins/ fs/ext) are miscounted as FAILED rather than SKIPPED (#767)
…ndle null gracefully - `translate_hgvs_to_variation` now attaches an `Expression` to each allele produced from a cis-phased or single HGVS string, mirroring the dcd_mapping authoritative-allele convention so `post_mapped` is self-describing without a separate round-trip. - `hgvs_from_vrs_allele` returns `None` instead of crashing when `expressions` is null or empty (valid for cis-phased block members), and `get_hgvs_from_post_mapped` propagates that as a `None` result. - `post_mapped` is now serialized with `exclude_none=True`, matching the mapper's output format. - Minor formatting clean-ups in reverse_translation.py (no logic change).
… model - `submit_score_set_mappings_to_car` now operates on Allele rows (authoritative + RT-derived) rather than MappedVariant, deduplicating by allele_id so each VRS allele is registered exactly once regardless of how many variants share it. Adds `force_reregister` param and per-allele outcome counters. - `submit_score_set_mappings_to_ldh` queries MappingRecord + Allele for pre/post-mapped data instead of the deprecated MappedVariant join. - `construct_ldh_submission_entity` signature updated to accept MappingRecord and Allele separately, since those fields now live on different models. - `warm_clingen_cache` switched to the shared `get_alleles_for_score_set` helper to keep allele scope consistent across all three jobs. - Extracts `get_alleles_for_score_set` and `ScoreSetAlleleRow` into `lib/clingen/alleles.py` as the single canonical query for both CAR and cache jobs.
get_hgvs_from_post_mapped can now join the members of a multi-variant block (Haplotype/CisPhasedBlock) into a single bracketed cis-phased expression via the new combine_cis flag, replacing the previous behavior of returning None for any multi-variant block. - combine_cis defaults off because some consumers cannot yet handle a bracketed expression — notably ClinGen submission, which has no single CAID for a multi-variant cis block (#764) - the CSV export fallback opts in, so g./p. HGVS columns are populated from cis-phased post-mapped output instead of left empty - drop the stale commented-out error branches and the resolved TODO
…-keyed refresh Replace MappedVariant-based gnomAD linkage with valid-time GnomadAlleleLink rows keyed on the deduplicated Allele. Linking covers every current allele of a score set (authoritative and RT-derived), so protein/coding score sets — whose genomic allele is RT-derived — are no longer dropped; per-variant annotation status flows through the _annotate_gnomad choke point against authoritative links only (interim bandaid, the AnnotationEvent migration seam). Refresh / idempotency model: - One live link per allele (unique index on allele_id); a gnomAD version bump supersedes rather than accumulating one live link per version. - Supersede only on change — an unchanged re-run leaves the live link untouched, so the valid-time history records no spurious boundary. - Version-keyed skip avoids re-fetching alleles already current at the version; a force param bypasses the skip (re-ingestion / heal) without churning unchanged links. - Per-variant status is a per-run audit event: created / preexisting / skipped. Normalize CAIDs across the Athena join: the gnomAD Hail dump drops leading zeros (CA025094 -> CA25094), so exact-string matching silently missed zero-padded CAIDs. Extract group_alleles_for_annotation as the shared allele-grouping primitive for allele-subject annotation jobs (adopted by gnomAD; VEP/ClinVar to follow). Refs #742. Partially addresses #722 (CAID-completeness tracking there stays open).
Migrate the VEP functional-consequence job (Step 2 of the annotation infrastructure migration) off MappedVariant onto the deduplicated allele model. - New ValidTime vep_allele_consequences table: a single allele-keyed row collapses record + link (the consequence is a scalar with no shared external entity, unlike gnomAD/ClinVar). One live consequence per allele via a partial unique index. - Job runs over the score set's full allele set (authoritative + RT-derived) via get_alleles_for_score_set + the shared grouping primitive; VAS still fans only to authoritative variants (the bandaid seam). - Version-key the refresh on the Ensembl release (/info/software): skip alleles already current, abort if the release can't be fetched. Supersede is value-keyed, not version-keyed — an unchanged consequence at a new release bumps source_version/access_date in place to avoid churning history. force bypasses the skip (e.g. after editing VEP_CONSEQUENCES). - A no-result is treated as a non-answer, never a negative: held consequences are not retired on an empty/failed VEP fetch. - Annotation links stay one-directional to Allele (no reverse back-ref). Adds lib + job tests covering linkage, RT-derived scope, version skip, in-place bump, supersede-on-change, no-result handling, and release-fetch failure.
Step 3 of the #742 external-annotation migration. ClinVar linkage moves off MappedVariant onto the deduplicated allele model. - Rename clinical_controls -> clinvar_controls (ORM model + table + unique constraint). Internal only: the ClinicalControl* view models, the /clinical-controls serving endpoints, and the frozen mapped_variants_clinical_controls association are unchanged, since the view-model name is both the OpenAPI schema name and the record_type discriminator consumed by the UI. - New clinvar_allele_links ValidTime table + ClinvarAlleleLink model. Multi-live: partial unique index (allele_id, clinvar_control_id) WHERE valid_to IS NULL, so an allele accumulates one live link per release. - Refactor refresh_clinvar_controls onto get_alleles_for_score_set + group_alleles_for_annotation (payload = CAID, full allele scope). Links are get-or-create; a same-version re-resolution to a different control supersedes newest-wins (gap-free retire+insert) rather than leaving two live links. VAS writes funnel through the _annotate_clinvar choke point, fanned only to authoritative_variant_ids and version-scoped. - Additively capture ClinVar's VariationID: nullable clinvar_variation_id column populated forward from the variant_summary TSV (parse degrades to None on archival schemas lacking the column). Unserved; the dedicated clinvar_variants remodel is deferred to the read-cutover. Tests rewritten for the allele model, covering the multi-live link writes, the version-scoped supersede guard, and the authoritative-only VAS fan-out.
Steps 4 & 5 of the #742 migration. Both jobs are redundant under the allele model: Allele.hgvs_g/c/p are populated by the mapping job, and the reverse-translation equivalence space (genomic/coding/protein alleles linked per MappingRecord) replaces the ClinGen PA<->CA translation table. - Remove populate_hgvs_for_score_set and populate_variant_translations_for_score_set from the pipeline DAG (both were leaf nodes — no dependency edges to repair), the worker registry (BACKGROUND_FUNCTIONS + STANDALONE_JOB_DEFINITIONS), and the external_services package exports. - Delete the two job modules and the now-orphaned ClinGen HGVS helpers (extract_hgvs_from_ca_allele_data / extract_hgvs_from_pa_allele_data), used only by the HGVS job. - Keep lib/variant_translations.py and the variant_translations table/model, marked FROZEN (serving-only) — they back old-model serving and are dropped at read-cutover. - Delete the obsolete job tests and their conftest fixtures.
get_allele_translations(db, allele_id, *, as_of=None) returns an allele's full cross-layer equivalence set (genomic/coding/protein) by traversing the MappingRecordAllele link graph: allele -> its live links -> mapping record(s) -> all co-linked alleles. The relation is co-membership in a MappingRecord's allele set, not a shared identifier — ClinGen's CAID spans only the nucleotide layers (the protein allele carries a distinct PA), so the link graph is the only thing tying all layers together. This replaces what the retired variant_translations PA<->CA table provided. Forward-compatible with temporal reads: the same half-open valid-time predicate is applied at both the anchor and fan-out hops, so passing as_of reconstructs the equivalence set as of any instant. Defaults to the currently-live set.
…bulary Introduces the AnnotationEvent log: a single append-only event table whose subjects are Variant and Allele, selected by annotation_type via a polymorphic CHECK. "Current" is derived (DISTINCT ON … id DESC), never stored. Adds the shared Disposition (present/absent/not_applicable/failed) and EventReason vocabulary, the v_current_annotation_events view, and the supporting migrations.
Add a nullable projection_group column to mapping_record_alleles and thread the coding/genomic pairing it records through the reverse- translation worker and the serving layer. - migration e7b2c9a1f4d3: add nullable projection_group (no index, no backfill; existing rows stay NULL and re-group on next re-map) - RT worker: consume the library's ProjectionPair list, stamp both links of a pair with a shared per-record group id, leave the protein apex ungrouped, and fold a pair member that equals the record's authoritative allele onto the existing link instead of duplicating it - lean variant view: replace assayLevelHgvs/proteinLevelHgvs with an assayLevel pointer plus a mapped MappedTriple (genomic/cdna/protein), filling the other nucleotide slot from the authoritative link's projection_group sibling via one indexed 1:1 join - variant detail: add derivation (authoritative/projection/candidate) and projection_of (the sibling's VRS digest) to AlleleIdentity Unpopulated projection_group (pre-RT data, protein apex) degrades gracefully to a null sibling / empty slot.
…emove legacy lookup endpoint
- Add `include_nucleotide_siblings` parameter to `get_allele_measurements`
and expose it as a query param on `GET /clingen-alleles/{id}/measurements`;
for a CA query, widens the equivalence class through the queried change's
protein consequence to surface sibling nt variants encoding the same
amino-acid change (relationship=nucleotide_encoding)
- Fix relationship labeling to key off measured-allele level rather than
entry level, correcting the sibling-nt branch so sibling nt records
get nucleotide_encoding instead of protein_consequence
- Remove `POST /variants/clingen-allele-id-lookups` and its associated
view models (ClingenAlleleIdVariantLookupsRequest, ClingenAlleleVariants,
ClingenAlleleIdVariantLookupResponse, VariantEffectMeasurementWithShortScoreSet)
This was referenced Jul 9, 2026
…pe assay_level
Rename the AnnotationLayer enum to SequenceLevel to reflect that the
same closed set (protein/cdna/genomic) serves several duties across the
schema — the assayed level, the alignment level, and an allele's level —
rather than reading as dcd-mapping's QC vocabulary alone. The name also
harmonizes with the *_level columns and fields it types.
Carry the enum through to consumers so assay_level is documented as a
closed set instead of a free string:
- type assay_level as Optional[SequenceLevel] on the lean-variant,
variant-detail, and allele-measurement responses, so OpenAPI emits the
enum members
- type the matching lib transit dataclasses the same way, coercing the
stored column string to the enum at each construction boundary
- rename the mapping worker's local to sequence_level where it holds the
enum (the dcd wire-code loop var keeps its name)
The stored values and wire format are unchanged (SequenceLevel is a str
enum), so no data migration is required and existing clients are
unaffected.
Add an is_current dimension as the top precedence in _ordering_key so current measurements sort ahead of superseded ones, above the existing direct-vs-related, evidence-strength, published-date, and urn keys. - Extend _ordering_key with the current/superseded ordinal - Add pure unit tests pinning the full precedence chain and the null-published-date fallback - Add router tests covering direct-before-related, current-before- superseded, newest-published-first, and the urn tiebreak in the serialized response body
Reshape the score-set clinical-controls response around the allele-link substrate and off the legacy mapped-variant model. - Introduce ControlVariantLink in the clinical-controls lib, carrying the VRS digest of the allele each ClinVar control annotates so clients can apply D78 precedence against the variant's authoritative assay-level digest - Extract the response leaf model out of mapped_variant.py into clinical_control.py as ClinvarVariantLink, dropping its dead variant-URN coercion validator and unused from_attributes - Rename the container view models to ClinicalControlWithClinvarLinks (and Saved variant) and the field mapped_variants -> clinvar_links - Update the router builder and clinical-control tests, including coverage for two controls reaching one variant via distinct alleles
…de RUO Rename the measurement's inline classification field from ``primary_classification``/``primaryClassification`` to ``preferred_classification``/``preferredClassification`` to reflect that the surfaced value is the UI's default from the primary-first preference cascade, not necessarily the promoted primary calibration. - Exclude research-use-only calibrations outright from the preferred classification on this clinical surface, rather than merely deprioritizing them as the shared cascade does - Add a test covering RUO exclusion and fix a self-contradictory cascade fixture that marked a calibration primary while asserting an investigator-provided one should win
Reverse translation can produce very large allele counts, so the CAR submission now chunks HGVS into fixed-size batches instead of a single PUT. Each batch is dispatched and reconciled independently, so a transient failure or a broken one-result-per-input response fails only that batch's alleles rather than the whole run. - add DEFAULT_CAR_SUBMISSION_BATCH_SIZE (env-configurable) and loop the dispatch/reconcile logic per batch, scaling progress across chunks - add an explicit (connect, read) timeout to all ClinGen HTTP calls so a stalled server can no longer hang the worker indefinitely; widen the LDH authenticate catch to RequestException - cover batch splitting and per-batch failure isolation with tests
ClinVar and gnomAD are nucleotide-level resources keyed on genomic/coding changes. A protein allele's clinical calls and population frequencies are carried by its underlying g./c. siblings, so linking them at the protein level is a category error now that projection pairing exposes those siblings. Exclude protein-level alleles from both linking jobs. - Carry the allele's sequence level on ScoreSetAlleleRow so nucleotide-only jobs can filter without a second query - Add EventReason.PROTEIN_LEVEL_ALLELE and record protein alleles as a not-applicable structural gap (mirrors the multi-variant-CAID skip) - refresh_clinvar_controls: skip protein alleles first in the loop, before any ClinGen resolution - link_gnomad_variants: drop protein alleles from the Athena query set and skip them in the annotation loop, sparing the round-trip
Under a nucleotide assay the reverse-translation fan leaves synonymous "cousins" on the record: nt alleles in other projection groups that encode the same protein consequence as the measured change. They are distinct, unmeasured variants, not representations of the measured change, and were previously mislabelled as candidates/projections. - Cat-VRS: add the CO_ENCODES relation and carry cousins as members in the full-closure detail object (include_convergent=True); the VA-Spec subject (include_convergent=False) drops them, keeping only the measured change's coordinate partner and protein consequence. - variant detail: add the `convergent` derivation, held in lockstep with the co_encodes relation, so cousins are no longer labelled `candidate` (which now means only protein-assay reverse-translation ambiguity). - relation/derivation now key off projection_group to distinguish the measured change's coordinate partner (same group) from a cousin (a different group).
The annotation util.py had grown into an unrelated grab-bag. Split it so each helper lives with the concept it serves. - Extract VRS rehydration into lib/vrs.py, the annotation eligibility predicates into eligibility.py, calibration selection into calibration.py, and the VariantAnnotationContext builder into context.py. - Co-locate sequence_feature_for_variant in proposition.py, its sole consumer. - Repoint the importers (annotate, classification, document, evidence_line, proposition, statement, study_result) and split the tests to match the new module homes, with patch targets now pointing at the consuming module rather than the old util module.
Move annotation serving off the legacy MappedVariant model onto the MappingRecord/Allele substrate written by the mapping pipeline. New score sets, whose mapping data lands only on the new substrate, previously enumerated no annotatable variants and silently returned nothing. - Replace get_current_mapped_variants_for_annotation with get_annotatable_variants, querying live MappingRecord/Allele links and threading an as_of instant for temporal reconstruction. - Thread as_of end to end through the three streaming annotation endpoints: into get_annotatable_variants (set enumeration) as well as the per-variant context, so the set and its annotations share one instant and cannot drift. - Return 200 with an empty stream when a score set exists but has no annotatable variants (never mapped, or none live at as_of); 404 is reserved for an unresolvable URN or a permission failure, so as_of acts as a filter and never manufactures a 404. - Derive mavedb_vrs_contribution provenance from the live MappingRecord via VariantAnnotationContext instead of MappedVariant. - Add find_variants_by_vrs_identifier over Allele.vrs_digest, and repoint the public-data export onto get_annotatable_variants.
…ed-variants router
Move the single-variant annotation surface onto the variants router,
served from the MappingRecord/Allele substrate, and remove the legacy
mapped_variant router entirely.
- Add GET /variants/{urn}/va/{study-result,functional-statement,
pathogenicity-statement}, each building a VA-Spec resource from the
variant's live mapping context and threading as_of for temporal
reconstruction.
- Add GET /variants/vrs/{identifier}: resolve a GA4GH VRS id to the
readable variants whose mapping links that allele, filtered by as_of.
- Delete routers/mapped_variant.py and its registration in server_main.
404 semantics follow the collection-vs-resource split: the VRS lookup is
collection-shaped, so no readable match returns 200 with an empty list
(an absent id and a private-only match are indistinguishable, hiding
existence); the single derived /va/* resources 404 when the variant is
unknown or the statement does not exist at the requested instant.
Add GET /alleles/{identifier}, the allele-grain sibling of GET
/variants/{urn}: anchor identity + the full cross-layer equivalence class
(each member labelled relative to the focus — projection / candidate /
convergent) + digest-keyed annotations. One path overloaded across a VRS
digest, a nucleotide CAID, or a protein PAID via a custom Starlette
convertor. Public and measurement-agnostic; no Cat-VRS (that stays rooted
at the measured allele on the variant view).
Unify the molecular core with variant detail: extract a shared
AlleleIdentity (lib + view model) used by both endpoints' `alleles` map,
and mark the anchored allele with `isFocus` — replacing the
`derivation: "authoritative"` value on variant detail (there is no
authoritative *variant* at the allele grain).
Factor the nucleotide-level set into SequenceLevel.NUCLEOTIDE_LEVELS,
shared by cat_vrs and allele_detail.
…_sets router score_sets.py had its own private `enqueue_pipeline_entrypoint` helper for building `Pipeline`/`JobRun` records and enqueuing the `start_pipeline` ARQ job, duplicated at both of its call sites. Move that logic into `mavedb.lib.workflow.kickoff.enqueue_pipeline_for_score_set` so the `run_pipeline` script and other future callers share the same enqueue contract instead of reimplementing it. - Restore the discard-pipeline-on-enqueue-failure safety net that the router's original helper had but the extracted version initially dropped, so a failed ARQ enqueue never leaves an orphaned pipeline behind in the database. - Add unit and integration tests for `enqueue_pipeline_for_score_set`, covering correlation-id precedence, param merging, ARQ dedup handling, and pipeline cleanup on enqueue failure.
…lines - Add run_score_set_pipelines.py, which selects unmapped or unenriched score sets and enqueues the appropriate pipeline for each, ordered by gene to maximize ClinGen CAR cache reuse - Refactor run_pipeline.py to share pipeline creation/enqueueing logic via the new enqueue_pipeline_for_score_set helper instead of duplicating PipelineFactory/redis wiring inline
…ost-mapped vrs get_hgvs_from_post_mapped only matched a bare "Allele" type, so post-mapped payloads wrapped in a VariationDescriptor fell through to the unhandled branch and returned None instead of their hgvs.
…variants Historical score sets have no mapping_records/alleles data yet, since that substrate is populated only by the live mapping job going forward. This reconstructs it deterministically from existing mapped_variants rows so the new serving/read layer (v_variant_annotations, published_variants MV) resolves for old data too, without redoing the reverse-translation fan-out (deferred to a decoupled enrichment pass). - add migrate_mapped_variants_to_allele_substrate manual migration, with verify/rollback/rebuild-annotations subcommands and idempotent re-run support - freeze the legacy MV-index migration's column signature as a literal instead of importing it from the model, since a later migration rewrites that model - add EventReason.MIGRATED for backfill-reconstructed annotation events, since a migration can't know the original creation history, only present/absent/failed
…ords The read layer (v_variant_annotations, published_variants MV, statistics router) still joined the legacy mapped_variants table, so it never resolved the mapping_records/alleles substrate the mapping job now writes to. This completes the read-side cutover onto the new tables. - rewrite v_variant_annotations to pivot the record's live allele links into the flat hgvs_g/hgvs_c/hgvs_p triple, and pull clingen_allele_id and VEP consequence from the authoritative allele, via correlated scalar subqueries - rebuild published_variants_materialized_view on mapping_record_id / current_mapping_record, keeping every record version (not just the live one) so history-aware counts still work - update statistics router and tests to the renamed columns - freeze all historical migrations' view/MV SQL as inline literals instead of importing from the model, so replaying an old revision builds the shape that existed at that point in history, not whatever the model looks like after later migrations rewrite it - add view-level tests for the flat-triple reconstruction, protein-only assays, and unmapped variants under the new substrate
link_gnomad_variants_to_alleles queried alleles once per gnomAD variant row inside the loop. The predicate wraps clingen_allele_id in regexp_replace to bridge zero-padded (MaveDB) and stripped (gnomAD dump, #722) CAIDs, which is non-sargable and forces a sequential scan of the alleles table on every execution. Over a linking run this becomes N full scans, saturating database IO (observed as sustained IO:DataFileRead above max vCPUs on the enlarged post-backfill alleles table). Resolve all rows in a single IN scan before the loop and group the result in Python by normalize_caid, so the loop reads from an in-memory map. N scans collapse to one per run, with no schema change. Grouping reuses normalize_caid on both the input and result sides, so the SQL and Python normalization forms are now exercised together. - add test_links_a_batch_of_rows_in_one_pass covering multi-row batching: distinct CAIDs, zero-padding match (#722), and shared-CAID fan-out preserved across a single call
The --select unenriched path enqueues annotate_score_set, whose reverse-translation step needs a transcript. It previously selected any mapped-but-not-enriched score set, so ones with no usable transcript were re-selected on every run and burned worker cycles skipping every variant as transcript_unresolved. - split _needs_enrichment into _mapped_without_enrichment and a new _has_usable_transcript predicate (cdna TargetGeneMapping reference_accession, or a live NP_/XP_ RefSeq protein MappingRecord), mirroring the two transcript sources in reverse_translation.py - require a usable transcript for unenriched eligibility - report the count of mapped score sets skipped for lack of a transcript so they are visibly dropped rather than silently lost
- add preMapped/postMapped VRS pair to VariantDetail and switch
GET /variants/{urn} to exclude_none=False so absent fields
serialize as null instead of being dropped
- add streaming NDJSON GET /score-sets/{urn}/variant-details,
replacing the retired GET /score-sets/{urn}/mapped-variants
- replace MappedVariant-keyed gnomAD lookups with a substrate-based
get_gnomad_variants_with_variant_urns, pairing each gnomAD variant
with the score-set variant URNs it links to
(GnomADVariantWithVariantLinks)
- add as_of time-travel support to the CSV variants export and the
gnomad-variants endpoint, echoed via X-As-Of response headers
- extract mapped_hgvs_by_level/get_protein_hgvs_by_record helpers
out of score_set_variants for reuse by the CSV export
- replace the MappedVariant-keyed CSV joins with a substrate query (_csv_substrate_query) over the live mapping record, its authoritative allele, and the projection-group nucleotide sibling - add _gnomad_by_allele/_clinvar_by_allele batch fetches keyed by allele id, replacing the old mapped_variant_id-keyed lookups - reuse mapped_hgvs_by_level/get_protein_hgvs_by_record from score_set_variants so the CSV resolves post-mapped HGVS per level the same way the lean whole-set view does, instead of falling back to a per-row VRS parse - extract column planning (_plan_csv_columns) and header assembly (_csv_header_columns) out of get_score_set_variants_as_csv - thread as_of through the CSV export to time-travel the annotation layer (post-mapped HGVS, VEP, gnomAD, ClinVar) independently of the immutable scores/counts namespaces - add seed_csv_substrate test helper for the substrate-based fixtures
The mapped-variant endpoints removed earlier on this branch are still
live on main, so merging is what actually retires them for public API
consumers. Soften that landing per-route, based on whether the
replacement is wire-compatible:
- GET /score-sets/{urn}/mapped-variants now returns 410 Gone pointing
at variant-details, since the streaming NDJSON shape isn't
compatible with the old JSON array and a redirect would mislead
callers
- /mapped-variants/{urn} and its va/* and vrs/{identifier} siblings
301-redirect onto their /variants equivalents, since that move was a
straight relocation with an unchanged response shape
- POST /variants/clingen-allele-id-lookups stays fully removed with no
stub; it was effectively an internal-only route
- add a reusable 410 entry to routers/shared.py's BASE_RESPONSES
The include_nucleotide_siblings flag never actually gated siblings — reverse translation cross-links every record to its full synonymous nt set, so a sibling's record already links the anchor allele. Remove the flag (api + query param) and make the protein-apex fold-in unconditional for a CA query, so one behavior serves both the variant page and search: a CA reaches its protein consequence even when that protein assay hasn't been reverse-translated (its record links only the protein node). Resolve the apex once in _resolve_protein_apex, and instrument it: warn on a divergent apex (>1 PAID), and publish apex PAID/unregistered counts and the protein-consequence / apex-only-unresolved rates to the request log. Update tests to the one-behavior model.
Even with higher timeout settings, some VEP jobs were still hitting against the timeout window. Given the new checkpoint features, increasing the coarse timeout is lower risk and, in the aggregate, will help more jobs run to completion.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.