Implement Beacon v2 genomic data-sharing endpoint

[davmlaw]

🤖 Written by Claude

Split from #40 (external DB API integrations).

Expose VariantGrid variants via a Beacon v2 endpoint. Beacon v2 is the approved GA4GH standard (latest v2.2.0, July 2025); v1 is dead. v2 uses proper /api/... paths, so the old "endpoint must be at /" concern from #40 is resolved — we can host on e.g. beacon.shariant.org.

What v2 offers over v1: presence/counts, cohort/dataset metadata, phenotypic info, and access/contact details for follow-up, tiered by auth (boolean → counts → record-level for authorized users).

Reference implementations

• EGA-archive/beacon2-ri-api + beacon2-ri-tools (CRG B2RI)
• ga4gh/ga4gh-starter-kit-beacon

Work involved

• ETL from Variant/Allele → Beacon v2 data model
• Map Guardian object-level permissions → Beacon auth tiers
• Optionally federate into the ELIXIR Beacon Network and/or register with the EGA-hosted registry (federated model — there is no single central server to "join")

Docs: https://docs.genomebeacons.org/

[davmlaw]

This will need to be behind a settings feature flag, so it can be disabled in Shariant (make it disabled by default)

[davmlaw]

🤖 Written by Claude

Wrote an implementation plan for this — claude/beacon_v2_plan.md (same place/style as the MatchMaker Exchange plan for #1662).

Feasibility: high. Every primitive Beacon v2 needs already exists; the work is mostly spec-conformance, not data access:

• Coordinates / query → VariantCoordinate, Variant.get_contigs_q(); cross-build identity + ClinGen ID via Allele.
• Auth tiers → GuardianPermissionsMixin.filter_for_user() — and it already falls back to the public group for AnonymousUser, which is the "anonymous externals get public records" model. No new permission code.
• Flag / anon access → *_ENABLED convention + gated URL include + PUBLIC_PATHS (needed because GlobalLoginRequiredMiddleware rejects anon requests before DRF runs).

Count strategy (two-stage, correctness-first):

1. Fast gate — global VariantZygosityCount. One precomputed indexed row per variant; a zero/absent count → exists: false immediately (covers the common "not in DB" case). Used only as a negative gate — the global count includes private samples, so a positive value is never reported as presence (would leak private-only variants).
2. Permission-scoped count — CohortGenotype. Only when stage 1 says "present": Sample.filter_for_user(user) (anon → public) → visible het/hom count. Presence/counts are only ever asserted from samples the requester may read.

This reuses the logic already in snpdb/variant_sample_information.py (VariantSampleInformation already computes visible vs hidden observation counts). Plan proposes extracting its permission-aware core out of the pandas/template presentation so Beacon and the existing Variantopedia view share it — low risk, single caller today.

Scope: all three granularity tiers (boolean → count → record), genomicVariant entry type. datasets/individuals/biosamples entry types, ontology filtering_terms, and ELIXIR/EGA federation are deferred to later phases.

Feature flag: BEACON_ENABLED = False by default, explicitly off in the Shariant env files.

See the plan doc for the full endpoint map, field-by-field data mapping, phasing, tests, and open questions (notably: per-sample model first vs. a BeaconDataset per-dataset opt-in registry; and whether to apply count suppression / k-anonymity on the anonymous tier).

[davmlaw]

Implementation prompt

  Implement Phase 1 of the Beacon v2 endpoint (issue #1661). The plan is the spec:
  claude/beacon_v2_plan.md — follow it, especially §3–§9.

  Read first:
  - claude/beacon_v2_plan.md (the full plan)
  - snpdb/variant_sample_information.py (the permission-aware per-variant sample lookup to reuse)
  - variantopedia/views.py:743 (its only current caller)
  - snpdb/models/models_zygosity_counts.py (the fast VariantZygosityCount gate)
  - snpdb/views/views_rest.py and snpdb/serializers.py (the DRF pattern to mirror)
  - variantgrid/urls.py and variantgrid/settings/components/default_settings.py
    (flag wiring, PUBLIC_PATHS, APPS_WITH_URLS)

  Build order and overrides:
  - Start by extracting the permission-aware core out of VariantSampleInformation per §5.3,
    and update variantopedia/views.py to call the extracted core so behaviour stays identical.
  - Use the §5.2 two-stage per-sample count model (global VariantZygosityCount gate → CohortGenotype
    permission walk). Keep the §5.4 BeaconDataset registry out of Phase 1.
  - Create the new `beacon` app with framework endpoints + g_variants supporting boolean,
    count, and record granularity for the genomicVariant entry type, tiered via filter_for_user.
  - Add BEACON_ENABLED (default False) plus BEACON_CONFIG, gate the URL include, add the beacon
    prefix to PUBLIC_PATHS, register `beacon` in INSTALLED_APPS and APPS_WITH_URLS, and set
    BEACON_ENABLED = False explicitly in the Shariant env files.
  - Add tests per §9, with the permission-tiering test (anonymous sees only public-group samples;
    a private-only variant returns exists:false to anonymous, visible to an authorised user) as
    the priority case.

  Work on branch beacon-v2-plan-1661 (where the plan is committed). Run the new tests with
  `python3 manage.py test --keepdb`.

  Report back: files added/changed, the endpoints live, how the permission tiering was verified,
  and any deviations from the plan with reasons.

[davmlaw]

Can use this post deployment
https://beacon-verifier-demo.ega-archive.org/

[davmlaw]

🤖 Written by Claude

Outbound query (§9): search for external Beacon sites

Before enabling the outbound side (querying external Beacons from the variant page), we probed every candidate we could find, to check whether any hold data we don't already have. Conclusion: there is nothing worth querying today, so outbound is implemented but ships disabled (BEACON_OUTBOUND_ENABLED = False, empty BEACON_QUERY_NODES).

What we probed

• All ~39 beacons on the CRG community list (Google sheet)
• The progenetix-family beacons (progenetix.org, cancercelllines.org, refcnv.org)
• The ELIXIR beacon-network aggregator and its shipped member config

Test variants: rs6025 (Factor V Leiden, ~2% MAF) and rs4988235 (LCT, ~40% MAF).

Findings

• Only one node returns real population counts — the EGA Allele Frequency Browser, which is a gnomAD v4.1 frontend (rs4988235: AF 39.7%, AC 60,356 / AN 1,613,166). We already hold gnomAD locally in VariantAnnotation, so querying it over the wire adds nothing.
• progenetix / cancercelllines / refcnv — live and spec-clean, but cancer/CNV-only: no hits for germline variants.
• BSC Beacon Network aggregator + the ELIXIR beacon-network shipped members — only synthetic/demo datasets; empty even for a ~40%-MAF variant.
• ~28 others — dead / 404 / 503 / DNS gone / non-v2.

Corroboration

The GA4GH Beacon AMA (11 Mar 2025) confirms this is ecosystem-wide, not a probing artifact — the product team's own words: "most beacons are not data rich in their data delivery." A public implementation registry was promised "in the next few months" there, but is still not live 16 months on. (We raised the missing-registry point upstream: ga4gh-beacon/ga4gh-beacon.github.io#1.)

Decision

• Inbound Beacon ships — that is this issue's main deliverable and what makes us listable once a registry exists.
• Outbound stays behind its flag until a real peer network exists.
• The genuinely valuable outbound target is a private clinical Beacon network (count-granularity + OIDC), which the AMA repeatedly ties back to Matchmaker Exchange — the federation we already run in mme/. Our Beacon client was deliberately modeled on the MME client, so that is a natural next step rather than new infrastructure.

Minor hygiene from the exercise: the outbound client no longer raises an admin notification on an expected dead-node failure (that condition is logged + surfaced as an error row), and now sends a descriptive User-Agent (some beacon WAFs 403 default Python clients).

[davmlaw]

🤖 Written by Claude

Outbound §9 update: per-node gating + CNV/somatic querying (commit 6be141d)

Following the outbound-site probing above (germline SNVs found nothing additive — only synthetic/demo nodes or gnomAD we already hold), we re-examined the CNV/somatic domain and found the opposite: the bycon/beaconPlus family holds real, queryable copy-number data. So outbound is now enabled for that domain, with per-node routing so it never wastes calls on the wrong domain.

Per-node gating

New beacon/query_targets.py introduces a per-server gate + query-builder:

• BeaconQueryTarget.accepts(variant) — the gate (is this variant in the server's domain?)
• .build_params(variant) — the coordinate query to send
• SnvBeaconTarget (exact ref/alt) and CnvBeaconTarget (symbolic <DEL>/<DUP> → range query)
• eligible_queries() loops the configured nodes and fans out only to servers whose gate the variant passes.

Node config gains type ("cnv"/"snv") and an optional assemblies allow-list. A germline SNV page with only CNV nodes configured now renders empty (no calls, no error rows) rather than blindly querying every node.

CNV query mapping

Symbolic variants (VG stores CNVs as <DEL>/<DUP> + svlen; per-sample CN lives on CohortGenotype and stays internal) map to a Beacon v2 range query: referenceName + start/end (= position + abs(svlen)) + variantType, using GA4GH/EFO copy-number classes (<DEL>→EFO:0030067 loss, <DUP>→EFO:0030070 gain). Only the public span + copy-number class leaves.

Nodes wired (vgtest2, GRCh38, open tier)

Validated live with a CDKN2A-region deletion query:

Node	count
progenetix.org	57,677
cancercelllines.org	2,437
refcnv.org	1,231

Also landed from the investigation

• Dropped the admin notification on expected dead-node failures (logged + surfaced as an error row instead — it was firing per-variant).
• Added a descriptive User-Agent — several beacon WAFs 403 default Python-urllib/requests clients; our UA is accepted.
• Request count granularity outbound (we only render presence/count).

Not yet done

• Manual smoke test of a real symbolic CNV variant page on test.variantgrid.com (the emitted query was validated live against progenetix, but not yet through the VG view with a real CNV record).
• Somatic SNV/small-variant querying (e.g. COSMIC) is a separate snv-type node set, not yet added.
• prod (variantgrid.com) still has outbound off; enable there once the test smoke-check passes.