Skip to content

Add LUCID: regime-adaptive deconfounding for latent confounders - #36

Merged
fesanghary merged 1 commit into
mainfrom
confounders-lucid
Aug 27, 2026
Merged

Add LUCID: regime-adaptive deconfounding for latent confounders#36
fesanghary merged 1 commit into
mainfrom
confounders-lucid

Conversation

@fesanghary

Copy link
Copy Markdown
Contributor

Summary

  • Adds LUCID: regime-adaptive deconfounding for causal discovery under latent confounders, in a new causalts.confounders module. run_lucid() -> LucidResult, plus .deconfound() / .tetrad_filter() / .pds_filter() added to every CausalResult subclass (works on CDNOTS, CEDAR, GRACE, ... results, not just LUCID's own).
  • LUCID diagnoses whether latent confounding is sparse or pervasive from the residual spectrum (against a Marchenko-Pastur no-factor null) and applies the matching correction. Both regimes run the same base discovery engine, so routing only changes the correction applied afterward — which is what makes .deconfound() valid for reusing an already-discovered graph from any algorithm.
  • Adds causalts.synthetic_data.confounding.apply_confounding (needed by the test suite), a new "Unobserved Confounders (LUCID)" tutorial notebook + gallery entry, and an API doc page.
  • Exports are added to causalts.__init__'s existing lazy (_LAZY_ATTRS) pattern rather than eager, since causalts.confounders pulls in statsmodels/scikit-learn, which plain import causalts doesn't currently load — keeping that path cheap is this file's whole design intent.

Test plan

  • pytest tests/test_confounders.py -v — 35/35 pass
  • pytest -q (full suite) — 308/308 pass, no regressions from the causalts/result.py / causalts/__init__.py / causalts/synthetic_data/__init__.py edits
  • Notebook executes end-to-end (jupyter nbconvert --execute): 0 error cells, sane metrics (CDNOTS F1=0.172 -> tetrad F1=0.385 -> LUCID F1=0.556 on the same pervasive-confounding scenario); the committed notebook's baked-in outputs are byte-identical to what ships (confirmed clean, no stderr/warnings) — docs/conf.py sets nb_execution_mode = "off" so Sphinx renders exactly those saved outputs, not a fresh re-run
  • Lint, matching this repo's exact CI invocation: black==24.* pinned (black --check .), isort --check ., flake8 . (with Flake8-pyproject installed, since flake8 needs it to read [tool.flake8] from pyproject.toml at all) — all clean; codespell clean (added one justified retuned ignore-word for a real word codespell mistakes for "returned")
  • sphinx-build -b html docs docs/_build — succeeds; no new warnings beyond the pre-existing autoapi duplicate-object-description pattern already present for every other result class

Adds LUCID: regime-adaptive deconfounding for causal discovery under latent
confounders, in a new causalts.confounders module. run_lucid() -> LucidResult,
plus .deconfound()/.tetrad_filter()/.pds_filter() on every CausalResult
subclass (works on CDNOTS, CEDAR, GRACE, ... results, not just LUCID's own),
and the standalone filters (pds_filter, tetrad_filter) and router diagnostics
(spectral_gap, mp_factor_count) in causalts.confounders.

LUCID diagnoses whether latent confounding is sparse or pervasive from the
residual spectrum (against a Marchenko-Pastur no-factor null) and applies the
matching correction; both regimes run the same base discovery engine, so the
routing decision only changes the correction applied afterward -- which is
what makes .deconfound() valid for reusing an already-discovered graph from
any algorithm.

Also adds causalts.synthetic_data.confounding.apply_confounding (needed by
the test suite) and a new Unobserved Confounders (LUCID) tutorial notebook,
gallery entry, and API doc page.

Exports are lazy (causalts.__init__'s _LAZY_ATTRS pattern) rather than eager:
confounders pulls in statsmodels + scikit-learn, which plain `import
causalts` doesn't currently load, and keeping that path cheap is this file's
whole design intent.

Verified: 35/35 test_confounders.py, 308/308 full suite, notebook executes
clean (0 errors) with sane metrics (CDNOTS F1=0.172 -> tetrad F1=0.385 ->
LUCID F1=0.556 on the same pervasive-confounding scenario), black 24 (CI's
pin)/isort/flake8/codespell clean via the exact commands+config CI runs,
sphinx-build succeeds with no new warnings beyond the pre-existing
autoapi duplicate-object-description pattern shared by every other result
class.
@fesanghary
fesanghary requested a review from a team as a code owner August 27, 2026 17:59
@github-actions

Copy link
Copy Markdown

⚠️ Deprecation Warning: The deny-licenses option is deprecated for possible removal in the next major release. For more information, see issue 997.

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@fesanghary
fesanghary merged commit 56bf3f7 into main Aug 27, 2026
10 checks passed
@fesanghary
fesanghary deleted the confounders-lucid branch August 30, 2026 00:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant