Skip to content

Add LUCID: regime-adaptive deconfounding for latent confounders - #35

Closed
fesanghary wants to merge 1 commit into
mainfrom
lucid-public-port
Closed

Add LUCID: regime-adaptive deconfounding for latent confounders#35
fesanghary wants to merge 1 commit into
mainfrom
lucid-public-port

Conversation

@fesanghary

@fesanghary fesanghary commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds LUCID: regime-adaptive deconfounding for causal discovery under latent confounders, in a new causalts.confounders module. run_lucid() -> LucidResult, plus .deconfound() / .tetrad_filter() / .pds_filter() added to every CausalResult subclass (works on CDNOTS, CEDAR, GRACE, ... results, not just LUCID's own).
  • LUCID diagnoses whether latent confounding is sparse or pervasive from the residual spectrum (against a Marchenko-Pastur no-factor null) and applies the matching correction. Both regimes run the same base discovery engine, so routing only changes the correction applied afterward — which is what makes .deconfound() valid for reusing an already-discovered graph from any algorithm.
  • Adds causalts.synthetic_data.confounding.apply_confounding (needed by the test suite), a new "Unobserved Confounders (LUCID)" tutorial notebook + gallery entry, and an API doc page.
  • Exports are added to causalts.__init__'s existing lazy (_LAZY_ATTRS) pattern rather than eager, since causalts.confounders pulls in statsmodels/scikit-learn, which plain import causalts doesn't currently load — keeping that path cheap is this file's whole design intent.

Test plan

  • pytest tests/test_confounders.py -v — 35/35 pass
  • pytest -q (full suite) — 308/308 pass, no regressions from the causalts/result.py / causalts/__init__.py / causalts/synthetic_data/__init__.py edits
  • Notebook executes end-to-end (jupyter nbconvert --execute): 0 error cells, sane metrics (CDNOTS F1=0.172 -> tetrad F1=0.385 -> LUCID F1=0.556 on the same pervasive-confounding scenario); the committed notebook's baked-in outputs are byte-identical to what ships (confirmed clean, no stderr/warnings) — docs/conf.py sets nb_execution_mode = "off" so Sphinx renders exactly those saved outputs, not a fresh re-run
  • Lint, matching this repo's exact CI invocation: black==24.* pinned (black --check .), isort --check ., flake8 . (with Flake8-pyproject installed, since flake8 needs it to read [tool.flake8] from pyproject.toml at all) — all clean; codespell clean (added one justified retuned ignore-word for a real word codespell mistakes for "returned")
  • sphinx-build -b html docs docs/_build — succeeds; no new warnings beyond the pre-existing autoapi duplicate-object-description pattern already present for every other result class

Ports the LUCID library (shipped internally via qfr-ml/causal-ts#25, #26) to
the public mirror: run_lucid() -> LucidResult, .deconfound()/.tetrad_filter()/
.pds_filter() on every CausalResult subclass, and the standalone filters
(pds_filter, tetrad_filter) and router diagnostics (spectral_gap,
mp_factor_count) in causalts.confounders.

LUCID diagnoses whether latent confounding is sparse or pervasive from the
residual spectrum (against a Marchenko-Pastur no-factor null) and applies the
matching correction; both regimes run the same base discovery engine, so the
routing decision only changes the correction applied afterward -- which is
what makes .deconfound() valid for reusing an already-discovered graph from
any algorithm (CDNOTS, CEDAR, GRACE, ...).

Also adds causalts.synthetic_data.confounding.apply_confounding (needed by
the test suite) and a new Unobserved Confounders (LUCID) tutorial notebook,
gallery entry, and API doc page.

Exports are lazy (causalts.__init__'s _LAZY_ATTRS pattern) rather than eager,
matching internal: confounders pulls in statsmodels + scikit-learn, which
plain `import causalts` doesn't currently load, and keeping that path cheap
is this file's whole design intent.

Verified: 35/35 test_confounders.py, 308/308 full suite, notebook executes
clean (0 errors) with sane metrics (CDNOTS F1=0.172 -> tetrad F1=0.385 ->
LUCID F1=0.556 on the same pervasive-confounding scenario), black 24 (CI's
pin)/isort/flake8/codespell clean via the exact commands+config CI runs,
sphinx-build succeeds with no new warnings beyond the pre-existing
autoapi duplicate-object-description pattern shared by every other result
class.
@fesanghary
fesanghary requested a review from a team as a code owner August 27, 2026 17:03
@github-actions

Copy link
Copy Markdown

⚠️ Deprecation Warning: The deny-licenses option is deprecated for possible removal in the next major release. For more information, see issue 997.

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@fesanghary

Copy link
Copy Markdown
Contributor Author

Closing to redo with a corrected commit message that referenced an internal-only repo name. Reopening as a fresh PR.

@fesanghary fesanghary closed this Aug 27, 2026
@fesanghary
fesanghary deleted the lucid-public-port branch August 27, 2026 17:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant