benchmark comparison code and report by iwfarr · Pull Request #2 · fastomop/omopy

iwfarr · 2026-04-23T12:31:33Z

load equivalent R scripts
code to generate ~10k Synthea patients
run comparison scripts and report

iwfarr · 2026-04-23T13:12:17Z

Adds a complete benchmarking framework that runs 10 OMOP CDM analysis modules in both R (HADES packages) and Python (OMOPy), then auto-generates a side-by-side comparison report.

Benchmark scripts (28 files)

benchmarks/python/ — 10 Python benchmark scripts + helpers + runner
benchmarks/r/ — 10 R benchmark scripts + helpers + runner + package installer
benchmarks/compare.py — reads R and Python results, generates docs/comparison.md automatically
benchmarks/generate_synthea_1k.R — downloads the Eunomia synthea-medications-10k dataset (~10,681 patients) into DuckDB for benchmarking
benchmarks/README.md — full setup and usage instructions

Documentation (4 files)

docs/comparison.md — auto-generated comparison report covering all 10 modules
docs/comparison_files/cohort_overview.svg — pipeline diagram showing cohort flow with subject/event counts
docs/comparison_files/cohort_overview.mmd — mermaid source for the diagram
docs/r-package-mapping.md — maps R HADES packages to OMOPy equivalents

Copilot

Pull request overview

This PR adds an end-to-end benchmark suite to compare OHDSI R package outputs against OMOPy’s Python implementations on the same Synthea-derived DuckDB dataset, and publishes a generated concordance report into the docs site.

Changes:

Adds R and Python benchmark runners plus per-benchmark scripts that generate CSV outputs and timing files.
Adds a Python report generator (benchmarks/compare.py) and commits the generated comparison report + cohort overview diagram to docs/.
Adds supporting documentation (benchmark README, R→Python module mapping) and updates .gitignore for generated benchmark artifacts and datasets.

Reviewed changes

Copilot reviewed 31 out of 33 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
docs/r-package-mapping.md	New documentation mapping OHDSI R packages to OMOPy modules and key tech substitutions.
docs/comparison_files/cohort_overview.svg	Generated cohort/benchmark flow diagram embedded in docs.
docs/comparison_files/cohort_overview.mmd	Mermaid source for the cohort overview diagram.
docs/comparison.md	Generated R vs Python comparison report (timings, row counts, value concordance, notes).
benchmarks/r/run_all.R	Runner that executes all numbered R benchmark scripts sequentially.
benchmarks/r/install_packages.R	Script to install required OHDSI R packages and dependencies for benchmarks.
benchmarks/r/10_drug_diagnostics.R	R benchmark for DrugExposureDiagnostics checks.
benchmarks/r/09_treatment_patterns.R	R benchmark for TreatmentPatterns pathway computation/export.
benchmarks/r/08_codelist.R	R benchmark for CodelistGenerator keyword candidate codes.
benchmarks/r/07_survival.R	R benchmark for CohortSurvival single-event survival.
benchmarks/r/06_drug_utilisation.R	R benchmark for DrugUtilisation cohort + summary.
benchmarks/r/05_incidence.R	R benchmark for IncidencePrevalence incidence estimation.
benchmarks/r/04_characteristics.R	R benchmark for CohortCharacteristics summary.
benchmarks/r/03_patient_profiles.R	R benchmark for PatientProfiles demographics enrichment.
benchmarks/r/02_cohort_generation.R	R benchmark for CDMConnector concept cohort generation.
benchmarks/r/01_snapshot.R	R benchmark for CDMConnector snapshot.
benchmarks/r/00_helpers.R	Shared R helpers/constants for benchmark scripts.
benchmarks/python/run_all.py	Runner that executes all numbered Python benchmark scripts sequentially.
benchmarks/python/helpers.py	Shared Python helpers for connecting, saving results, and timing.
benchmarks/python/10_drug_diagnostics.py	Python benchmark for omopy drug diagnostics checks.
benchmarks/python/09_treatment_patterns.py	Python benchmark for omopy treatment patterns compute + summary.
benchmarks/python/08_codelist.py	Python benchmark for omopy codelist candidate codes (flattened to CSV).
benchmarks/python/07_survival.py	Python benchmark for omopy survival estimates.
benchmarks/python/06_drug_utilisation.py	Python benchmark for omopy drug utilisation summary.
benchmarks/python/05_incidence.py	Python benchmark for omopy incidence estimate.
benchmarks/python/04_characteristics.py	Python benchmark for omopy characteristics summary.
benchmarks/python/03_patient_profiles.py	Python benchmark for omopy demographics enrichment.
benchmarks/python/02_cohort_generation.py	Python benchmark for omopy concept cohort generation + counts.
benchmarks/python/01_snapshot.py	Python benchmark for omopy snapshot.
benchmarks/generate_synthea_1k.R	Script to download Eunomia Synthea dataset and build DuckDB file for benchmarking.
benchmarks/compare.py	Generates `docs/comparison.md` by comparing R vs Python CSV outputs.
benchmarks/README.md	Documentation for running benchmarks, directory structure, and dataset details.
.gitignore	Ignores generated benchmark data/results and adds rules for sensitive/local directories.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 31 out of 33 changed files in this pull request and generated 15 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-26T09:55:25Z

+import sys; sys.path.insert(0, "benchmarks/python")
+from helpers import connect_cdm, save_result, save_timing, Timer
+
+print("=== 03: Patient Profiles ===")
+t = Timer()
+cdm = connect_cdm()
+
+from omopy.generics import Codelist
+from omopy.connector import generate_concept_cohort_set
+from omopy.profiles import add_demographics
+
+cdm = generate_concept_cohort_set(cdm, Codelist({"coronary_artery": [317576]}), name="target_cohort")
+cohort = cdm["target_cohort"]
+
+enriched = add_demographics(cohort, cdm)
+df = enriched.collect()
+save_result(df.head(100), "03_patient_profiles")
+save_timing("03_patient_profiles", t.elapsed())
+print(f"Done in {t.elapsed():.2f} seconds")


Same Ruff issue here: sys.path is modified and then there are module-level imports after runtime code starts, which will fail ruff-check (E402/E702). Please refactor to avoid sys.path hacks and keep imports at the top-level (or move runtime code under main()).

Suggested change

import sys; sys.path.insert(0, "benchmarks/python")

from helpers import connect_cdm, save_result, save_timing, Timer

print("=== 03: Patient Profiles ===")

t = Timer()

cdm = connect_cdm()

from omopy.generics import Codelist

from omopy.connector import generate_concept_cohort_set

from omopy.profiles import add_demographics

cdm = generate_concept_cohort_set(cdm, Codelist({"coronary_artery": [317576]}), name="target_cohort")

cohort = cdm["target_cohort"]

enriched = add_demographics(cohort, cdm)

df = enriched.collect()

save_result(df.head(100), "03_patient_profiles")

save_timing("03_patient_profiles", t.elapsed())

print(f"Done in {t.elapsed():.2f} seconds")

import importlib.util

from pathlib import Path

from omopy.generics import Codelist

from omopy.connector import generate_concept_cohort_set

from omopy.profiles import add_demographics

def _load_helpers():

helpers_path = Path(__file__).with_name("helpers.py")

spec = importlib.util.spec_from_file_location("benchmark_helpers", helpers_path)

helpers = importlib.util.module_from_spec(spec)

assert spec is not None and spec.loader is not None

spec.loader.exec_module(helpers)

return helpers

_helpers = _load_helpers()

connect_cdm = _helpers.connect_cdm

save_result = _helpers.save_result

save_timing = _helpers.save_timing

Timer = _helpers.Timer

def main():

print("=== 03: Patient Profiles ===")

t = Timer()

cdm = connect_cdm()

cdm = generate_concept_cohort_set(

cdm,

Codelist({"coronary_artery": [317576]}),

name="target_cohort",

)

cohort = cdm["target_cohort"]

enriched = add_demographics(cohort, cdm)

df = enriched.collect()

save_result(df.head(100), "03_patient_profiles")

save_timing("03_patient_profiles", t.elapsed())

print(f"Done in {t.elapsed():.2f} seconds")

if __name__ == "__main__":

main()

Copilot · 2026-04-26T09:55:26Z

+import sys; sys.path.insert(0, "benchmarks/python")
+from helpers import connect_cdm, save_result, save_timing, Timer
+import polars as pl
+
+print("=== 10: Drug Diagnostics ===")
+t = Timer()
+cdm = connect_cdm()
+
+from omopy.drug_diagnostics import execute_checks
+
+result = execute_checks(
+    cdm,
+    ingredient_concept_ids=[1322184],  # clopidogrel
+    checks=["missing", "exposure_duration", "type", "route", "quantity"],
+)
+
+# Combine all check DataFrames
+frames = []
+for check_name in result:
+    df = result[check_name]
+    if df is not None and df.height > 0:
+        df = df.with_columns(pl.lit(check_name).alias("check_name"))
+        frames.append(df)
+
+if frames:
+    # All frames may have different schemas, so save individually
+    combined = pl.concat(frames, how="diagonal_relaxed")
+    save_result(combined, "10_drug_diagnostics")
+else:
+    save_result(pl.DataFrame({"check_name": [], "note": []}), "10_drug_diagnostics")
+
+save_timing("10_drug_diagnostics", t.elapsed())
+print(f"Done in {t.elapsed():.2f} seconds")


Ruff will flag this script due to the sys.path one-liner and module-level imports after runtime statements (E702/E402). Please refactor to a path-stable helper import (relative to __file__) and keep imports at the top, or move delayed imports inside a main() function.

Suggested change

import sys; sys.path.insert(0, "benchmarks/python")

from helpers import connect_cdm, save_result, save_timing, Timer

import polars as pl

print("=== 10: Drug Diagnostics ===")

t = Timer()

cdm = connect_cdm()

from omopy.drug_diagnostics import execute_checks

result = execute_checks(

cdm,

ingredient_concept_ids=[1322184], # clopidogrel

checks=["missing", "exposure_duration", "type", "route", "quantity"],

)

# Combine all check DataFrames

frames = []

for check_name in result:

df = result[check_name]

if df is not None and df.height > 0:

df = df.with_columns(pl.lit(check_name).alias("check_name"))

frames.append(df)

if frames:

# All frames may have different schemas, so save individually

combined = pl.concat(frames, how="diagonal_relaxed")

save_result(combined, "10_drug_diagnostics")

else:

save_result(pl.DataFrame({"check_name": [], "note": []}), "10_drug_diagnostics")

save_timing("10_drug_diagnostics", t.elapsed())

print(f"Done in {t.elapsed():.2f} seconds")

import sys

from pathlib import Path

BENCHMARKS_PYTHON_DIR = Path(__file__).resolve().parent

if str(BENCHMARKS_PYTHON_DIR) not in sys.path:

sys.path.insert(0, str(BENCHMARKS_PYTHON_DIR))

from helpers import connect_cdm, save_result, save_timing, Timer

import polars as pl

from omopy.drug_diagnostics import execute_checks

def main():

print("=== 10: Drug Diagnostics ===")

t = Timer()

cdm = connect_cdm()

result = execute_checks(

cdm,

ingredient_concept_ids=[1322184], # clopidogrel

checks=["missing", "exposure_duration", "type", "route", "quantity"],

)

# Combine all check DataFrames

frames = []

for check_name in result:

df = result[check_name]

if df is not None and df.height > 0:

df = df.with_columns(pl.lit(check_name).alias("check_name"))

frames.append(df)

if frames:

# All frames may have different schemas, so save individually

combined = pl.concat(frames, how="diagonal_relaxed")

save_result(combined, "10_drug_diagnostics")

else:

save_result(pl.DataFrame({"check_name": [], "note": []}), "10_drug_diagnostics")

save_timing("10_drug_diagnostics", t.elapsed())

print(f"Done in {t.elapsed():.2f} seconds")

if __name__ == "__main__":

main()

Copilot · 2026-04-26T09:55:26Z

+## Prerequisites
+
+- **Python** with OMOPy installed (`uv sync`)
+- **R** 4.5+ available on `PATH`
+- **Java** 11+ (for Synthea data generation)
+


The prerequisites list says Java 11+ is required for Synthea data generation, but benchmarks/generate_synthea_1k.R downloads a pre-built Eunomia dataset and doesn’t invoke Synthea/Java. Please update the prerequisites (and any other references) so users don’t install Java unnecessarily.

Copilot · 2026-04-26T09:55:26Z

+connect_cdm <- function() {
+  con <- dbConnect(duckdb(), dbdir = DB_PATH, read_only = TRUE)
+  cdm <- cdmFromCon(
+    con = con,
+    cdmSchema = "main",
+    writeSchema = "main",
+    cdmName = "synthea_1k"
+  )
+  return(list(con = con, cdm = cdm))


connect_cdm() opens DuckDB with read_only = TRUE, but these benchmarks generate cohorts / write results into schemas (e.g., generateConceptCohortSet() in multiple scripts). If these helpers are used later, this setting will break the benchmarks. Either remove read_only=TRUE (or make it a parameter) and update scripts to use connect_cdm(), or drop the unused helper functions to avoid a misleading default.

Copilot · 2026-04-26T09:55:26Z

+## How to Reproduce
+
+```bash
+# 1. Generate the test database (requires R + Java)


The reproduction instructions claim generating the test DB requires “R + Java”, but the generator script downloads an existing Eunomia dataset and doesn’t use Java. Please update this line to avoid misleading setup requirements.

Suggested change

# 1. Generate the test database (requires R + Java)

# 1. Generate the test database (requires R)

Copilot · 2026-04-26T09:55:28Z

+The comparison report is written to `docs/comparison.md` and appears in
+the mkdocs site under **Project → R vs Python Comparison**.


This README says the report “appears in the mkdocs site under Project → R vs Python Comparison”, but mkdocs.yml currently doesn’t list comparison.md (or r-package-mapping.md) under the Project nav. Unless mkdocs is intentionally relying on orphan pages, this claim is currently inaccurate—either add these pages to the nav or adjust the wording.

Suggested change

The comparison report is written to `docs/comparison.md` and appears in

the mkdocs site under **Project → R vs Python Comparison**.

The comparison report is written to `docs/comparison.md` and can be

included in the mkdocs site if that page is added to the navigation.

Copilot · 2026-04-26T09:55:28Z

+def read_csv_rows(path: Path) -> list[dict]:
+    if not path.exists():
+        return []
+    with open(path, encoding="utf-8") as f:
+        return list(csv.DictReader(f))
+
+
+def read_timing(results_dir: Path, name: str) -> str:
+    rows = read_csv_rows(results_dir / f"{name}_timing.csv")
+    if rows:
+        return f"{float(rows[0]['elapsed_s']):.2f}s"
+    return "—"
+
+
+def count_rows(results_dir: Path, name: str) -> str:
+    path = results_dir / f"{name}.csv"
+    if not path.exists():
+        return "—"
+    with open(path) as f:
+        return str(sum(1 for _ in f) - 1)
+


CSV files are opened without newline="" (and count_rows() also omits an explicit encoding). For cross-platform correctness with the csv module (especially on Windows), open CSVs with encoding="utf-8" and newline="" consistently.

Copilot · 2026-04-26T09:55:28Z

+import sys; sys.path.insert(0, "benchmarks/python")
+from helpers import connect_cdm, save_result, save_timing, Timer
+
+print("=== 04: Cohort Characteristics ===")
+t = Timer()
+cdm = connect_cdm()
+
+from omopy.generics import Codelist
+from omopy.connector import generate_concept_cohort_set
+from omopy.characteristics import summarise_characteristics
+
+cdm = generate_concept_cohort_set(cdm, Codelist({"coronary_artery": [317576]}), name="target_cohort")
+cohort = cdm["target_cohort"]
+
+result = summarise_characteristics(cohort)
+save_result(result.data, "04_characteristics")
+save_timing("04_characteristics", t.elapsed())
+print(f"Done in {t.elapsed():.2f} seconds")


This benchmark script also uses sys.path mutation and then performs module-level imports after executable statements, which will fail Ruff under the repo’s pre-commit config (E702/E402). Please refactor to a path-stable helper import and keep imports at the top (or move runtime code under main()).

Suggested change

import sys; sys.path.insert(0, "benchmarks/python")

from helpers import connect_cdm, save_result, save_timing, Timer

print("=== 04: Cohort Characteristics ===")

t = Timer()

cdm = connect_cdm()

from omopy.generics import Codelist

from omopy.connector import generate_concept_cohort_set

from omopy.characteristics import summarise_characteristics

cdm = generate_concept_cohort_set(cdm, Codelist({"coronary_artery": [317576]}), name="target_cohort")

cohort = cdm["target_cohort"]

result = summarise_characteristics(cohort)

save_result(result.data, "04_characteristics")

save_timing("04_characteristics", t.elapsed())

print(f"Done in {t.elapsed():.2f} seconds")

from importlib import util

from pathlib import Path

from omopy.characteristics import summarise_characteristics

from omopy.connector import generate_concept_cohort_set

from omopy.generics import Codelist

def _load_helpers_module():

helpers_path = Path(__file__).resolve().parent / "helpers.py"

spec = util.spec_from_file_location("benchmark_helpers", helpers_path)

if spec is None or spec.loader is None:

raise ImportError(f"Unable to load helpers module from {helpers_path}")

module = util.module_from_spec(spec)

spec.loader.exec_module(module)

return module

_helpers = _load_helpers_module()

connect_cdm = _helpers.connect_cdm

save_result = _helpers.save_result

save_timing = _helpers.save_timing

Timer = _helpers.Timer

def main():

print("=== 04: Cohort Characteristics ===")

t = Timer()

cdm = connect_cdm()

cdm = generate_concept_cohort_set(

cdm,

Codelist({"coronary_artery": [317576]}),

name="target_cohort",

)

cohort = cdm["target_cohort"]

result = summarise_characteristics(cohort)

save_result(result.data, "04_characteristics")

save_timing("04_characteristics", t.elapsed())

print(f"Done in {t.elapsed():.2f} seconds")

if __name__ == "__main__":

main()

Copilot · 2026-04-26T09:55:28Z

+import sys; sys.path.insert(0, "benchmarks/python")
+from helpers import connect_cdm, save_result, save_timing, Timer
+
+print("=== 05: Incidence ===")
+t = Timer()
+cdm = connect_cdm()
+
+from omopy.generics import Codelist
+from omopy.connector import generate_concept_cohort_set
+from omopy.incidence import generate_denominator_cohort_set, estimate_incidence
+


This benchmark script has module-level imports after executable statements and a one-line sys.path edit; Ruff will flag this (E702/E402). Refactor so imports are at the top (and avoid sys.path hacks by loading helpers relative to __file__ or by packaging the benchmarks).

Suggested change

import sys; sys.path.insert(0, "benchmarks/python")

from helpers import connect_cdm, save_result, save_timing, Timer

print("=== 05: Incidence ===")

t = Timer()

cdm = connect_cdm()

from omopy.generics import Codelist

from omopy.connector import generate_concept_cohort_set

from omopy.incidence import generate_denominator_cohort_set, estimate_incidence

from importlib.util import module_from_spec, spec_from_file_location

from pathlib import Path

from omopy.generics import Codelist

from omopy.connector import generate_concept_cohort_set

from omopy.incidence import generate_denominator_cohort_set, estimate_incidence

_HELPERS_PATH = Path(__file__).resolve().parent / "helpers.py"

_HELPERS_SPEC = spec_from_file_location("benchmark_helpers", _HELPERS_PATH)

if _HELPERS_SPEC is None or _HELPERS_SPEC.loader is None:

raise ImportError(f"Could not load helpers module from {_HELPERS_PATH}")

_HELPERS = module_from_spec(_HELPERS_SPEC)

_HELPERS_SPEC.loader.exec_module(_HELPERS)

connect_cdm = _HELPERS.connect_cdm

save_result = _HELPERS.save_result

save_timing = _HELPERS.save_timing

Timer = _HELPERS.Timer

print("=== 05: Incidence ===")

t = Timer()

cdm = connect_cdm()

Copilot · 2026-04-26T09:55:29Z

+import sys; sys.path.insert(0, "benchmarks/python")
+from helpers import connect_cdm, save_result, save_timing, Timer
+
+print("=== 06: Drug Utilisation ===")
+t = Timer()
+cdm = connect_cdm()
+
+from omopy.drug import generate_ingredient_cohort_set, summarise_drug_utilisation
+
+cdm = generate_ingredient_cohort_set(cdm, name="drug_cohort", ingredient="clopidogrel")
+cohort = cdm["drug_cohort"]
+
+result = summarise_drug_utilisation(cohort, ingredient_concept_id=1322184, gap_era=30)
+save_result(result.data, "06_drug_utilisation")
+save_timing("06_drug_utilisation", t.elapsed())
+print(f"Done in {t.elapsed():.2f} seconds")


Ruff will flag this file for the one-line sys.path edit and module-level imports after runtime statements (E702/E402). Please restructure to keep imports at the top and use a path-stable way to load helpers.py (or move delayed imports inside main()).

Suggested change

import sys; sys.path.insert(0, "benchmarks/python")

from helpers import connect_cdm, save_result, save_timing, Timer

print("=== 06: Drug Utilisation ===")

t = Timer()

cdm = connect_cdm()

from omopy.drug import generate_ingredient_cohort_set, summarise_drug_utilisation

cdm = generate_ingredient_cohort_set(cdm, name="drug_cohort", ingredient="clopidogrel")

cohort = cdm["drug_cohort"]

result = summarise_drug_utilisation(cohort, ingredient_concept_id=1322184, gap_era=30)

save_result(result.data, "06_drug_utilisation")

save_timing("06_drug_utilisation", t.elapsed())

print(f"Done in {t.elapsed():.2f} seconds")

import importlib.util

from pathlib import Path

from omopy.drug import generate_ingredient_cohort_set, summarise_drug_utilisation

_HELPERS_PATH = Path(__file__).resolve().parent / "helpers.py"

_HELPERS_SPEC = importlib.util.spec_from_file_location("benchmark_helpers", _HELPERS_PATH)

if _HELPERS_SPEC is None or _HELPERS_SPEC.loader is None:

raise ImportError(f"Unable to load helpers module from {_HELPERS_PATH}")

_HELPERS_MODULE = importlib.util.module_from_spec(_HELPERS_SPEC)

_HELPERS_SPEC.loader.exec_module(_HELPERS_MODULE)

connect_cdm = _HELPERS_MODULE.connect_cdm

save_result = _HELPERS_MODULE.save_result

save_timing = _HELPERS_MODULE.save_timing

Timer = _HELPERS_MODULE.Timer

def main():

print("=== 06: Drug Utilisation ===")

t = Timer()

cdm = connect_cdm()

cdm = generate_ingredient_cohort_set(cdm, name="drug_cohort", ingredient="clopidogrel")

cohort = cdm["drug_cohort"]

result = summarise_drug_utilisation(cohort, ingredient_concept_id=1322184, gap_era=30)

save_result(result.data, "06_drug_utilisation")

save_timing("06_drug_utilisation", t.elapsed())

print(f"Done in {t.elapsed():.2f} seconds")

if __name__ == "__main__":

main()

vvcb · 2026-04-26T10:39:33Z

@copilot apply changes based on the comments in this thread

Copilot

Pull request overview

Copilot reviewed 31 out of 33 changed files in this pull request and generated 23 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-26T11:00:13Z

@@ -0,0 +1,34 @@
+"""Benchmark 10: Drug Exposure Diagnostics — omopy.drug_diagnostics.execute_checks()"""
+import sys; sys.path.insert(0, "benchmarks/python")


These benchmark scripts don’t need to modify sys.path to import helpers (Python automatically adds the script’s directory to sys.path). Keeping import sys; ... on one line will also fail Ruff/formatting pre-commit hooks; remove it and keep imports in standard, top-of-file blocks.

Suggested change

import sys; sys.path.insert(0, "benchmarks/python")

Copilot · 2026-04-26T11:00:13Z

+        for metric_name, r_val, p_val in metrics:
+            # Metrics ending with _info are informational, not pass/fail
+            is_info = metric_name.endswith("_info")
+            display_name = metric_name.removesuffix("_info")
+            icon = _match_icon(r_val, p_val, info=is_info)
+            total_checks += 1
+            if icon in ("✅", "≈", "ℹ️"):
+                total_pass += 1
+            lines.append(f"| {display_name} | {_fmt(r_val)} | {_fmt(p_val)} | {icon} |")


generate() currently counts informational metrics (ℹ️) as “passed” in the concordance summary, which can inflate the pass rate and blur the distinction between expected format differences and true value matches. Consider excluding _info metrics from total_checks/total_pass (or tracking a separate “informational” count) and only treating ✅/≈ as pass for the headline percentage.

Copilot · 2026-04-26T11:00:14Z

@@ -0,0 +1,19 @@
+"""Benchmark 04: Cohort Characteristics — omopy.characteristics.summarise_characteristics()"""
+import sys; sys.path.insert(0, "benchmarks/python")


These benchmark scripts don’t need to modify sys.path to import helpers (Python automatically adds the script’s directory to sys.path). Keeping import sys; ... on one line will also fail Ruff/formatting pre-commit hooks; remove it and keep imports in standard, top-of-file blocks.

Suggested change

import sys; sys.path.insert(0, "benchmarks/python")

Copilot · 2026-04-26T11:00:14Z

@@ -0,0 +1,26 @@
+"""Benchmark 05: Incidence — omopy.incidence.estimate_incidence()"""
+import sys; sys.path.insert(0, "benchmarks/python")


These benchmark scripts don’t need to modify sys.path to import helpers (Python automatically adds the script’s directory to sys.path). Keeping import sys; ... on one line will also fail Ruff/formatting pre-commit hooks; remove it and keep imports in standard, top-of-file blocks.

Suggested change

import sys; sys.path.insert(0, "benchmarks/python")

Copilot · 2026-04-26T11:00:14Z

@@ -0,0 +1,29 @@
+"""Benchmark 07: Cohort Survival — omopy.survival.estimate_single_event_survival()"""
+import sys; sys.path.insert(0, "benchmarks/python")


These benchmark scripts don’t need to modify sys.path to import helpers (Python automatically adds the script’s directory to sys.path). Keeping import sys; ... on one line will also fail Ruff/formatting pre-commit hooks; remove it and keep imports in standard, top-of-file blocks.

Suggested change

import sys; sys.path.insert(0, "benchmarks/python")

Copilot · 2026-04-26T11:00:17Z

+
+print("=== 04: Cohort Characteristics ===")
+t = Timer()
+cdm = connect_cdm()
+
+from omopy.generics import Codelist
+from omopy.connector import generate_concept_cohort_set
+from omopy.characteristics import summarise_characteristics
+


Module imports (from omopy...) are currently placed after runtime statements (print, connect_cdm()), which triggers Ruff E402 and makes dependency failures happen mid-script. Move all imports to the top of the file (after the docstring) so the script fails fast and stays lint-clean.

Suggested change

print("=== 04: Cohort Characteristics ===")

t = Timer()

cdm = connect_cdm()

from omopy.generics import Codelist

from omopy.connector import generate_concept_cohort_set

from omopy.characteristics import summarise_characteristics

from omopy.generics import Codelist

from omopy.connector import generate_concept_cohort_set

from omopy.characteristics import summarise_characteristics

print("=== 04: Cohort Characteristics ===")

t = Timer()

cdm = connect_cdm()

Copilot · 2026-04-26T11:00:17Z

+
+print("=== 06: Drug Utilisation ===")
+t = Timer()
+cdm = connect_cdm()
+
+from omopy.drug import generate_ingredient_cohort_set, summarise_drug_utilisation
+


Module imports (from omopy...) are currently placed after runtime statements (print, connect_cdm()), which triggers Ruff E402 and makes dependency failures happen mid-script. Move all imports to the top of the file (after the docstring) so the script fails fast and stays lint-clean.

Suggested change

print("=== 06: Drug Utilisation ===")

t = Timer()

cdm = connect_cdm()

from omopy.drug import generate_ingredient_cohort_set, summarise_drug_utilisation

from omopy.drug import generate_ingredient_cohort_set, summarise_drug_utilisation

print("=== 06: Drug Utilisation ===")

t = Timer()

cdm = connect_cdm()

Copilot · 2026-04-26T11:00:17Z

+
+print("=== 07: Survival ===")
+t = Timer()
+cdm = connect_cdm()
+
+from omopy.generics import Codelist
+from omopy.connector import generate_concept_cohort_set
+from omopy.survival import estimate_single_event_survival
+


Module imports (from omopy...) are currently placed after runtime statements (print, connect_cdm()), which triggers Ruff E402 and makes dependency failures happen mid-script. Move all imports to the top of the file (after the docstring) so the script fails fast and stays lint-clean.

Suggested change

print("=== 07: Survival ===")

t = Timer()

cdm = connect_cdm()

from omopy.generics import Codelist

from omopy.connector import generate_concept_cohort_set

from omopy.survival import estimate_single_event_survival

from omopy.generics import Codelist

from omopy.connector import generate_concept_cohort_set

from omopy.survival import estimate_single_event_survival

print("=== 07: Survival ===")

t = Timer()

cdm = connect_cdm()

Copilot · 2026-04-26T11:00:18Z

+from helpers import connect_cdm, save_result, save_timing, Timer
+
+print("=== 08: Codelist ===")
+t = Timer()
+cdm = connect_cdm()
+
+from omopy.codelist import get_candidate_codes
+
+codes = get_candidate_codes(cdm, keywords=["coronary"], domains=["Condition"], include_descendants=True)
+# codes is a Codelist (dict-like) — flatten to a DataFrame
+import polars as pl


import polars as pl is inside the script body and from omopy... imports come after runtime statements, which will trigger Ruff E402. Move all imports to the top of the file (after the docstring) and keep them grouped.

Suggested change

from helpers import connect_cdm, save_result, save_timing, Timer

print("=== 08: Codelist ===")

t = Timer()

cdm = connect_cdm()

from omopy.codelist import get_candidate_codes

codes = get_candidate_codes(cdm, keywords=["coronary"], domains=["Condition"], include_descendants=True)

# codes is a Codelist (dict-like) — flatten to a DataFrame

import polars as pl

import polars as pl

from helpers import connect_cdm, save_result, save_timing, Timer

from omopy.codelist import get_candidate_codes

print("=== 08: Codelist ===")

t = Timer()

cdm = connect_cdm()

codes = get_candidate_codes(cdm, keywords=["coronary"], domains=["Condition"], include_descendants=True)

# codes is a Codelist (dict-like) — flatten to a DataFrame

Copilot · 2026-04-26T11:00:18Z

+
+print("=== 09: Treatment Patterns ===")
+t = Timer()
+cdm = connect_cdm()
+
+from omopy.generics import Codelist
+from omopy.connector import generate_concept_cohort_set
+from omopy.treatment import compute_pathways, summarise_treatment_pathways, CohortSpec
+


Module imports (from omopy...) are currently placed after runtime statements (print, connect_cdm()), which triggers Ruff E402 and makes dependency failures happen mid-script. Move all imports to the top of the file (after the docstring) so the script fails fast and stays lint-clean.

Suggested change

print("=== 09: Treatment Patterns ===")

t = Timer()

cdm = connect_cdm()

from omopy.generics import Codelist

from omopy.connector import generate_concept_cohort_set

from omopy.treatment import compute_pathways, summarise_treatment_pathways, CohortSpec

from omopy.generics import Codelist

from omopy.connector import generate_concept_cohort_set

from omopy.treatment import compute_pathways, summarise_treatment_pathways, CohortSpec

print("=== 09: Treatment Patterns ===")

t = Timer()

cdm = connect_cdm()

iwfarr · 2026-04-27T11:42:32Z

commits address comments above.

Fixed sys.path and imports in Python scripts
Fixed exception syntax and updated compare.py
Updated Rscript paths and removed read_only flag
Updated docs (Darwin EU, mkdocs config)
Applied Ruff formatting and fixed end of file newlines

vvcb · 2026-04-27T12:43:52Z

@copilot resolve the merge conflicts in this pull request

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

vvcb requested a review from Copilot April 26, 2026 09:16

Copilot started reviewing on behalf of vvcb April 26, 2026 09:16 View session

Copilot AI reviewed Apr 26, 2026

View reviewed changes

Comment thread benchmarks/README.md Outdated

Comment thread benchmarks/r/00_helpers.R Outdated

Comment thread benchmarks/r/02_cohort_generation.R Outdated

Comment thread benchmarks/python/01_snapshot.py Outdated

Comment thread docs/comparison.md Outdated

vvcb requested review from Copilot and vvcb April 26, 2026 09:50

Copilot started reviewing on behalf of vvcb April 26, 2026 09:50 View session

Copilot AI reviewed Apr 26, 2026

View reviewed changes

vvcb requested a review from Copilot April 26, 2026 10:51

Copilot started reviewing on behalf of vvcb April 26, 2026 10:51 View session

Copilot AI reviewed Apr 26, 2026

View reviewed changes

iwfarr and others added 3 commits April 27, 2026 14:23

benchmark comparison code and report

882ee05

resolve typos and ensure ruff checks are passed.

c4cb0c9

Apply suggestions from Copilot code review

2de4f28

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

iwfarr force-pushed the if-module-comparison branch from 3ad1ec1 to 2de4f28 Compare April 27, 2026 13:34

vvcb merged commit 4a9fd43 into fastomop:main Apr 30, 2026
3 checks passed

iwfarr deleted the if-module-comparison branch May 1, 2026 10:05

	# 1. Generate the test database (requires R + Java)
	# 1. Generate the test database (requires R)

		The comparison report is written to `docs/comparison.md` and appears in
		the mkdocs site under Project → R vs Python Comparison.

		@@ -0,0 +1,34 @@
		"""Benchmark 10: Drug Exposure Diagnostics — omopy.drug_diagnostics.execute_checks()"""
		import sys; sys.path.insert(0, "benchmarks/python")

		@@ -0,0 +1,19 @@
		"""Benchmark 04: Cohort Characteristics — omopy.characteristics.summarise_characteristics()"""
		import sys; sys.path.insert(0, "benchmarks/python")

		@@ -0,0 +1,26 @@
		"""Benchmark 05: Incidence — omopy.incidence.estimate_incidence()"""
		import sys; sys.path.insert(0, "benchmarks/python")

		@@ -0,0 +1,29 @@
		"""Benchmark 07: Cohort Survival — omopy.survival.estimate_single_event_survival()"""
		import sys; sys.path.insert(0, "benchmarks/python")

Uh oh!

Conversation

iwfarr commented Apr 23, 2026

Uh oh!

iwfarr commented Apr 23, 2026

Benchmark scripts (28 files)

Documentation (4 files)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

vvcb commented Apr 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 26, 2026