Skip to content

Replace mattn/go-sqlite3 with modernc.org/sqlite (drop cgo dependency) #79

Description

@e7217

Motivation

EDG Core currently depends on github.com/mattn/go-sqlite3 for its metadata store. go-sqlite3 is a cgo wrapper around the SQLite C library, which means:

  1. Windows builds require a gcc toolchain (mingw-w64 or equivalent). Default go build on Windows ships without cgo, producing a binary that fails at runtime:

    Failed to create store: ... go-sqlite3 requires cgo to work. This is a stub
    

    Field engineers spinning up edg-core on a Windows IPC, devs running quick smoke tests, or CI runners that don't have gcc installed all hit this wall.

  2. Cross-compilation is awkward. GOOS=linux go build from a Windows host without a Linux gcc cross-toolchain silently produces the same broken stub.

  3. CGO_ENABLED=1 leaks into downstream Dockerfiles, CI matrices, and release pipelines. Every contributor environment needs to be cgo-capable.

Discovered during Beckhoff adapter smoke testing on Windows — edg-core started NATS successfully but exited immediately when initializing the metadata store.

Proposed change

Replace the SQLite driver with modernc.org/sqlite, a pure-Go transpilation of the SQLite C source. No cgo required.

Scope

  • go.mod: drop github.com/mattn/go-sqlite3, add modernc.org/sqlite
  • internal/core/store*.go (and anywhere else using sql.Open("sqlite3", ...)): change driver name to "sqlite"
  • golang-migrate driver: swap github.com/golang-migrate/migrate/v4/database/sqlite3.../sqlite (the modernc-compatible package)
  • PRAGMA / DSN compatibility audit (foreign keys, WAL mode, busy timeout) — modernc.org/sqlite follows SQLite-standard PRAGMAs but some _pragma= DSN extensions differ
  • Run full test suite locally and in CI (Windows + Linux)
  • Re-test migration up/down paths against an existing metadata.db produced by mattn/go-sqlite3

Out of scope

  • Adapter SDK code (no sqlite usage)
  • Migration of any production deployments — the on-disk SQLite format is identical, so existing metadata.db files keep working.

Risk

  • Performance: modernc.org/sqlite is ~10-30% slower than cgo in microbenchmarks. EDG Core uses sqlite for asset metadata (low write rate, mostly read), so this should not matter in practice. Worth a quick benchmark before merging.
  • Subtle SQL behaviour differences: Both drivers target the same SQLite engine version, so feature coverage matches; differences are usually in driver-level features (custom functions, connection hooks) which EDG Core doesn't currently use.
  • golang-migrate driver maturity: The modernc.org-backed migrate driver is less battle-tested than the mattn variant. Verify by running all existing migrations on a fresh DB.

Acceptance criteria

  • go build ./... succeeds on Windows with no gcc toolchain installed.
  • edg-core -config deploy/configs/core/config.dev.yaml starts cleanly on Windows and creates metadata.db.
  • All existing unit/integration tests pass on Linux and Windows.
  • A fresh metadata.db from this branch is byte-readable by sqlite3 CLI (rules out exotic file format issues).

Workaround until then

Install mingw-w64 (e.g. via winget install BrechtSanders.WinLibs.POSIX.UCRT) and build with CGO_ENABLED=1. This is what we're doing in the Beckhoff smoke test setup currently.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions