Skip to content

Use DuckDB for the default DuckLake catalog - #91

Merged
viggy28 merged 1 commit into
mainfrom
feat/duckdb-default-ducklake-catalog
Sep 4, 2026
Merged

Use DuckDB for the default DuckLake catalog#91
viggy28 merged 1 commit into
mainfrom
feat/duckdb-default-ducklake-catalog

Conversation

@viggy28

@viggy28 viggy28 commented Sep 4, 2026

Copy link
Copy Markdown
Owner

Closes #90.

What changed

  • Change the default DuckLake catalog store from sqlite to duckdb.
  • Change the default metadata path to ~/.streambed/ducklake-catalog.duckdb.
  • Preserve explicit SQLite catalog support through CLI flags and environment variables.
  • Document the catalog/data split and the upgrade procedure for legacy implicit SQLite catalogs.
  • Strengthen config tests to assert the exact default path/store and explicit SQLite environment overrides.

Upgrade behavior

This is an intentional default change rather than an automatic migration. Existing installations that used the previous implicit SQLite catalog must explicitly configure both legacy values:

--ducklake-catalog="$HOME/.streambed/ducklake-catalog.sqlite" \
--ducklake-catalog-store=sqlite

Streambed does not convert or overwrite the old catalog. Explicit CLI flags and STREAMBED_DUCKLAKE_CATALOG* environment variables continue to override defaults.

Validation

go test ./config/... -count=1
go test ./internal/... ./config/... -count=1
go build -o /tmp/streambed ./cmd/streambed
/tmp/streambed sync --help

The generated CLI help reports .duckdb and duckdb as the defaults.

@viggy28
viggy28 merged commit 38a2971 into main Sep 4, 2026
3 of 4 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dbe136d9d4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread config/config.go
TargetFormat: "iceberg",
DuckLakeCatalog: defaultDuckLakeCatalogPath(),
DuckLakeCatalogStore: "sqlite",
DuckLakeCatalogStore: "duckdb",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep the CDC oracle on a compatible catalog store

The required integration-smoke workflow runs TestCDCDataIntegrityOracle, whose DuckLake sync subprocess passes a catalog path but no store, so this new default makes the subprocess create a native DuckDB catalog. Its checkpoint reader concurrently opens the same path with an empty internal ducklake.Config.CatalogStore, which normalizes to SQLite; the first checkpoint therefore cannot attach the catalog (and switching that reader to DuckDB would encounter DuckDB's cross-process writer lock). Explicitly keep this live-reader test on SQLite or restructure it so the writer is stopped before opening a native DuckDB reader.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use DuckDB as the default DuckLake catalog store

1 participant