Add clickhouse docs

.env.example

@@ -31,10 +31,11 @@ export TESTS_BIGQUERY_SA_KEY_PATH=
 #   docker run -d --name clickhouse -p 8123:8123 -p 9001:9000 clickhouse/clickhouse-server
 export TESTS_CLICKHOUSE_URL=http://localhost:8123
 #
-# USER / PASSWORD: ClickHouse credentials. The default server has no auth,
-#   but the test harness creates a dedicated user. Typical local values:
-export TESTS_CLICKHOUSE_USER=default
-export TESTS_CLICKHOUSE_PASSWORD=
+# USER / PASSWORD / DATABASE: ClickHouse credentials and target database.
+# The values below match the local Docker Compose service.
+export TESTS_CLICKHOUSE_USER=etl
+export TESTS_CLICKHOUSE_PASSWORD=etl
+export TESTS_CLICKHOUSE_DATABASE=default
 
 # Snowflake tests, examples, and benchmarks.
 # See crates/etl-destinations/src/snowflake/README.md.

README.md

@@ -204,7 +204,7 @@ one of the modules shipped in `etl-destinations`.
 | Feature | Destination | Status | Notes |
 | --- | --- | --- | --- |
 | `bigquery` | Google BigQuery | Stable | Full CRUD-capable replication for analytics workloads. |
-| `clickhouse` | ClickHouse | In progress | Columnar OLAP replication with current-state or append-only layouts. |
+| `clickhouse` | ClickHouse | Closed beta | Columnar OLAP replication with current-state or append-only layouts. |
 | `ducklake` | DuckLake | In progress | Open data lake replication with local or S3-compatible storage. |
 | `iceberg` | Apache Iceberg | Deprecated for now | The module remains available, but new deployments should prefer BigQuery or DuckLake. |
 | `snowflake` | Snowflake | In progress | Cloud data warehouse replication example and destination module. |

crates/etl-destinations/README.md

@@ -7,7 +7,7 @@ Enable the destination modules you need with crate features:
 | Feature | Destination | Status |
 | --- | --- | --- |
 | `bigquery` | Google BigQuery | Stable |
-| `clickhouse` | ClickHouse | In progress |
+| `clickhouse` | ClickHouse | Closed beta |
 | `ducklake` | DuckLake | In progress |
 | `iceberg` | Apache Iceberg | Deprecated for now |
 | `snowflake` | Snowflake | In progress |

crates/etl-destinations/src/bigquery/test_utils.rs

@@ -25,8 +25,8 @@ use crate::{
 /// Maximum number of times we re-run a verification query.
 ///
 /// Sized generously because a view dropped and recreated under the same name
-/// can serve stale NOT_FOUND responses well past the first few seconds.
-const BIGQUERY_QUERY_MAX_ATTEMPTS: u32 = 120;
+/// can serve stale metadata for several minutes.
+const BIGQUERY_QUERY_MAX_ATTEMPTS: u32 = 600;
 /// Maximum number of times we poll for a table to report zero rows.
 ///
 /// Kept short: the expected end state is observable as soon as the deletes

crates/etl-destinations/src/clickhouse/README.md

@@ -0,0 +1,158 @@
+# ClickHouse Destination
+
+> **Status: Closed beta.** ClickHouse is a closed beta destination.
+> Availability is limited while the integration stabilizes.
+
+## Requirements
+
+- ClickHouse **23.5 or newer** is required for the default
+  `ReplacingMergeTree` engine.
+- The source table replica identity must be primary key or full.
+- The default `ReplacingMergeTree` engine requires a source primary key. The
+  `MergeTree` engine works for source tables without a primary key.
+
+## Running the example
+
+For the repository's local services, copy `.env.example` to `.env` and load
+it. Then initialize and seed the services:
+
+```bash
+source .env
+cargo x init
+cargo x seed
+```
+
+Run the ClickHouse example directly with Cargo:
+
+```bash
+cargo run -p etl-examples --bin clickhouse --features clickhouse -- \
+    --db-host "$TESTS_DATABASE_HOST" \
+    --db-port "$TESTS_DATABASE_PORT" \
+    --db-name etl_testdata \
+    --db-username "$TESTS_DATABASE_USERNAME" \
+    --publication seed_pub
+```
+
+Both passwords come from the variables loaded from `.env`.
+
+Alternatively, use the xtask wrapper. It reads the `TESTS_DATABASE_*` and
+`TESTS_CLICKHOUSE_*` variables and supplies the local database and publication
+defaults:
+
+```bash
+cargo x example clickhouse
+```
+
+## Table engines
+
+The destination supports two layouts. Select one per pipeline with
+`--clickhouse-engine`:
+
+| Flag value                       | Engine               | Use it for                                              |
+| -------------------------------- | -------------------- | ------------------------------------------------------- |
+| `replacing_merge_tree` (default) | `ReplacingMergeTree` | Current-state replicas. Source must have a primary key. |
+| `merge_tree`                     | `MergeTree`          | Append-only event log. Works for PK-less source tables. |
+
+Table names derive from the Postgres schema and table name. They use
+double-underscore escaping. For example, `public.orders` becomes
+`public_orders`, and `my_schema.t` becomes `my__schema_t`.
+
+### ReplacingMergeTree (default)
+
+Each replicated table uses
+`ReplacingMergeTree(_etl_version, _etl_deleted)`, keyed on the source primary
+key. Two trailing columns control deduplication and tombstone handling:
+
+- `_etl_version UInt128` -- the packed Postgres event sequence key:
+  `(commit_lsn << 64) | tx_ordinal`. Higher values win during a `FINAL` merge.
+  Thus, the latest event for each primary key wins. The commit LSN and the
+  in-transaction ordinal give a total order for all events. This includes
+  multiple row events that share a WAL record.
+- `_etl_deleted UInt8` -- tombstone flag. `1` for DELETE events and `0` for
+  other events.
+
+The destination also creates a `<table>__current` view for each table. This
+view hides the `ReplacingMergeTree` internals:
+
+```sql
+CREATE VIEW IF NOT EXISTS "public_orders__current" AS
+SELECT <user columns>
+FROM "public_orders" FINAL
+WHERE _etl_deleted = 0
+```
+
+Read patterns:
+
+- Use the `__current` view for current-state queries.
+- Or query the base table directly:
+
+  ```sql
+  SELECT <user columns>
+  FROM "public_orders" FINAL
+  WHERE _etl_deleted = 0
+  ```
+
+`OPTIMIZE` guidance:
+
+- The replicator never runs `OPTIMIZE ... FINAL CLEANUP`. Background merges
+  collapse duplicates over time. Operators control physical tombstone removal.
+- To reclaim deleted rows on disk, run
+  `OPTIMIZE TABLE "<table>" FINAL CLEANUP` on a schedule that matches your
+  retention requirements.
+
+### MergeTree
+
+Each replicated table uses `MergeTree() ORDER BY tuple()`. Two CDC metadata
+columns follow each row:
+
+- `cdc_operation`: `INSERT`, `UPDATE`, or `DELETE`.
+- `cdc_lsn`: the Postgres commit LSN at the time of the change.
+
+Read patterns:
+
+- For current state by primary key, take the latest event by `cdc_lsn` with
+  `LIMIT 1 BY`. Then filter out tombstones:
+
+  ```sql
+  SELECT <user columns> FROM (
+      SELECT * FROM "public_orders"
+      ORDER BY cdc_lsn DESC LIMIT 1 BY (id)
+  )
+  WHERE cdc_operation != 'DELETE'
+  ```
+
+- For event log queries, read the table directly. The table keeps every CDC
+  event.
+
+## Connection notes
+
+For HTTPS connections, provide an `https://` URL. TLS uses webpki root
+certificates automatically.
+
+Set `TESTS_CLICKHOUSE_PASSWORD` when ClickHouse requires authentication. The
+example reads this variable directly, so the secret does not appear in process
+arguments. The `--clickhouse-password` flag remains available for one-off local
+runs.
+
+## CLI flags
+
+| Flag                           | Default                | Description                                                   |
+| ------------------------------ | ---------------------- | ------------------------------------------------------------- |
+| `--db-host`                    | _(required)_           | Postgres host                                                 |
+| `--db-port`                    | _(required)_           | Postgres port (`u16`)                                         |
+| `--db-name`                    | _(required)_           | Postgres database name                                        |
+| `--db-username`                | _(required)_           | Postgres user (must have REPLICATION)                         |
+| `--db-password`                | _(optional)_           | Password; env: `TESTS_DATABASE_PASSWORD`                      |
+| `--clickhouse-url`             | _(required)_           | HTTP(S) endpoint; env: `TESTS_CLICKHOUSE_URL`                 |
+| `--clickhouse-user`            | _(required)_           | User name; env: `TESTS_CLICKHOUSE_USER`                       |
+| `--clickhouse-password`        | _(optional)_           | Password; env: `TESTS_CLICKHOUSE_PASSWORD`                    |
+| `--clickhouse-database`        | `default`              | Target database; env: `TESTS_CLICKHOUSE_DATABASE`             |
+| `--clickhouse-engine`          | `replacing_merge_tree` | Table engine: `replacing_merge_tree` or `merge_tree`          |
+| `--max-batch-fill-duration-ms` | `5000`                 | Max time to wait before flushing a batch                      |
+| `--max-table-sync-workers`     | `4`                    | Concurrent workers during initial copy                        |
+| `--publication`                | _(required)_           | Postgres publication name                                     |
+
+## Metrics
+
+See [`./METRICS.md`](./METRICS.md) for the metrics that the ClickHouse
+destination emits.

crates/etl-examples/README.md

@@ -7,7 +7,7 @@ This crate contains practical examples demonstrating how to replicate data from
 | Example                         | Binary       | Feature      | Destination                                | Status      |
 | ------------------------------- | ------------ | ------------ | ------------------------------------------ | ----------- |
 | [BigQuery](#bigquery)           | `bigquery`   | `bigquery`   | Google BigQuery (cloud data warehouse)     | Stable      |
-| [ClickHouse](#clickhouse-setup) | `clickhouse` | `clickhouse` | ClickHouse (column-oriented OLAP database) | In progress |
+| [ClickHouse](#clickhouse-setup) | `clickhouse` | `clickhouse` | ClickHouse (column-oriented OLAP database) | Closed beta |
 | [DuckLake](#ducklake)           | `ducklake`   | `ducklake`   | DuckLake (open data lake format)           | In progress |
 | [Snowflake](#snowflake)         | `snowflake`  | `snowflake`  | Snowflake (cloud data warehouse)           | In progress |
 
@@ -158,108 +158,6 @@ cargo run --bin ducklake -p etl-examples --features ducklake -- \
     --s3-secret-access-key my-secret-key
 ```
 
-## ClickHouse Setup
-
-To run the ClickHouse example, you'll need a running ClickHouse instance accessible over HTTP(S).
-ClickHouse **23.5 or newer** is required for the default `ReplacingMergeTree` engine.
-
-Create a publication in Postgres:
-
-```sql
-create publication my_pub
-for table table1, table2;
-```
-
-Then run the ClickHouse example:
-
-```bash
-cargo run -p etl-examples --bin clickhouse --features clickhouse -- \
-        --db-host localhost \
-        --db-port 5432 \
-        --db-name postgres \
-        --db-username postgres \
-        --db-password password \
-        --clickhouse-url http://localhost:8123 \
-        --clickhouse-user default \
-        --clickhouse-database default \
-        --publication my_pub
-```
-
-### Table engines
-
-The destination supports two layouts, chosen per pipeline via `--clickhouse-engine`:
-
-| Flag value                       | Engine               | Use it for                                              |
-| -------------------------------- | -------------------- | ------------------------------------------------------- |
-| `replacing_merge_tree` (default) | `ReplacingMergeTree` | Current-state replicas. Source must have a primary key. |
-| `merge_tree`                     | `MergeTree`          | Append-only event log. Works for PK-less source tables. |
-
-Table names are derived from the Postgres schema and table name using double-underscore
-escaping (e.g. `public.orders` -> `public_orders`, `my_schema.t` -> `my__schema_t`).
-
-#### ReplacingMergeTree (default)
-
-Each replicated table is created as `ReplacingMergeTree(_etl_version, _etl_deleted)` keyed
-on the source primary key. Two trailing columns drive dedup and tombstone handling:
-
-- `_etl_version UInt128` -- the packed Postgres event sequence key:
-  `(commit_lsn << 64) | tx_ordinal`. Higher values win during a `FINAL` merge, so the
-  latest event per primary key wins. Encoding both the commit LSN and the in-transaction
-  ordinal gives a total order across all events, including multiple row events that
-  share a WAL record.
-- `_etl_deleted UInt8` -- tombstone flag. `1` for DELETE events, `0` otherwise.
-
-Alongside each table, the destination also creates a `<table>__current` view that hides
-the ReplacingMergeTree internals:
-
-```sql
-CREATE VIEW IF NOT EXISTS "public_orders__current" AS
-SELECT <user columns>
-FROM "public_orders" FINAL
-WHERE _etl_deleted = 0
-```
-
-Read patterns:
-
-- Prefer the `__current` view for current-state queries.
-- Or query the base table with `SELECT ... FROM "public_orders" FINAL WHERE _etl_deleted = 0`
-  directly.
-
-`OPTIMIZE` guidance:
-
-- The replicator never runs `OPTIMIZE ... FINAL CLEANUP`. Background merges already
-  collapse duplicates over time; physical removal of tombstones is operator-driven.
-- To reclaim deleted rows on disk, run `OPTIMIZE TABLE "<table>" FINAL CLEANUP` on a
-  schedule that matches your retention requirements.
-
-#### MergeTree
-
-Each replicated table is created as `MergeTree() ORDER BY tuple()` with two CDC metadata
-columns appended to every row:
-
-- `cdc_operation`: `INSERT`, `UPDATE`, or `DELETE`
-- `cdc_lsn`: the Postgres commit LSN at the time of the change
-
-Read patterns:
-
-- Current state per primary key: take the latest event by `cdc_lsn` with `LIMIT 1 BY`,
-  then filter out tombstones. Example:
-
-  ```sql
-  SELECT <user columns> FROM (
-      SELECT * FROM "public_orders"
-      ORDER BY cdc_lsn DESC LIMIT 1 BY (id)
-  )
-  WHERE cdc_operation != 'DELETE'
-  ```
-
-- Event log queries: read the table directly; every CDC event is preserved.
-
-### Connection notes
-
-For HTTPS connections, provide an `https://` URL -- TLS is handled automatically using
-webpki root certificates. Use `--clickhouse-password` if your ClickHouse instance requires
-authentication.
 
 ### Example configuration
 
@@ -380,6 +278,47 @@ RUST_LOG=debug cargo run --bin ducklake -p etl-examples --features ducklake -- [
 
 ---
 
+## ClickHouse Setup
+
+**Status: Closed beta.** ClickHouse is a closed beta destination. Availability
+is limited while the integration stabilizes.
+
+See the [ClickHouse destination guide][clickhouse-guide] for full setup and
+engine details.
+
+[clickhouse-guide]: ../etl-destinations/src/clickhouse/README.md
+
+The ClickHouse example needs an HTTP(S) endpoint. ClickHouse **23.5 or newer**
+is required for the default `ReplacingMergeTree` engine.
+
+The repository's local setup creates the `seed_pub` publication. Load `.env`
+before the direct command so neither password appears in process arguments:
+
+```bash
+source .env
+cargo run -p etl-examples --bin clickhouse --features clickhouse -- \
+    --db-host "$TESTS_DATABASE_HOST" \
+    --db-port "$TESTS_DATABASE_PORT" \
+    --db-name etl_testdata \
+    --db-username "$TESTS_DATABASE_USERNAME" \
+    --publication seed_pub
+```
+
+### Table engines
+
+Select one layout per pipeline with `--clickhouse-engine`:
+
+| Flag value                       | Engine               | Use it for                                              |
+| -------------------------------- | -------------------- | ------------------------------------------------------- |
+| `replacing_merge_tree` (default) | `ReplacingMergeTree` | Current-state replicas. Source must have a primary key. |
+| `merge_tree`                     | `MergeTree`          | Append-only event log. Works for PK-less source tables. |
+
+Table names derive from the Postgres schema and table name. They use
+double-underscore escaping. For example, `public.orders` becomes
+`public_orders`, and `my_schema.t` becomes `my__schema_t`.
+
+---
+
 ## BigQuery
 
 Replicates a Postgres publication to a Google BigQuery dataset.