feat(iceberg): Apache Iceberg export (zero-copy, opt-in) (#532)

[xe-nvdk]

• feat(iceberg): reconciler core for Iceberg export (Phase 1)

New internal/iceberg package that projects Arc's existing Parquet files into Apache Iceberg tables, so external engines (Spark/Trino/DuckDB) can read Arc's data without changing the ingest path. Pure library; not yet wired into main.go.

• Exporter over a SQLite Iceberg catalog, reusing Arc's existing mattn/go-sqlite3 driver (Arc is already a CGo build via duckdb-go + mattn — no new SQLite impl).
• EnsureTable: day(time) partition spec (daily-compacted files span 24h, so hour() would reject them) + time->timestamptz mapping (Arc writes UTC-adjusted TIMESTAMP_MICROS; plain timestamp fails AddFiles).
• ReconcileMeasurement: diffs Arc's durable file set vs the table's live files (Scan().PlanFiles) and commits the delta via ReplaceDataFiles in one snapshot. Idempotent (no-op when converged) and driven by durable state, so a missed/failed commit self-heals next tick — unlike an event-stream design.
• SchemaFromParquet: pure-Go footer read -> Iceberg schema (no DuckDB handle).
• PathResolver: storage-relative key -> file://|s3://|azure:// URI.

Verified: iceberg-go v0.6.0 AddFiles auto-emits schema.name-mapping.default so Arc's field-ID-less Parquet reads correctly from a non-DuckDB engine (PyIceberg, Phase 0b). Test covers add + idempotency + hourly->daily supersession.

• feat(iceberg): config + storage-walk source + scheduler + wiring (Phase 2)

Wires the Iceberg export reconciler into Arc as an opt-in feature (storage.file_format unaffected; iceberg.enabled=false by default). Verified end-to-end in the running binary: ingest -> flush -> reconcile -> external DuckDB read, including a column added mid-stream.

• IcebergConfig (config + setDefaults + ARC_ICEBERG_* env); opt-in.
• FileSetSource = StorageWalkSource: lists measurements/files straight from the storage backend, so it works in OSS and cluster with no tiering/Raft dependency (tier_files is only populated when tiering is enabled — would be empty in OSS).
• Scheduler: interval loop, writer-gated (reuses the compaction cluster gate; nil/always-run in OSS), converges each pass (idempotent reconciler).
• main.go: constructed OUTSIDE the cluster block so it runs standalone; clean shutdown hook; warehouse defaults to the storage root.

Two bugs the running binary caught that unit tests missed, both fixed + regression tested:

• Schema evolution: Arc's schema-flexible ingest yields files with differing columns; a wider file failed AddFiles. Fixed via UnionSchema (superset across a measurement's files) + evolveSchema (AddColumn as optional).
• Name-mapping not extended on evolution: UpdateSchema.AddColumn leaves the evolution-added column out of schema.name-mapping.default, so external readers (Arc's Parquet has no field IDs) fail with "no field-mapping exists". Fixed by refreshing the mapping from the actual post-evolution schema.

• feat(iceberg): version-hint.text + v.metadata.json for directory-based readers

Publishes Hadoop-catalog discovery files next to each table's metadata so readers that point at the table DIRECTORY (no exact metadata filename, no catalog) can find the current metadata:

• metadata/version-hint.text — the version integer, NO trailing newline (DuckDB reads it verbatim; a newline makes it seek "v\n.metadata.json").
• metadata/v.metadata.json — a copy of the current metadata under the Hadoop-convention name. Required IN ADDITION to the hint: iceberg-go/the SQL catalog write "NNNNN-.metadata.json", but Spark and DuckDB resolve the hint strictly to "v.metadata.json" (both fail "metadata file for version N missing" with only the hint).

Written after every commit (create/reconcile/evolve) via the storage backend, so it works for local and S3 warehouses. Best-effort: failures are logged, not fatal — the SQL catalog stays the source of truth and the next reconcile rewrites these. Catalog-aware readers (PyIceberg, iceberg-go) are unaffected.

Verified against Arc's binary + both directory-based engines (previously failing): DuckDB iceberg_scan('') and Spark read.format("iceberg").load('') now read correctly. NewExporter gains a storage.Backend param.

• feat(iceberg): snapshot expiry + metadata/v pruning (Phase 4)

Bounds snapshot history and metadata growth so Iceberg tables don't accumulate unbounded state. New config iceberg.retain_snapshots (default 10).

• After each reconcile commit, ExpireSnapshots(WithRetainLast(retain), WithOlderThan(0)). WithRetainLast alone is a FLOOR, not a cap: iceberg-go only expires a snapshot older than maxSnapshotAgeMs (default ~5d) AND beyond the retain count, so with the default age nothing expires. WithOlderThan(0) satisfies the age gate so retain-last becomes the effective cap.
• Table created with write.metadata.delete-after-commit.enabled=true + write.metadata.previous-versions-max=retain, so iceberg-go prunes its own NNNNN-*.metadata.json history.
• pruneOldVersionFiles keeps only the newest retain of OUR v.metadata.json copies (iceberg-go doesn't know about those). Scan-based (lists the dir), robust to non-contiguous versions — each reconcile commits twice (ReplaceDataFiles + ExpireSnapshots) so versions skip. Never deletes the current version.

All best-effort: an expiry/prune failure doesn't undo the reconcile; the next pass retries. Writer-gated (single writer), so no reader-vs-expire race beyond Iceberg's snapshot isolation.

Verified in the binary: 8 distinct writes -> live snapshot count capped at 3 (retain=3), v copies capped at 3, and DuckDB dir-scan still reads all 8 rows after expiry. NewExporter gains a retain param.

• feat(iceberg,backup): back up + restore Iceberg warehouse metadata (Phase 5)

CreateBackup previously copied only .parquet files, silently dropping the Iceberg warehouse metadata (metadata.json / .avro manifests / version-hint.text) — a restore kept the parquet data but lost the Iceberg tables pointing at it. Now those metadata files are copied via the same mechanism (isIcebergMetadata predicate: files under a "/metadata/" dir ending .metadata.json or .avro, or version-hint.text), kept out of the db/measurement inventory. Restore needs no change: restoreDataFiles round-trips everything under {backupID}/data/ by path, so the metadata returns to its original location and the SQLite catalog's pointers resolve.

Verified (TestBackupRestore_IcebergMetadata): backup -> delete metadata from the data store -> restore -> all Iceberg metadata files return with correct content + the parquet. Predicate has a unit test incl. the false-positive guard (a measurement named "metadata" holding parquet must not match).

Note: found a pre-existing, unrelated gap — cfg.Backup is never populated in config.go Load(), so the backup API is not enabled regardless of the [backup] TOML section. The Iceberg backup change is correct + tested at the Manager level; that wiring gap deserves its own fix. (Documented in docs/progress.)

• fix(iceberg): address adversarial review — 2 blockers + hardening

Adversarial review of the Iceberg export feature found two blockers and several high/medium issues, all verified against iceberg-go v0.6.0 source and fixed:

B1 (SQLite): the catalog opened the shared SQLite file with no pragmas, unlike auth (WAL + busy_timeout + SetMaxOpenConns(1)). Under concurrent load on the shared DB (auth/audit/tiering/retention/MQTT) the catalog got immediate SQLITE_BUSY, failing reconciles. Now opened with ?_journal_mode=WAL&_busy_timeout=5000&_foreign_keys=ON and SetMaxOpenConns(1), mirroring auth. (cmd/arc/main.go)

B2 (partition panic): iceberg-go panics->errors "more than one value for partition field" when a single Parquet file's time min/max straddle a UTC-day boundary (rare: backfill / midnight flush), which wedged the whole measurement's export forever. replaceDataFilesResilient now tries the batch, and on that specific error falls back to per-file adds, SKIPPING only the straddling file(s) (logged at Error) so the rest still export. Test with a 2-day-spanning file. (exporter.go)

H1 (self-referential walk): the warehouse (arc_.db/) lives under the storage root, so Measurements() enumerated it as a phantom database. NewStorageWalkSource now takes nsPrefix and skips _*.db dirs. New source_test.go covers this + that Files() recurses nested Y/M/D/H partitions.

H2 (cold tiering): a file migrated to a cold S3 tier vanishes from the local walk and would be DELETED from the Iceberg table. config.Load now rejects iceberg.enabled + tiered_storage.cold.enabled. Test added.

Also: H3 — documented the single-compaction-node requirement for Phase-4 clusters (non-transactional warehouse writes). M1 — ExpireSnapshots failures now log at Error (persistent failure grows history unbounded). L3 — fixed package-doc drift (ReplaceDataFiles, not AddFiles+Delete).

• feat(iceberg): incremental reconcile + per-measurement timeout; release notes

Deferred optimizations from the adversarial review:

• Incremental reconcile: the scheduler now fingerprints each measurement's file set (data files + local files) and SKIPS measurements whose set is unchanged since the last successful pass — no footer reads, no diff, no commit. UnionSchema (the O(files) footer scan) runs only when the set actually changed. At steady state a pass is cheap regardless of total file count. Stale cache entries for deleted measurements are pruned.
• Per-measurement timeout (2m) so one slow/wedged measurement can't block the whole pass or graceful shutdown; ctx cancellation is checked between measurements.

Pass log now reports reconciled/unchanged/failed. Test: unchanged pass creates no new snapshot; adding a file triggers a reconcile.

Also adds RELEASE_NOTES_2026.09.1.md coverage for the Iceberg export feature (what/why/how, config table, limits). User guide added in docs.basekick.net (Integrations -> Apache Iceberg).

• fix(config): populate cfg.Backup in Load() so the backup API actually enables

BackupConfig had a struct + setDefaults entries (backup.enabled=true, backup.local_path) but was never read in config.Load(), so cfg.Backup was always the zero value — cfg.Backup.Enabled=false — and the backup/restore API (POST /api/v1/backup) was never registered regardless of the [backup] TOML section. Pre-existing bug, found while testing the Iceberg backup work.

Load() now populates Backup from the "backup.enabled" / "backup.local_path" keys, mirroring the other optional-feature blocks. Verified against the binary: startup now logs "Backup/restore enabled" and GET /api/v1/backup/ returns 200 (was 404). Test asserts the defaults land and env overrides apply.

• fix(iceberg): enforce local-only backend at config load (matrix finding)

The pre-PR config matrix surfaced a gap: the guard refused iceberg + cold-tier tiering but NOT iceberg + a non-local PRIMARY backend (S3/Azure). Iceberg export is local-only in v1 (the reconciler walks the single backend and reads Parquet footers from the local filesystem; verified only against local), and the docs/release notes say so — but nothing enforced it, so a primary-S3 deployment would run the reconciler against S3 unguarded.

config.Load now also requires storage.backend="local" when iceberg.enabled=true, alongside the existing cold-tier refusal. Both fire before startup. Test added (TestLoad_IcebergRequiresLocalBackend); the deep review confirmed the guard is correct and breaks no existing valid config (only fires when iceberg.enabled, default false).

• fix(iceberg): address Gemini review — cross-platform URIs, name-mapping self-heal, incremental schema

• file:// URIs now absolute + forward-slashed (Windows-safe) via localFileURI
• heal schema.name-mapping.default if a prior evolution's mapping txn failed
• incremental schema derivation: read only newly-added footers, MergeSchemas
• scheduler: Ticker→Timer so a slow pass can't cause back-to-back runs
• single storage List per measurement (FilesAndLocal)
• tests: incremental schema evolution + MergeSchemas conflict

Declined the snapshot-expiry finding: WithOlderThan takes a time.Duration, not Unix millis; WithOlderThan(0) is correct (verified: 5 snaps, retain=3 → 3 kept). Gemini's suggested time.Now().UnixMilli() would disable expiry.

• fix(iceberg): address Gemini round 2 — parallel footer reads + filepath.Base for Windows keys

• UnionSchema now reads Parquet footers in parallel (errgroup, limit 8) so first-sight full derivation of a many-file measurement can't exceed measurementTimeout. Merge stays deterministic (results collected by index, folded in path order). x/sync already a dep.
• path.Base → filepath.Base at the two key-parsing sites (isDataFile, pruneOldVersionFiles/ parseVersionAndMetaDir): LocalBackend.List returns backslash keys on Windows (filepath.Rel, no ToSlash) and Iceberg export is local-only. path.Join/path.Dir kept as-is — they BUILD forward-slash storage keys written back to the backend, where filepath would corrupt them.
• test: TestUnionSchema_ParallelDeterministic (order-stable under concurrency + error surfaced).

• fix(iceberg): reconcile table to empty when all data files are deleted (Gemini round 3)

Retention/compaction deleting every file of a measurement left the Iceberg table pointing at deleted paths forever: Arc's Delete only os.Remove()s the file, so the {db}/{measurement}/... dir tree survives, Measurements() keeps yielding it, and reconcileOne's len(localFiles)==0 early-return skipped the pass. External engines then fail on the missing files. Reproduced, then fixed: files=[] now reconciles the table to empty.

Two guards beyond the reported finding:

• Only empty a table that ALREADY exists (TableExists). A stray empty directory that never held data must not mint a zero-column table — EnsureTable with an empty schema would create one with no fields and no partition spec.
• Fingerprint-cache the empty state so a permanently-empty measurement is emptied ONCE and skipped thereafter, instead of re-committing and re-logging every tick. The cached state has a nil localFiles, so a later re-ingest falls through to full re-derivation.

Verified on the running binary: 40 rows -> delete all -> exactly one "reconciled to
empty" across 5 ticks -> DuckDB iceberg_scan returns 0 rows (was: read failure).

Tests: TestScheduler_AllFilesDeletedEmptiesTable (empty-out + no-repeat + re-ingest recovery), TestScheduler_EmptyDirNeverCreatesTable.

• fix(iceberg,backup): Type.Equals, int32/float32 mapping, future-proof metadata backup (Gemini round 4)

• schema: compare Iceberg types with Type.Equals() instead of String() comparison (idiomatic; the interface exposes Equals(Type) bool).
• schema: map arrow Int32/Float32 -> iceberg int/float. Arc's own ingest only emits Int64/Float64/String/Boolean/Timestamp_us, but Parquet also arrives via the bulk import path (internal/api/import.go) carrying externally-produced 32-bit columns — those previously failed the measurement's whole export with "unsupported Arrow type". Same reason Decimal128 was already mapped.
• backup: isIcebergMetadata is now a catch-all (any non-parquet under a /metadata/ segment) instead of an extension allowlist, so new Iceberg metadata types (e.g. Puffin .puffin stats/index files) can't silently drop out of the backup and be lost on restore. Safe because CreateBackup's switch tests the .parquet branch first, so data files never reach the predicate.
• tests: TestArrowToIceberg (full mapping incl. timestamptz vs timestamp + unsupported type error); isIcebergMetadata cases for .puffin and stray non-parquet.

Declined the S3 trailing-slash finding: S3Backend.prefix is sanitized at construction via SanitizeS3Prefix ("" or guaranteed trailing /), so the malformed-URI case can't occur — and the S3 arm is unreachable anyway (Iceberg export refuses non-local backends).

Verified on the running binary: backup captured all 7 Iceberg metadata files (metadata.json, .avro manifest+snapshot, version-hint.text) alongside the parquet data.

• fix(iceberg): ctx-aware UnionSchema, reject column type changes, typed already-exists (Gemini round 5)

• UnionSchema now takes a context and uses errgroup.WithContext: the parallel footer reads abort on the first failure or on reconciler timeout/shutdown instead of grinding through every remaining file. Callers/tests updated.

• evolveSchema now rejects a column whose type CHANGED on an existing table. This was a real silent-corruption gap: UnionSchema only compares the current pass's files against each other, and MergeSchemas compares against the in-memory cache — which is empty after a restart or an empty-out. So value long -> all old files age out -> restart -> new files arrive with value as double would derive a self-consistent schema, see the column as "present", skip it, and register type-incompatible files into the table. Now the measurement fails loudly and the table stays readable.

• healNameMapping: never write schema.name-mapping.default="null". NameMapping is a []MappedField, so a nil mapping marshals to the literal "null" — an invalid mapping that would break the external readers the property exists to serve.

• isAlreadyExists: match iceberg-go's typed sentinels via errors.Is (the SQL catalog wraps them with %w) instead of substring-matching "exist", which also matched "does not exist" and could swallow a genuine not-found/backend error as success.

Tests: UnionSchema cancellation; evolveSchema type-mismatch rejected + unchanged-schema no-op. Verified on the running binary: two measurements in one namespace both create and reconcile (exercises the already-exists path), DuckDB reads both.

• fix(iceberg): never write a null name-mapping from evolveSchema (Gemini round 6)

Round 5 guarded healNameMapping against writing schema.name-mapping.default="null" (NameMapping is a []MappedField, so a nil mapping marshals to the literal "null" — an invalid mapping that breaks the external readers the property exists to serve), but the sibling site in evolveSchema did the same marshal+SetProperties WITHOUT the guard.

Rather than duplicate the check, both paths now route through one helper, setNameMappingFromSchema — the only place that property is set. The two sites doing the same thing and drifting apart is exactly how this was missed, so the guard now lives in a single place by construction. healNameMapping keeps its best-effort semantics (warn, retry next pass) by wrapping the shared helper.

Verified on the running binary: narrow -> wide ingest evolves the schema ("added_columns":1), the resulting name-mapping includes the evolution-added column, and DuckDB reads the evolved table (60 rows, humidity populated in 30, NULL in the older 30).

• refactor(iceberg): drop redundant .tmp. prefix check in isDataFile (Gemini round 7)

strings.HasPrefix(base, ".tmp.") was unreachable — anything starting with ".tmp." already starts with ".", which the preceding check catches. Same behaviour, one condition. Kept the ".tmp." rationale in the comment, since skipping Arc's in-flight writes is the reason the dotfile check exists at all.

Added TestIsDataFile to lock the behaviour in (it was untested): .tmp.* and other dotfiles skipped, plain/compacted .parquet accepted, non-parquet rejected.

---

[gemini-code-assist]

Code Review

This pull request introduces an opt-in Apache Iceberg export layer that publishes Arc's existing Parquet files as Iceberg tables without modifying the ingest write path. It adds the internal/iceberg package containing the exporter, scheduler, and path resolver, integrates Iceberg configuration, and updates the backup/restore process to preserve Iceberg metadata. Feedback on the changes highlights a high-severity issue in warehouseRelKey where a custom warehouse configured as a subdirectory of the storage root causes file operations to fail, and a fix is suggested to resolve relative keys against the backend's storage root instead.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[xe-nvdk]

Good catch on warehouseRelKey — that was a real bug, and a genuinely new one rather than a re-flag. Fixed in #534 (against the 26.09.1 collection branch, since this PR tracks the eventual 26.09.1 → main merge).

Reproduced it before fixing. The root cause is that backends' Read/Write take keys relative to the storage root, while warehouseRelKey trimmed the warehouse — the same string only when iceberg.warehouse is the storage root, which is the default. That is exactly why every test and every prior binary run missed it: they all used the default warehouse. With warehouse=<root>/warehouse:

before: <root>/arc_mydb.db/cpu/metadata/version-hint.text            (leaked to the storage root)
after:  <root>/warehouse/arc_mydb.db/cpu/metadata/version-hint.text  (correct)

I took the suggested approach but added one guard it lacked: if the warehouse sits entirely outside the storage root, the backend cannot address it, so warehouseRelKey now returns ok=false rather than producing a key that silently escapes the root.

Verified on the running binary with iceberg.warehouse pointed at a subdirectory: the warehouse writes under data/warehouse/, nothing leaks to the storage root, and duckdb iceberg_scan resolves via version-hint.text and reads 40 rows — the precise path that was broken.

---

Note for future passes on this PR: it stays open deliberately as a reminder to merge the whole 26.09.1 collection branch into main at release time — it is not a duplicate of #532. Substantive findings are welcome (this one was), but the WithOlderThan(0) item has been raised and declined twice with the iceberg-go source and a live run (details); it is a time.Duration max-age, not a timestamp.