Repository navigation
docs(integrations): X-to-Y integration pages foundation + Postgres to Snowflake pilot - #588
shubham19may wants to merge 1 commit into
Conversation
…res to Snowflake pilot Adds the data layer, shared blocks, components and checks for the "source to query engine" integration pages under /docs/integrations/, plus the first page, Postgres to Snowflake via Iceberg (Horizon catalog), backed by a real OLake v0.11.2 run. - src/data/integrations: sources, catalogs (ordered Glue, Polaris, Lakekeeper, then vendor-native), engines with vendor-sourced delete format and catalog support, pairs matrix, people - src/components/integrations: byline, How we tested, compatibility strip, delete-format table, catalog tabs, FAQ with FAQPage JSON-LD, HowTo JSON-LD, related pages - docs/shared/integrations: partials for Postgres prereqs and connection, Horizon and Glue setup and writer config, job creation, Snowflake/DuckDB/Spark query snippets, with a static registry - scripts/integrations: check.mjs (front matter vs matrix, partial ids, own-words ratio, uniqueness, unlisted state) and scaffold.mjs - integrationsSidebar, hub page, Integrations breadcrumb label - CLAUDE.md and .cursor rule describing the page contract The pilot page is unlisted (noindex) and its pair is marked draft until review. Lint, typecheck, check.mjs and a CI build pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
| - "Snowflake (Horizon catalog, Snowflake-managed storage, AWS ap-south-1)" | ||
| - "DuckDB 1.5.5" | ||
| - "Apache Spark 3.5.6 with Iceberg 1.10.0" | ||
| summary: "A 2-table Postgres 16 database (8 customers, 40 orders) synced with Full Refresh + CDC and deletion vectors, then 7 inserts and 5 updates replayed through a second sync. All three engines returned identical rows." |
There was a problem hiding this comment.
synced with Full Refresh + CDC and deletion vectors : deletion vectors as write format mill make it much clearer
| <TLDR title='Key takeaways'> | ||
|
|
||
| - OLake Go writes into Snowflake's own catalog. Tables land as Snowflake-managed Iceberg tables; you query them in Snowsight with no `CREATE ICEBERG TABLE`, no external volume and no refresh. | ||
| - **Pick deletion vectors, never equality deletes.** Snowflake cannot read equality delete files. Set the job's target query engine to Snowflake and OLake Go hides the formats Snowflake cannot open. |
There was a problem hiding this comment.
Pick deletion vectors, Why pick deletion vectors ? Olake go resolves format as positional when query engine is selected as snowflake. Also it appears here that snowflake only supports DV which is not true.
There was a problem hiding this comment.
OLake Go hides the formats Snowflake cannot open: Better explanation can be resolves to only those formats which snowflake can read/ compatible with snowflake
|
|
||
| - OLake Go writes into Snowflake's own catalog. Tables land as Snowflake-managed Iceberg tables; you query them in Snowsight with no `CREATE ICEBERG TABLE`, no external volume and no refresh. | ||
| - **Pick deletion vectors, never equality deletes.** Snowflake cannot read equality delete files. Set the job's target query engine to Snowflake and OLake Go hides the formats Snowflake cannot open. | ||
| - The same tables open from DuckDB and Spark through the same Horizon endpoint, and all three engines returned identical rows in the run on this page. |
There was a problem hiding this comment.
The same tables open from DuckDB and Spark through the same Horizon endpoint : what is meant by open here. instead we can write it better
| - OLake Go writes into Snowflake's own catalog. Tables land as Snowflake-managed Iceberg tables; you query them in Snowsight with no `CREATE ICEBERG TABLE`, no external volume and no refresh. | ||
| - **Pick deletion vectors, never equality deletes.** Snowflake cannot read equality delete files. Set the job's target query engine to Snowflake and OLake Go hides the formats Snowflake cannot open. | ||
| - The same tables open from DuckDB and Spark through the same Horizon endpoint, and all three engines returned identical rows in the run on this page. | ||
| - **Watch `numeric` columns.** OLake Go maps Postgres `numeric` to Iceberg `double`, which Snowflake shows as `FLOAT`. |
There was a problem hiding this comment.
why is this important enough to mention here ? Isn't this a blog to show olake go compatibility with snowflake. IMO data type handling of postgres-> olake -> snowflake shouldn;t be mention here
| - **Pick deletion vectors, never equality deletes.** Snowflake cannot read equality delete files. Set the job's target query engine to Snowflake and OLake Go hides the formats Snowflake cannot open. | ||
| - The same tables open from DuckDB and Spark through the same Horizon endpoint, and all three engines returned identical rows in the run on this page. | ||
| - **Watch `numeric` columns.** OLake Go maps Postgres `numeric` to Iceberg `double`, which Snowflake shows as `FLOAT`. | ||
| - A Snowflake service user needs an authentication policy before its programmatic access token works; the SQL below includes it. |
|
|
||
| <I.Partial id="sources/postgres/connect" /> | ||
|
|
||
| Here is the source form from the run on this page. Postgres ran in another container, so the host is `host.docker.internal`; the update method is **CDC**, with the slot and publication from step 1. |
There was a problem hiding this comment.
we have already merged the new source improvements it does not have update method yet. instead it is a enable CDC toggle.
|
|
||
| Here is the source form from the run on this page. Postgres ran in another container, so the host is `host.docker.internal`; the update method is **CDC**, with the slot and publication from step 1. | ||
|
|
||
| <Figure src='/img/docs/integrations/postgres-to-snowflake/olake-source-postgres-cdc.webp' alt='OLake UI Create source form for Postgres with update method CDC, replication slot olake_slot and publication olake_pub' caption='The Postgres source in OLake UI: CDC update method with the slot and publication from step 1.' width='760px' /> |
There was a problem hiding this comment.
might need to change the SS also
|
|
||
| <Figure src='/img/docs/integrations/postgres-to-snowflake/olake-streams-deletion-vector.webp' alt='OLake UI Streams page with customers selected, sync mode Full Refresh + CDC, ingestion mode Upsert and upsert type Deletion Vector on Iceberg format version V3' caption='Each stream: Full Refresh + CDC, Upsert, Deletion Vector. The destination schema is named after the job, the source database and the source schema.' /> | ||
|
|
||
| On this page the first run copied 8 customers and 40 orders and finished in **27 seconds**. OLake Go names the schema after the job, the source database and the source schema: `pg_to_snowflake_shop_public`. |
There was a problem hiding this comment.
names the schema based on default , which can be changed also if required
|
|
||
| ## Updates and deletes: what Snowflake can read | ||
|
|
||
| Every engine reads Iceberg data files. They differ on the delete files that hide old row versions. OLake Go offers three formats, and the job must use one every reader understands: |
There was a problem hiding this comment.
hide old row versions ? what is this supposed to mean? stores the deleted row info ?
There was a problem hiding this comment.
OLake Go offers three formats, and the job must use one every reader understands: Olake go resolves it in a common format compatible with all the query readers selected
| 1. In the OLake UI, open **Sources** and click **Create Source**. | ||
| 2. Pick **Postgres** as the connector and keep the OLake Go version on the latest release. | ||
| 3. Fill in the host, port, database, username and password. Leave **Schemas** empty to discover every non-system schema, or list the ones you want. | ||
| 4. Set **Update Method** to **CDC**, then enter the **Replication Slot** and **Publication** you created above. |
There was a problem hiding this comment.
the update method here can be removed as this is removed as a part of source improvements
|
|
||
| OLake Go captures Postgres changes with logical replication and commits them as Iceberg tables straight into Snowflake's Horizon catalog. Files sit on Snowflake-managed storage, and nothing else sits in the path: no Kafka, no Spark, no S3 bucket of your own. | ||
|
|
||
| <I.IntegrationByline /> |
There was a problem hiding this comment.
This byline box sits between the intro and the key takeaways, so the summary gets pushed down.
Could it move to the end of the page, after the OLake Go box?
| - **Pick deletion vectors, never equality deletes.** Snowflake cannot read equality delete files. Set the job's target query engine to Snowflake and OLake Go hides the formats Snowflake cannot open. | ||
| - The same tables open from DuckDB and Spark through the same Horizon endpoint, and all three engines returned identical rows in the run on this page. | ||
| - **Watch `numeric` columns.** OLake Go maps Postgres `numeric` to Iceberg `double`, which Snowflake shows as `FLOAT`. | ||
| - A Snowflake service user needs an authentication policy before its programmatic access token works; the SQL below includes it. |
There was a problem hiding this comment.
I set this up on a fresh trial: without a policy, Snowflake wouldn't even create the token ("Network Policy is required when creating a programmatic access token for user OLAKE_SVC").
So the rule is a network policy, or an auth policy that waives it, which is how troubleshooting #3 already says it.
| - A Snowflake service user needs an authentication policy before its programmatic access token works; the SQL below includes it. | |
| - A Snowflake service user needs a network policy, or an authentication policy that waives it, before its programmatic access token works; the SQL below sets the second. |
|
|
||
| **The short answer.** To replicate Postgres to Snowflake with change data capture (CDC), run OLake Go with Postgres logical replication as the source and the Snowflake Horizon catalog as the Iceberg destination. One sync snapshots the tables; every later sync replays inserts, updates and deletes from the write-ahead log. Snowflake, DuckDB and Spark then query one copy of the data, in an open format. | ||
|
|
||
| ## Four ways to move Postgres data to Snowflake |
There was a problem hiding this comment.
The table below has five rows (four alternatives plus OLake Go)
| ## Four ways to move Postgres data to Snowflake | |
| ## Five ways to move Postgres data to Snowflake |
|
|
||
| ## Four ways to move Postgres data to Snowflake | ||
|
|
||
| | Method | Freshness | Deletes and updates | Open format | What you run | Cost model | Where it falls short | |
There was a problem hiding this comment.
Table Spacing Issue
The data types table further down faces same issue
| | --- | --- | --- | --- | --- | --- | --- | | ||
| | [Snowflake Openflow connector for PostgreSQL](https://docs.snowflake.com/en/user-guide/data-integration/openflow/connectors/postgres/about) | CDC through logical replication | Yes, as soft deletes plus journal tables | Native Snowflake tables (an Iceberg target exists) | An Openflow deployment in Snowpark Container Services or your own AWS account | Snowflake credits for the runtime | Single-node runtimes, password auth only, Snowflake is the only reader | | ||
| | Debezium, Kafka and the Snowflake Kafka connector | Seconds | Yes | Kafka topics, then Snowflake tables | Kafka, Kafka Connect, Debezium, schema registry | Your cluster plus Snowflake ingest | Four moving parts to run and upgrade; deletes need the connector's tombstone handling | | ||
| | Managed ELT (Fivetran, Estuary, Airbyte Cloud) | Minutes, by plan | Yes | Snowflake tables | Nothing | Per row, per GB or per connector | A second copy you pay to move and to store; the vendor owns the pipeline | |
There was a problem hiding this comment.
Small writing-rule thing:
ELT isn't expanded on its first use.
Maybe "Managed extract, load, transform (ELT): Fivetran, Estuary, Airbyte Cloud".
| | Manual export (`COPY` to S3, `COPY INTO`) | Hours to daily | No; full reloads only | Files you control | A script and a scheduler | Snowflake compute for the loads | No updates or deletes, growing load windows, stale between runs | | ||
| | **OLake Go to Iceberg on Horizon (this guide)** | CDC on a schedule down to one minute | Yes, as Iceberg row-level deletes | Apache Iceberg tables Snowflake, DuckDB, Spark and Trino read | One Docker Compose stack, open source | Snowflake storage plus your compute | No transformations in flight; equality deletes unreadable by Snowflake, so the job must use positional deletes or deletion vectors; Horizon API calls become billable in the second half of 2026 | | ||
|
|
||
| The honest summary: Openflow is simpler when Snowflake is the only reader and you want Snowflake to run the pipeline. Writing Iceberg once is the cheaper design as soon as a second engine needs the same rows, or you want to own the files. |
There was a problem hiding this comment.
"the cheaper design" is a cost claim without a source. "avoids paying to move and store a second copy" says the same thing, and the table already backs it up.
| flowchart TB | ||
| subgraph pg["Postgres"] | ||
| direction LR | ||
| T[(customers, orders)] --> WAL[write-ahead log] --> SLOT[replication slot, pgoutput] | ||
| end | ||
| subgraph olake["OLake Go"] | ||
| direction LR | ||
| SNAP[first sync: chunked snapshot] --> W[Iceberg writer: Parquet data files + Puffin deletion vectors] | ||
| CDC[later syncs: c / u / d events] --> W | ||
| end | ||
| subgraph sf["Snowflake"] | ||
| direction LR | ||
| H[Horizon catalog, Iceberg REST API] -. vends storage credentials .-> S[(Snowflake-managed storage)] | ||
| end | ||
| T --> SNAP | ||
| SLOT --> CDC | ||
| W -- "1. PUT data and DV files" --> S | ||
| W -- "2. commit snapshot" --> H | ||
| H --> Q1[Snowflake SQL] | ||
| H --> Q2[DuckDB] | ||
| H --> Q3[Spark] | ||
| ``` |
There was a problem hiding this comment.
I checked this against Snowflake's Horizon docs and a live run.
A few arrows are off: Horizon hands the temporary storage keys to the writer (not to storage), the engines read the files from storage after getting the file list from Horizon, and the order is credentials, then upload, then commit.
This version renders cleanly:
| flowchart TB | |
| subgraph pg["Postgres"] | |
| direction LR | |
| T[(customers, orders)] --> WAL[write-ahead log] --> SLOT[replication slot, pgoutput] | |
| end | |
| subgraph olake["OLake Go"] | |
| direction LR | |
| SNAP[first sync: chunked snapshot] --> W[Iceberg writer: Parquet data files + Puffin deletion vectors] | |
| CDC[later syncs: c / u / d events] --> W | |
| end | |
| subgraph sf["Snowflake"] | |
| direction LR | |
| H[Horizon catalog, Iceberg REST API] -. vends storage credentials .-> S[(Snowflake-managed storage)] | |
| end | |
| T --> SNAP | |
| SLOT --> CDC | |
| W -- "1. PUT data and DV files" --> S | |
| W -- "2. commit snapshot" --> H | |
| H --> Q1[Snowflake SQL] | |
| H --> Q2[DuckDB] | |
| H --> Q3[Spark] | |
| ``` | |
| %%{init: {"flowchart": {"nodeSpacing": 30, "rankSpacing": 40}}}%% | |
| flowchart TB | |
| subgraph pg["Postgres"] | |
| T[(customers, orders)] | |
| SLOT[write-ahead log, read through the replication slot with pgoutput] | |
| T ~~~ SLOT | |
| end | |
| subgraph olake["OLake Go"] | |
| W[Iceberg writer] | |
| end | |
| subgraph sf["Snowflake"] | |
| H[Horizon catalog, Iceberg REST API] | |
| S[(Snowflake-managed storage)] | |
| end | |
| R[Query engines: Snowflake SQL, DuckDB, Spark] | |
| T -- "first sync: chunked snapshot" --> W | |
| SLOT -- "later syncs: inserts, updates, deletes" --> W | |
| H -. "1. vended storage credentials" .-> W | |
| W -- "2. Parquet data files and deletion vectors" --> S | |
| W -- "3. commit snapshot" --> H | |
| H -- "table metadata and read credentials" --> R | |
| S -- "data and delete files" --> R |
|
|
||
| <I.IntegrationFaq /> | ||
|
|
||
| <I.HowWeTested /> |
There was a problem hiding this comment.
How we tested this: its last line ("Drafted with AI help ...") repeats the author and reviewer names.
If the names move to a card at the end of the page this line could go.
| <I.CatalogTabs> | ||
| <TabItem value="horizon"> | ||
|
|
||
| <I.Partial id="catalogs/horizon/writer" /> |
There was a problem hiding this comment.
The field table here scrolls sideways and has big empty boxes
|
|
||
| ## Step 5: Create the job and run the first sync | ||
|
|
||
| <I.Partial id="jobs/create" engines="Snowflake, plus DuckDB and Apache Spark if they will read the tables too" upsert="Deletion Vector" updateType="dv" /> |
There was a problem hiding this comment.
Two things from testing that block:
-
UI: with Snowflake, DuckDB and Spark as target engines, the Streams page still preselected Positional Deletes (Iceberg v2) on each table, so "check that the Upsert Type reads Deletion Vector" doesn't hold. "Set the Upsert Type to Deletion Vector for each table" would be safer, especially since Horizon won't let you change the format after the first sync.
-
CLI: the first sync with --state fails ("FATAL /mnt/config/state.json does not exist"). It works without --state, and OLake writes the file for later runs. The command also doesn't set OLAKE_INDEX_DB_DIR, which the text says deletion vectors need.
|
|
||
| Postgres to Snowflake comes down to one choice. Copy the rows into tables Snowflake owns, or write them once as Apache Iceberg tables that Snowflake and every other engine can read. This guide does the second. | ||
|
|
||
| OLake Go captures Postgres changes with logical replication and commits them as Iceberg tables straight into Snowflake's Horizon catalog. Files sit on Snowflake-managed storage, and nothing else sits in the path: no Kafka, no Spark, no S3 bucket of your own. |
There was a problem hiding this comment.
we are also demonstrating with glue+s3 bucket so mentioning that files sit on snowflake-managed storage and nothing else (s3 bucket) is wrong.
|
|
||
| OLake Go captures Postgres changes with logical replication and commits them as Iceberg tables straight into Snowflake's Horizon catalog. Files sit on Snowflake-managed storage, and nothing else sits in the path: no Kafka, no Spark, no S3 bucket of your own. | ||
|
|
||
| <I.IntegrationByline /> |
There was a problem hiding this comment.
Do we need to mention who did the technical review and the other information which is mentioned here? Basically is this component even needed?
| <TLDR title='Key takeaways'> | ||
|
|
||
| - OLake Go writes into Snowflake's own catalog. Tables land as Snowflake-managed Iceberg tables; you query them in Snowsight with no `CREATE ICEBERG TABLE`, no external volume and no refresh. | ||
| - **Pick deletion vectors, never equality deletes.** Snowflake cannot read equality delete files. Set the job's target query engine to Snowflake and OLake Go hides the formats Snowflake cannot open. |
There was a problem hiding this comment.
Why are we only mentioning to pick deletion vectors? Snowflake also supports positional deletes.
| flowchart TB | ||
| subgraph pg["Postgres"] | ||
| direction LR | ||
| T[(customers, orders)] --> WAL[write-ahead log] --> SLOT[replication slot, pgoutput] |
There was a problem hiding this comment.
Pointing replication slot after WAL gives info like WAL goes to replication slot. From replication slot we get position in WAL right?
|
|
||
| Two things happen in order. The first sync reads each table in parallel chunks and writes Parquet data files. A replication slot pins the write-ahead log (WAL) position from before that snapshot started. | ||
|
|
||
| Every later sync opens the slot and reads the `pgoutput` stream from that position. Each insert, update or delete becomes a row in a new data file, plus a deletion vector that hides the old version. |
There was a problem hiding this comment.
we should also mention positional deletes right? OLake Go supports both PD and DV
| ## What you need | ||
|
|
||
| - Postgres 10 or later with a user that can replicate (the tables on this page ran on Postgres 16). Hosted Postgres on Amazon Relational Database Service (RDS), Aurora, Azure or Cloud SQL works the same way; see the end of the page. | ||
| - A Snowflake account with `ACCOUNTADMIN` for the one-time setup. A 30-day trial works; Snowflake-managed storage means no cloud bucket or IAM role on your side. |
There was a problem hiding this comment.
is it necessary to mention 30 day trial works in the blog?
| -- 1. Logical replication (needs a restart, or a parameter-group change on managed Postgres) | ||
| ALTER SYSTEM SET wal_level = 'logical'; | ||
| ALTER SYSTEM SET max_replication_slots = 10; | ||
| ALTER SYSTEM SET max_wal_senders = 10; |
There was a problem hiding this comment.
wal_sender_timeout = 0 - Prevents timeouts during long snapshots
We also have this mentioned in our docs
| ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO olake; | ||
|
|
||
| -- 3. A publication for the tables you sync, and a pgoutput slot | ||
| CREATE PUBLICATION olake_pub FOR TABLE customers, orders; |
There was a problem hiding this comment.
We should be giving the commands for creating the tables first before adding them into publication. Also we are mentioning this same below as well so why not mention only once?
| ```sql title="Sample schema used on this page" | ||
| CREATE TABLE customers ( | ||
| id SERIAL PRIMARY KEY, name TEXT, email TEXT, city TEXT, tier TEXT, | ||
| created_at TIMESTAMPTZ DEFAULT now(), updated_at TIMESTAMPTZ DEFAULT now()); | ||
|
|
||
| CREATE TABLE orders ( | ||
| id SERIAL PRIMARY KEY, customer_id INT REFERENCES customers(id), product TEXT, | ||
| amount NUMERIC(10,2), status TEXT, | ||
| created_at TIMESTAMPTZ DEFAULT now(), updated_at TIMESTAMPTZ DEFAULT now()); |
There was a problem hiding this comment.
this should be mentioned either before where we mentioned publication command that adds these tables, else we should remove that publication command from above because we are running publication command before even creating the tables. Anyhow we again have mentioned publication command below
| GRANT ROLE OLAKE_ROLE TO USER OLAKE_SVC; | ||
|
|
||
| -- Service users need a network policy to use a token, unless the authentication policy says otherwise. | ||
| CREATE AUTHENTICATION POLICY IF NOT EXISTS OLAKE_DEMO.PUBLIC.OLAKE_PAT_POLICY |
There was a problem hiding this comment.
We don't need to specify OLAKE_DEMO.PUBLIC here.
The policy name alone is sufficient, and this has been manually tested
This branch has not been deployed
4 participants
Summary
First piece of the "X to Y" integration pages programme: one page per (source, query engine) pair under
/docs/integrations/, each backed by a pipeline that was actually run. This PR adds the shared foundation and the pilot page, Postgres to Snowflake (CDC via Iceberg into Snowflake's Horizon catalog, read by Snowflake, DuckDB and Spark).The pilot is
unlisted: true(noindex) and its pair isstatus: 'draft'inpairs.ts. It stays that way until review signs off; flipping both is a one-line change.What's in it
src/data/integrations/: sources, catalogs (ordered AWS Glue, Polaris, Lakekeeper, then vendor-native), engines (delete-format support and catalog support with vendor page + check date), the pair matrix, people (author/reviewer).src/components/integrations/: byline, "How we tested this", compatibility strip, delete-format table, catalog tabs (?catalog=deep link), FAQ with FAQPage JSON-LD, HowTo JSON-LD from## Step Nheadings, related pages. All read the page'sintegration:front matter.docs/shared/integrations/: Postgres prerequisites and connection, Horizon and Glue setup and writer config, job creation, Snowflake/DuckDB/Spark query snippets, registered inregistry.tsand rendered with<I.Partial id="..." />.scripts/integrations/:check.mjs(front matter against the matrix, partial ids, own-words vs shared words, uniqueness, unlisted state) andscaffold.mjs. Need Node 23.6+ for the.tsresolve hook.integrationsSidebar, hub pagedocs/integrations/index.mdx, "Integrations" breadcrumb label insrc/theme/seo/helpers.ts.CLAUDE.mdand.cursor/rules/integrations.mdcso new pages follow the same rules;.claude/added to.gitignore.docs/integrations/postgres-to-snowflake.mdxwith 10 screenshots from the run (OLake v0.11.2, tested 2026-10-01).Verification
node scripts/integrations/check.mjs— 0 errorsnpm run lint:content -- docs/integrations docs/shared/integrations— 0 warningsnpm run typecheck— cleanCI=true npm run build— succeeds; all internal links on the page resolve?catalog=glue&engine=sparkswitches every tab groupReview asks
numeric→double→FLOATprecision note, the Openflow row in the methods table.check.mjs; happy to add one if we agree on it.🤖 Generated with Claude Code