Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion blog/2025-02-21-data-partitioning-in-s3.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ Although if you look closely, you can implement both of the above partitioning s

## Explaining Our Current Approach with Examples

Let me walk you through our setup. In our [`streams.json`](/docs/core/configs/catalog/), we define partitioning on a per-stream basis. Here’s a simplified example:
Let me walk you through our setup. In our [`streams.json`](/docs/install/docker-cli#streams-config), we define partitioning on a per-stream basis. Here’s a simplified example:


```json
Expand Down
2 changes: 1 addition & 1 deletion blog/2025-03-18-json-vs-bson-vs-jsonb.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ While JSON is perfect for many use cases, it has limitations:
**BSON** (Binary JSON) is a binary-encoded serialization of JSON-like documents, originally created for MongoDB. BSON extends JSON's capabilities by adding support for more complex data types and structures.

#### Why BSON Exists
As [MongoDB](/docs/connectors/mongodb/setup/local/) began to rise in popularity as a NoSQL document store, the need for a more efficient, flexible format than plain JSON became apparent. BSON was developed to:
As [MongoDB](/docs/connectors/mongodb/) began to rise in popularity as a NoSQL document store, the need for a more efficient, flexible format than plain JSON became apparent. BSON was developed to:
- **Handle Complex Data Types**: BSON supports more than just strings and numbers. It can store native types like dates, binary data, and embedded arrays or objects efficiently.
- **Optimize for Database Operations**: BSON is designed to be lightweight but still allow for fast queries and indexing inside a database like MongoDB.
- **Better for Large-Scale Data**: BSON was created to offer faster data reads/writes and a more compact size when dealing with large documents.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,7 @@ This setup can be used for other downstream processing as well, example PG --> D
answer: <div>
<p>Yes. Aurora PostgreSQL supports CDC through PostgreSQL logical decoding using the native <code>pgoutput</code> plugin, and OLake Go has a dedicated Aurora PostgreSQL setup guide.</p>
<p>Logical replication on Aurora is enabled through the cluster parameter group (by setting <code>rds.logical_replication = 1</code>) rather than an instance parameter group, and changes are streamed via <code>pgoutput</code> with a PostgreSQL publication.</p>
<p><strong>Note:</strong> The configuration differs from RDS PostgreSQL in a few areas. Aurora uses a cluster parameter group, requires a writer-node reboot to apply the change, and replication slot behavior can differ in a multi-AZ cluster. Follow <a href="https://olake.io/docs/connectors/postgres/setup/aurora/">OLake's Aurora PostgreSQL setup guide</a> rather than the standard RDS PostgreSQL guide to avoid misconfiguration.</p>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

<p><strong>Note:</strong> The configuration differs from RDS PostgreSQL in a few areas. Aurora uses a cluster parameter group, requires a writer-node reboot to apply the change, and replication slot behavior can differ in a multi-AZ cluster. Follow <a href="https://olake.io/docs/connectors/postgres/">OLake's Aurora PostgreSQL setup guide</a> rather than the standard RDS PostgreSQL guide to avoid misconfiguration.</p>
</div>
},
]} />
Expand Down
2 changes: 1 addition & 1 deletion blog/2025-05-08-olake-airflow-on-ec2.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ Before deploying the DAG, ensure the following are in place:
* **S3 Bucket (`S3_BUCKET_NAME` in the DAG):** Create an S3 bucket where Olake Go's configuration files and persistent state file will be stored.
* **S3 Prefix for Configurations (`S3_PREFIX` in the DAG):** Decide on a "folder" (S3 prefix) within your bucket where your Olake Go configuration files will reside (e.g., `olake/projectA/configs/`).

* **Upload Olake Go Configuration Files:** Before running the DAG, you must upload your Olake Go `source.json`, `streams.json`, and `destination.json` files to the S3 bucket under the prefix you defined. The DAG's SSH script will sync these files to the EC2 instance. Please visit[ OLake Docs](/docs) website to learn how the[ source](/docs/connectors/overview) and[ destinations](/docs/writers/iceberg/catalog/rest) can be set up.
* **Upload Olake Go Configuration Files:** Before running the DAG, you must upload your Olake Go `source.json`, `streams.json`, and `destination.json` files to the S3 bucket under the prefix you defined. The DAG's SSH script will sync these files to the EC2 instance. Please visit[ OLake Docs](/docs) website to learn how the[ source] and[ destinations](/docs/writers/iceberg/catalog/rest) can be set up.

We need to generate `streams.json` beforehand using the OLake Go `discover` command against your source database.
* Streams Generation Guides:
Expand Down
2 changes: 1 addition & 1 deletion blog/2025-05-08-olake-iceberg-athena.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ You’ll need two configuration files

- [`destination.json`](/docs/writers/iceberg/catalog/glue) file with your Writer ([Apache Iceberg](/docs/writers/iceberg/catalog/rest)) connection details.

You can read [here](/docs/connectors/postgres/setup/local) for detailed version OLake Configuration for Postgres
You can read [here](/docs/connectors/postgres/) for detailed version OLake Configuration for Postgres

### 2.4 Discover
Run the [Discover](/docs/connectors/postgres) command to generate a streams.json file, which contains the list of available data streams from your source database. You can refer to the official OLake documentation for complete documentation
Expand Down
2 changes: 1 addition & 1 deletion blog/2025-05-08-olake-iceberg-trino.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ You’ll need two configuration files

- [destination.json](/docs/writers/iceberg/catalog/glue) file with your writer ([Apache Iceberg](/docs/writers/iceberg/catalog/rest)) connection details

You can read here for detailed version [OLake Configuration](/docs/connectors/postgres/setup/local)
You can read here for detailed version [OLake Configuration](/docs/connectors/postgres/)

### 4 Discover
Run the [Discover](/docs/connectors/postgres) command to generate a streams.json file, which contains the list of available data streams from your source database. You can refer to the official OLake documentation for complete documentation
Expand Down
2 changes: 1 addition & 1 deletion blog/2025-05-28-olake-glue-snowflake.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ You’ll need two configuration files
* [source.json](/docs/connectors/postgres#configuration) file with your database connection details
* [destination.json](/docs/writers/iceberg/catalog/glue) file with your Writer ([Apache Iceberg](/docs/writers/iceberg/catalog/rest)) connection details

You can read here for detailed version[ Olake Configuration](/docs/connectors/postgres/setup/local)
You can read here for detailed version[ Olake Configuration](/docs/connectors/postgres/)

**2.4 Run the[ Discover](/docs/connectors/postgres) command** to generate a `streams.json` file, which contains the list of available data streams from your source database. You can refer to the official[ OLake documentation](/docs) for complete documentation

Expand Down
4 changes: 2 additions & 2 deletions blog/2025-09-04-creating-job-olake-docker-cli.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ Name it clearly, fill the Postgres endpoint config, and hit **Test Connection**.

> 📝 **Planning for CDC?**
> Make sure a **replication slot** exists in Postgres.
> See: [Replication Slot Guide](/docs/connectors/postgres/setup/generic).
> See: [Replication Slot Guide](/docs/connectors/postgres/).

### 3) Configure the Destination (Iceberg + Glue)

Expand Down Expand Up @@ -206,7 +206,7 @@ We'll put everything under `/path/to/config/`.
```

> 📝 If you plan to run CDC, ensure a Postgres **replication slot** exists.
> See: [Replication Slot Guide](/docs/connectors/postgres/setup/generic).
> See: [Replication Slot Guide](/docs/connectors/postgres/).

**Destination — `destination.json`**

Expand Down
2 changes: 1 addition & 1 deletion blog/2025-09-07-how-to-set-up-postgres-apache-iceberg.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-c

- Access OLake UI at: http://localhost:8000
- Default login credentials: admin / password
- Complete setup documentation: [OLake UI Getting Started Guide](/docs/getting-started/olake-ui)
- Complete setup documentation: [OLake UI Getting Started Guide](/docs/install/olake-ui/)

**Alternative**: OLake Go also provides a configurable CLI for advanced users who prefer command-line operations. CLI documentation is available at: [OLake CLI Guide](/docs/install/docker-cli).

Expand Down
2 changes: 1 addition & 1 deletion blog/2025-11-03-olake-bauplan.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ Ready to build this yourself? Here's how to set up the complete stack.

**Optional: Need a Test Postgres Database?**

If you don't have your own Postgres database available, you can easily spin up a local Postgres instance with sample data using Docker. Follow the [Setup Postgres via Docker Compose guide](/docs/connectors/postgres/setup/local/) to get started with a pre-configured database for testing.
If you don't have your own Postgres database available, you can easily spin up a local Postgres instance with sample data using Docker. Follow the Setup Postgres via Docker Compose guide to get started with a pre-configured database for testing.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?



**Step 1: Set up Lakekeeper**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -511,7 +511,7 @@ If the issue isn't listed, post to the OLake Slack with connector config (omit s
- Keep one logical dataset per top-level folder and avoid mixed formats in the same stream folder.
- Add monitoring and alerting for failure spikes and long-running syncs.

And once you're happy with your S3 setup and you're ready to expand your pipeline to other sources, check out our [other connector guides here](/docs/connectors/overview/).
And once you're happy with your S3 setup and you're ready to expand your pipeline to other sources, check out our other connector guides here.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?


## FAQs
<Faq showHeading={false} data={[
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -337,7 +337,7 @@ And if you do go the CDC route, just keep these two practical rules in mind beca
1. **CDC must be enabled at both the database and table level**
2. **if the source table schema changes, create a new CDC capture instance for the updated schema**

When you're ready to bring in more systems, you can follow our [other connector walkthroughs as well here](/docs/connectors/overview/).
When you're ready to bring in more systems, you can follow our other connector walkthroughs as well here.

## FAQs

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,7 @@ Db2 is still a big part of how a lot of enterprises run their core systems—and

If anything breaks along the way, don't stress around and drop at the OLake community and devs would be there to help you in no time. Most of the time it's a small network/permission/SSL thing and we can point you to the fix quickly.

And once you're happy with your Db2 setup and you're ready to expand your pipeline to other sources, check out our [other connector guides here](/docs/connectors/overview/).
And once you're happy with your Db2 setup and you're ready to expand your pipeline to other sources, check out our other connector guides here.

## FAQs
<Faq showHeading={false} data={[
Expand Down
2 changes: 1 addition & 1 deletion blog/2026-07-22-deep-dive-into-olake-architecture.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@ Every database exposes its change log differently, and OLake Go reads each one n
| **MongoDB** | Oplog | Change streams, resumed via resume tokens |
| **SQL Server** | CDC capture tables | Polled in LSN ranges through the native CDC functions |

Each source has its own CDC prerequisites, and OLake Go validates them when the connection is set up, so a misconfiguration surfaces immediately instead of hours into a sync. The setup guide for each connector covers exactly what to enable: [PostgreSQL](/docs/connectors/postgres), [MySQL](/docs/connectors/mysql), [SQL Server](/docs/connectors/mssql), and [MongoDB](/docs/connectors/mongodb/cdc_setup).
Each source has its own CDC prerequisites, and OLake Go validates them when the connection is set up, so a misconfiguration surfaces immediately instead of hours into a sync. The setup guide for each connector covers exactly what to enable: [PostgreSQL](/docs/connectors/postgres), [MySQL](/docs/connectors/mysql), [SQL Server](/docs/connectors/mssql), and [MongoDB](/docs/connectors/mongodb).

### One log or many: the execution topologies

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,15 +81,15 @@ For PostgreSQL, OLake Go supports [four sync modes](/docs/connectors/postgres/):

4. **Full Refresh + Incremental** for tables where a cursor column is enough

CDC runs on the native `pgoutput` plugin, since the older [`wal2json`](/docs/connectors/postgres/wal2json_plugin/) method is deprecated. Full loads split large tables into chunks that are read in parallel and checkpointed, so an interrupted sync resumes where it stopped instead of starting over.
CDC runs on the native `pgoutput` plugin, since the older [`wal2json`](/docs/connectors/postgres/) method is deprecated. Full loads split large tables into chunks that are read in parallel and checkpointed, so an interrupted sync resumes where it stopped instead of starting over.

The initial load writes append-only data files, and CDC syncs add data files together with equality delete files keyed on the [`_olake_id`](/blog/olake-arrow-based-iceberg-ingestion/) column, plus position deletes when the same row changes more than once inside a single sync, giving merge-on-read upserts without rewriting existing files. An `_op_type` metadata column marks each record's operation, so downstream queries can separate deletes from inserts and updates. OLake Go supports AWS Glue, Hive Metastore, JDBC, and REST catalogs, including Nessie, Lakekeeper, Amazon S3 Tables, Apache Polaris, Unity Catalog, and Google BigLake, and it evolves Iceberg schemas automatically when source columns change.

[Table maintenance](/blog/apache-iceberg-table-maintenance-olake-fusion/) is handled by the companion project [OLake Fusion](/docs/fusion/getting-started/overview/), which connects to the same catalog and runs compaction on a [per-table schedule](/docs/fusion/compaction/configuration/). Its [Lite, Medium, and Full compaction tiers](/docs/fusion/compaction/types-of-compaction/) can be scheduled independently, so light cleanup runs often and full rewrites run occasionally, without writing Spark maintenance scripts.

## How to Set Up PostgreSQL CDC With OLake

The steps below use a [self-managed PostgreSQL server](/docs/connectors/postgres/setup/generic/) and the OLake UI. [Amazon RDS](/docs/connectors/postgres/setup/rds/), [Aurora](/docs/connectors/postgres/setup/aurora/), [Google Cloud SQL](/docs/connectors/postgres/setup/gcp/), and [Azure Database for PostgreSQL](/docs/connectors/postgres/setup/azure/) enable logical replication through parameter groups or database flags instead, so use the matching guide in the [OLake PostgreSQL connector documentation](/docs/connectors/postgres/) for those.
The steps below use a [self-managed PostgreSQL server](/docs/connectors/postgres/) and the OLake UI. Amazon RDS, Aurora, Google Cloud SQL, and Azure Database for PostgreSQL enable logical replication through parameter groups or database flags instead, so use the matching guide in the OLake PostgreSQL connector documentation for those.

Prerequisites: PostgreSQL 10 or later with permission to edit `postgresql.conf` and `pg_hba.conf`, Docker with at least 4 GB of RAM and port 8000 free, and an Iceberg catalog with object storage, such as a REST catalog (e.g., Lakekeeper) or AWS Glue with Amazon S3

Expand Down
1 change: 1 addition & 0 deletions compare/2026-10-01-what-is-fivetran.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ description: "What is Fivetran? A plain guide to how its syncs work, its connect
tags: [comparison, etl, data-pipeline-tools, data-ingestion, alternatives]
authors: [priyansh]
image: /img/blog/cover/what-is-fivetran-cover.webp
date: 2026-09-28
---
import BlogCTA from '@site/src/components/BlogCTA';
import quiz from '@site/src/data/quizzes/what-is-fivetran.json';
Expand Down
Loading
Loading