[OLake Go](/docs/) | Ingest (database changes into Iceberg or Parquet) | Yes, [Apache 2.0](https://github.com/datazip-inc/olake) | Open source; you run it on your own infrastructure ([install docs](/docs/install/olake-ui/)) |
+| Debezium | Ingest (database changes) | Open source, per its docs | Free; you pay for servers and staff |
+| Apache Kafka | Stream storage and transport | Open-source project | Free; managed Kafka is priced by each vendor |
+| Apache Flink | Stream processing | Open-source project | Free; you pay for servers and staff |
+| Matillion | Bundle: load, transform, schedule | No | Credits by task hour; dollar price not shown |
+| AWS Glue | Bundle on AWS | No | Compute time, in data processing unit (DPU) hours; $0.44 or $0.308 per DPU-hour in US East (Ohio) |
+| Azure Data Factory | Bundle on Azure | No | Activity runs plus integration runtime hours |
+| dbt | Transform | Core is Apache 2.0 | Free seat; Starter $100 per user a month |
+| Apache Airflow | Orchestrate | Open-source project | Free; you pay for servers and staff |
+| Dagster | Orchestrate | Open-source project; Dagster+ is paid | Solo $10 a month, Starter $100 a month, plus credits |
+| Prefect | Orchestrate | Open-source project; Prefect Cloud is paid | Free Hobby plan; Starter $100 a month |
+
+Sources: Airbyte release notes, Hevo pricing, AWS Glue price list, dbt pricing, Dagster pricing, Prefect pricing. Checked October 2026.
+
+"Not published" means the vendor's page doesn't show the number.
+
+## Ingestion tools: get your data into one place
+
+### 1. Fivetran: best for hands-off loading from many sources
+
+
+
+*Fivetran home page, October 2026.*
+
+**At a glance**
+- Price: Free up to 500,000 monthly active rows (MAR, the rows that change in a month); paid rates in its [Service Consumption Table](https://www.fivetran.com/legal/service-consumption-table), not on the pricing page. Standard starts at $495 per million MAR plus a $5 base per connection, list price before discounts (October 2026)
+
+Fivetran's pricing page lists 700+ connectors, 15-minute syncs on Standard and 1-minute syncs on Enterprise. Fivetran says it runs the pipeline for you, a claim we did not test.
+
+One 2026 change matters. Fivetran announced that its merger with dbt Labs closed on June 1, 2026, so a managed loader and a transform tool now share an owner. If you use both, ask whether pricing or packaging will change.
+
+**What's good**
+
+- The Free plan includes Standard plan features, with up to 500,000 monthly active rows free for connections, 3,500 for activations and 5,000 monthly model runs for transformations (pricing page, October 2026). Its claims that loading is fully managed and runs hands-off are the vendor's own, and we did not test them.
+- Standard lists unlimited users, 15-minute syncs, 700+ fully managed connectors and a dbt Core integration. Enterprise adds 1-minute syncs, custom roles, SCIM and a Hybrid deployment option, and Business Critical adds customer-managed encryption keys and PCI DSS Level 1 certification (pricing page).
+- For database sources, its docs say it captures changes with the source's native change data capture mechanism or a Fivetran proprietary method, depending on the database. They also say database connectors sync new and modified data every 15 minutes by default (Fivetran's database connector docs).
+- Each connection is billed on its own volume curve, so the per-row rate falls as one connection grows. On Standard it goes from $495 per million rows at the start to $170 per million past 1M rows and $58 per million past 4M (Service Consumption Table, version V.10, effective February 1, 2026).
+- Transformations get 5,000 model runs a month free, then $0.01 per run up to 30,000, $0.007 up to 100,000 and $0.002 beyond that (Service Consumption Table).
+
+**Where it falls short**
+
+- Paid list rates live in a separate table, and each connection has its own curve, so you need rows per connection to predict a bill. The table also says you cannot combine rows across connections to get a better rate.
+- Our read: the $5 base per connection is the easiest line to miss (pricing page). By the consumption table formula, 20 connections of 2,000 rows each come to $119.80 a month (our arithmetic).
+- Deleted rows also count as paid rows: the pricing page counts updates as modifications to existing rows in your destination, including deletes.
+- Teleport Sync is its option when you cannot or do not want to give Fivetran access to your database's change logs. The docs say that if you update large portions of a table between syncs it may be unable to find the changes, and the connection then re-syncs the entire table (database connector docs).
+- The pricing page calls Oracle and High-Volume Agent connectors "enterprise database connectors", and the Enterprise plan card lists them where Standard lists plain database connectors (pricing page).
+
+
+
+*Fivetran's pricing page, October 2026.*
+
+### 2. Airbyte: best for a long connector list with a free self-hosted core
+
+
+
+*Airbyte home page, October 2026.*
+
+**At a glance**
+- Price: Core is free; Cloud Standard $20 a month with 5 credits and $5 per extra credit; Plus from $189 for 40 credits ([Airbyte release notes](https://docs.airbyte.com/release_notes/cloud), September 15, 2026)
+
+Airbyte's docs say it has over 600 pre-built connectors, so check the current status of the connector you actually need.
+
+Its CDC (change data capture) docs describe log-based replication, but they also say syncs are scheduled and CDC sources aren't treated as infinite streams.
+
+**What's good**
+
+- Per its docs, you can self-host the free core or pay for the cloud, and build connectors with a no-code builder or a Python kit. We did not test how well any of these work.
+- Plus runs from 40 credits for $189 ($4.75 a credit) up to 2,000 credits for $4,999, and overage credits fall from $5 to $2.50 on the top tier. All Plus plans include 15-minute syncs, two workspaces, single sign-on and premium support (release notes, September 15, 2026).
+- Unused Cloud credits now roll over for 3 months, up from 2, and Cloud free trials last 14 days (release notes, September and October 2026).
+- Its CDC docs list Postgres, MySQL, Microsoft SQL Server, MongoDB, Oracle, SAP HANA and IBM Db2 under current support. They name no plan or connector support level for Oracle, SAP HANA or Db2 (Airbyte's CDC docs).
+
+**Where it falls short**
+
+- It isn't a continuous streaming tool: CDC syncs are launched on a sync interval, and the CDC docs say "We do not treat CDC sources as infinite streaming sources."
+- For most sources, CDC incremental sync works only for tables with primary keys, and a refresh is required after any new table or collection is added to a CDC connection (CDC docs).
+- Changes made by TRUNCATE or ALTER won't appear in the logs, so they don't reach the destination (CDC docs).
+- On Cloud Standard, scheduled or cron syncs can run at most every 60 minutes and an organization gets 1 workspace; Plus allows every 15 minutes (Airbyte's Cloud limits docs). SSO is no longer available on Standard, except for existing users who keep their plan (Airbyte's Cloud release notes).
+- Standard got dearer in September 2026: the base price rose from $10 to $20 a month, free credits from 4 to 5 and extra credits from $2.50 to $5.00 (release notes).
+- The core uses the Elastic License 2.0, and Airbyte's license FAQ says it stops you providing the products to others as a managed service.
+
+### 3. Hevo: best for small teams that want no-code pipelines into a warehouse
+
+
+
+*Hevo Data home page, October 2026.*
+
+**At a glance**
+- Price: Free up to 1M events a month (each insert, update or delete in the destination is one event); Starter $265 a month on annual billing, or $299 monthly, from 5M events (Hevo pricing page, October 2026)
+
+Hevo's [pricing page](https://hevodata.com/pricing/) lists 150+ connectors. Streaming pipelines show up on the custom Business Critical card, and its comparison table also lists them for Professional.
+
+**What's good**
+
+- Its free plan lets you try it with real data before you pay (pricing page).
+- The Free plan is $0 forever with up to 1M events a month and up to 5 users. Starter adds up to 10 users, 150+ connectors, a dbt integration and SSH/SSL (pricing page, October 2026).
+- Professional is $849 a month, or $750 on annual billing, from 20M events (pricing page).
+- Its docs list the log read for each log-based source: BinLog for MySQL, logical replication for PostgreSQL, Streams for DynamoDB, Change Streams or OpLog for MongoDB, Change Tracking for SQL Server and Redo Logs for Oracle (Hevo's log-based pipelines docs).
+
+**Where it falls short**
+
+- Every update is an event, so busy tables eat a quota fast. The pricing page counts each insert, update or delete in the destination as one event.
+- Usage above your plan's quota is on-demand and charged on top of the plan, and the page shows no rate for it (pricing page).
+- Its docs say streaming pipelines cover those log-based sources except SQL Server and DynamoDB, and you must contact Hevo Sales to enable them (Hevo's log-based pipelines docs).
+- The comparison table gives a 1-hour sync frequency on Free and Starter (pricing page).
+- Its connector list is also smaller than Fivetran's or Airbyte's: Hevo lists 150+, Fivetran 700+ and Airbyte's docs over 600.
+- The Free card says "Free forever for limited connectors", while all available connectors start at Starter (pricing page).
+
+
+
+*Hevo's pricing page, October 2026.*
+
+### 4. OLake Go: best for copying Postgres, MySQL, MongoDB or SQL Server into Iceberg
+
+
+
+
+
+*OLake home page, October 2026.*
+
+**At a glance**
+- Price: Open source under [Apache 2.0](https://github.com/datazip-inc/olake); you run it on your own infrastructure with [Docker](/docs/install/docker-cli/), [Helm on Kubernetes](/docs/install/kubernetes/) or the [OLake UI](/docs/install/olake-ui/)
+
+OLake Go copies [PostgreSQL](/docs/connectors/postgres/), [MySQL and MariaDB](/docs/connectors/mysql/), [MongoDB](/docs/connectors/mongodb/) and [SQL Server](/docs/connectors/mssql/) with a full load plus change data capture, and writes Apache Iceberg tables or Parquet files on storage you control. [Oracle](/docs/connectors/oracle/), [Db2](/docs/connectors/db2/), [S3](/docs/connectors/s3/) and [Kafka](/docs/connectors/kafka/) also work as sources, without database change capture.
+
+**What's good**
+
+- One tool does the first load and then the change capture, with Full Refresh, Full Refresh + Incremental, Full Refresh + CDC and CDC Only sync modes ([docs](/docs/)).
+- It writes Iceberg tables to catalogs such as [AWS Glue](/docs/writers/iceberg/catalog/glue/), with eleven catalog types listed on the [catalog compatibility page](/docs/understanding/compatibility-catalogs/).
+- Engines such as Athena, Spark, Flink and Trino can read the Iceberg tables it writes ([query engine compatibility](/docs/understanding/compatibility-engines/)).
+- Syncs are stateful and resumable: after a crash, a network failure or a pause they resume from the last checkpoint instead of starting over ([docs](/docs/)).
+- Parallel chunking splits large tables into chunks that are read in parallel, in the Full Refresh, Full Refresh + CDC and Full Refresh + Incremental modes ([docs](/docs/)).
+- Schema discovery runs at the start of every sync, and a new source column is added to the Iceberg schema automatically ([schema evolution docs](/docs/features/schema/)).
+
+### 5. Debezium: best for engineers who need every database change as an event
+
+
+
+*Debezium home page, October 2026.*
+
+**At a glance**
+- Price: Free; you run it yourself; you pay for servers and staff
+
+Its FAQ says Debezium records all row-level changes committed to each database table, so applications can react to insert, update and delete events, though after a failure they can see an event more than once. The docs list eight main source connectors, including PostgreSQL, MySQL, MongoDB, SQL Server and Oracle.
+
+**What's good**
+
+- It's free, and its docs describe delay in the millisecond range for MySQL and PostgreSQL. That is the project's claim, not our measurement, so test it on your own load (Debezium's features page).
+- The connector index lists eight main source connectors: MongoDB, MariaDB, MySQL, PostgreSQL, SQL Server, Oracle, Db2 and Cassandra. Seven more are marked incubating: Vitess, Spanner, Informix, CockroachDB, YashanDB, Ingres and Milvus (Debezium's connector index).
+- Beyond Kafka Connect, the same docs list Debezium Server, a Debezium Operator, a Debezium Management Platform and guides for running on Kubernetes and OpenShift (connector index).
+- The FAQ says all events within a topic are totally ordered, and that even if events are duplicated during failures the end result after applying them all stays the same (FAQ).
+- The FAQ also says Debezium connectors do not store any information inside the upstream databases (FAQ).
+
+**Where it falls short**
+
+- It captures changes and nothing more. You still need a place to send them, a way to run it and someone to keep it healthy.
+- The FAQ says consumers see every event exactly once only when systems run nominally or shut down gracefully; when things go wrong they can see events at least once, so the destination has to cope with repeats (FAQ).
+- Running a connector may place additional load on the source database (FAQ).
+- On MySQL, the FAQ says delete events can be missing when CASCADE DELETE statements are used, because those deletions are not part of the binlog (FAQ).
+- On Amazon RDS for MySQL, log_bin is managed by AWS and off by default, and the FAQ's fix is to turn on read replicas or automated backups (FAQ).
+- Incubating connectors are released for preview and are subject to changes that may not always be backward compatible (connector index).
+
+## Streaming tools: react to each change as it happens
+
+### 6. Apache Kafka: best for moving event streams between many systems
+
+
+
+*Apache Kafka home page, October 2026.*
+
+**At a glance**
+- Price: Free; managed Kafka is priced by each vendor
+
+Kafka's own introduction calls it an event streaming platform that publishes, stores and processes streams of events. Its Kafka Connect part continuously imports and exports data from systems such as relational databases.
+
+**What's good**
+
+- It's free, and its docs say it stores streams of events durably, which we did not test.
+- Events are not deleted after consumption, so a topic can be read as often as needed. How long they are kept is a per-topic setting (Kafka intro).
+- Any consumer of a topic-partition reads that partition's events in the order they were written, and a common production setting is a replication factor of 3, meaning three copies of the data (Kafka intro).
+- It can run on bare-metal hardware, virtual machines and containers, on-premises or in the cloud, and you can self-manage it or use fully managed services from a variety of vendors (Kafka intro).
+- Its 4.2 design docs say it supports exactly-once delivery in Kafka Streams, and in the transactional producer paired with a read-committed consumer. The producer also has an idempotent delivery option, so resending does not create duplicate log entries (Kafka's design docs).
+
+**Where it falls short**
+
+- It's a platform you run, not a ready-made pipeline. Kafka itself doesn't reshape warehouse tables, and it needs connectors at both ends.
+- The intro describes capture as something a connector does, for example a PostgreSQL connector capturing every change to a set of tables, not something the brokers do. It describes no Iceberg writer.
+- The intro says the Kafka community provides hundreds of ready-to-use connectors, and the Apache project publishes no exact count (Kafka intro).
+- Exactly-once is not the default: the 4.2 design docs say Kafka guarantees at-least-once delivery by default.
+
+### 7. Apache Flink: best for reshaping streams as they flow
+
+
+
+*Apache Flink home page, October 2026.*
+
+**At a glance**
+- Price: Free; you pay for servers and staff
+
+Flink's site describes it as "stateful computations over data streams", and it shows 2.3 as the current stable release. The Apache Iceberg docs say the Flink sink for Iceberg guarantees exactly-once semantics.
+
+**What's good**
+
+- Its site describes stateful computations on streams and, per the Iceberg docs, it can write batch and streaming data to Iceberg tables. We did not test either.
+- The Flink home page lists low latency, high throughput and in-memory computing as features.
+- The Iceberg docs say it supports UPSERT based on the primary key when writing to a v2 table (Iceberg Flink writes).
+- The same docs say IcebergSink can run table maintenance after each commit: compacting small data files, expiring old snapshots and removing orphan files, so a separate maintenance job is not required (Iceberg Flink writes).
+
+**Where it falls short**
+
+- It's a framework you program and operate, and it's overkill for hourly batch jobs.
+- Streaming jobs do not support INSERT OVERWRITE; overwrite works in batch jobs (Iceberg Flink writes).
+- UPSERT needs the v2 table format and a primary key or identifier fields, and OVERWRITE and UPSERT cannot be enabled at the same time (Iceberg Flink writes).
+- The Iceberg docs call the newer SinkV2-based IcebergSink "currently an experimental feature so use it with caution" (Iceberg Flink writes).
+- Iceberg commits happen after a successful Flink checkpoint, in the notifyCheckpointComplete callback, so the job needs checkpointing set up (Iceberg Flink writes).
+
+## All-in-one cloud services: several jobs, one vendor
+
+### 8. Matillion: best for teams that load and transform inside a cloud warehouse
+
+
+
+*Matillion home page, October 2026.*
+
+**At a glance**
+- Price: Credits used by task hours; the pricing page shows no dollar price (Matillion pricing page, October 2026)
+
+Matillion's pricing page lists Developer, Teams and Scale editions. Its credit terms charge 1 credit per task hour for scheduling work and transformation 3 credits per task hour.
+
+Matillion's own comparison blog says Maia generates native SQL that runs inside Snowflake, Databricks or AWS Redshift.
+
+**What's good**
+
+- Matillion's pages describe one product for loading, transforming and scheduling, so there are fewer parts to join. We did not test it.
+- Developer lists 1 developer user, unlimited projects, pre-built connectors, a low-code canvas with SQL/Python and a built-in Git repository. Teams adds 5 developer users, an audit log and a Service Level Agreement, and Scale adds custom SSO, hybrid cloud deployment, data lineage tracking and streaming change data capture (pricing page).
+- Credits cover task hours and extra developer users, and the pricing FAQ says you can contract directly with Matillion or pay through your AWS, Azure or Snowflake account (pricing page).
+- Task hours are charged only while a pipeline runs, in 2-second increments, and validation or sampling does not count towards them (credit terms).
+- Log retention is 90 days on Developer and Teams and 180 days on Scale, with 365 days available (pricing page).
+- Its streaming docs say streaming pipelines can adapt to source schema changes and propagate them to the destination, which they call "self-healing" pipelines (Maia's streaming docs).
+
+**Where it falls short**
+
+- No dollar price is shown, so ask for a quote before you plan around it. The credit terms state no price per credit, say the overage rate "will be listed on the Order Form", and call consumption rates "fixed across the customer base and non-negotiable" (credit terms).
+- Streaming CDC is limited to five relational databases (Db2 for IBM i, SQL Server, MySQL, Oracle and PostgreSQL), sits on the Scale edition, and the docs say it is only available to customers on specific Maia plans (Maia's streaming docs).
+- The streaming docs list Amazon S3, Azure Blob Storage, Google Cloud Storage and Snowflake as destinations and say change records are batched into Avro files. We found no Apache Iceberg destination in its docs.
+- Extra developer users cost credits: the credit terms list 70 credits per Developer User a month. Customer support on the Developer edition is community support, with Standard on Teams (credit terms, pricing page).
+- If a contracted term ends, the Pay-Go period cannot exceed 60 days, after which you lose access to all products unless a new term is signed, and Pay-Go includes no support (credit terms).
+- Streaming schema changes are driven by data rather than DDL, so a change to a source table is not reflected in the target until the data changes. A column dropped at the source stays in the destination, with later values inserted as NULL (Maia's streaming docs).
+
+### 9. AWS Glue: best for teams that already write Spark jobs on AWS
+
+
+
+*AWS Glue home page, October 2026.*
+
+**At a glance**
+- Price: $0.44 per data processing unit (DPU) hour, a unit of compute, for ETL jobs, and $0.308 for Glue 6.0 and above, both in US East (Ohio) (AWS Glue [price list](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/us-east-2/index.json), October 2026)
+
+AWS lists Glue pricing per Region and bills by the second. Glue 6.0 charges $0.308 an hour per unit against $0.44 for Glue 5.1, per the price list above.
+
+Here's the catch: the Glue docs say jobs created without a version default to 5.1, the higher rate.
+
+**What's good**
+
+- AWS describes Glue as serverless, with a built-in data catalog and support for reading and writing Apache Iceberg tables. We did not test these.
+- The docs call it a serverless data integration service with no infrastructure to manage, and say you can connect to more than 70 data sources (AWS Glue docs).
+- Flex jobs for non-urgent work are $0.29 per DPU-hour in US East (Ohio) against $0.44 for standard ETL jobs, and $0.203 on Glue 6.0 and above (AWS price list).
+- On the Data Catalog, the first million objects stored and the first million requests a month are free, and above that storage is $1.00 per 100,000 objects (pricing page).
+- Glue 3.0 and later supports Iceberg: its docs say you can read and write Iceberg tables in Amazon S3 or work with them through the Data Catalog, and Glue 5.1 bundles Iceberg 1.10.0 (AWS Glue Iceberg docs). Catalog optimization of Iceberg tables is billed at $0.44 per DPU-hour (pricing page).
+
+**Where it falls short**
+
+- You write Spark code or compose jobs in Glue's visual editor, and it runs only on AWS (AWS Glue docs).
+- Glue's own zero-ETL feature (copying without a separate load step) lists Oracle, SQL Server, MySQL and PostgreSQL as self-managed sources, and the docs say those can replicate only to an Amazon Redshift data warehouse ([zero-ETL docs](https://docs.aws.amazon.com/glue/latest/dg/zero-etl-using.html)).
+- Zero-ETL has no extra integration fee, but you pay for the resources behind it. The price list shows $1.50 per GB for application data ingestion and $0.44 per DPU-hour of compute for data written to S3 (price list).
+- The Iceberg page says "Update is not supported for Iceberg tables" and does not mention MERGE INTO, so test the exact operation on your Glue version (AWS Glue Iceberg docs).
+
+### 10. Azure Data Factory: best for Microsoft-centred batch pipelines
+
+
+
+*Azure Data Factory home page, October 2026.*
+
+**At a glance**
+- Price: Per activity run plus integration runtime hours; Microsoft's retail price feed shows $1.00 per 1,000 activity runs and $0.25 per data integration unit (DIU) hour of data movement in US East ([Azure Retail Prices feed](https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Azure%20Data%20Factory%20v2'%20and%20armRegionName%20eq%20'eastus'), October 2026)
+
+Microsoft's pricing page says you pay for running pipelines by activity run and for execution by integration runtime hours. The page loads rates dynamically, so the run and data movement figures above come from the Azure Retail Prices feed, and the data flow rates below come from the page's own tab table.
+
+One more thing: Microsoft's docs say that if you're new to data integration you should start with Fabric Data Factory, which it calls the next generation.
+
+**What's good**
+
+- Microsoft's docs call it a managed service for ETL, ELT and data integration projects, so loading and running pipelines sit in one place.
+- The pricing page describes the integration runtime as "serverless in Azure and self-hosted in hybrid scenarios", and charges are prorated by the minute and rounded up, so 2 minutes 20 seconds bills as 3 minutes (pricing page).
+- Data flows on General Purpose compute list $0.274 per vCore-hour pay as you go, $0.205 with a 1-year reservation (about 25% savings) and $0.178 with a 3-year reservation (about 35%) (pricing page tab table).
+- It has a change data capture resource that can run continuously. Native CDC in mapping data flows needs no timestamp or ID columns and lists SAP CDC, Azure SQL Database, SQL Server, Azure SQL Managed Instance, Azure Cosmos DB and Snowflake as connectors (Azure Data Factory's CDC docs).
+- The Copy activity can write Iceberg format through an IcebergSink (Azure Data Factory's Iceberg format docs).
+
+**Where it falls short**
+
+- Pipelines are batch only. The CDC docs say "Pipelines in ADF are batch only, but the CDC resource can run continuously."
+- The Iceberg sink supports only Azure Data Lake Storage Gen2, and the page names no Iceberg catalog such as Glue, REST, Polaris or Unity (Azure Data Factory's Iceberg format docs).
+- The self-hosted runtime runs only on Windows (Azure Data Factory's integration runtime docs).
+- Native log-based CDC for PostgreSQL, MySQL, MongoDB or Oracle is not on the CDC page's connector list, and auto incremental extraction from databases needs an incremental column to find changes (CDC docs).
+- Data flows start at a minimum cluster of 8 vCores and also bill for the managed disk and blob storage they need (pricing page).
+- Copying data out of an Azure datacenter adds a separate outbound data transfer line item (pricing page).
+
+## Transformation: reshape data once it has landed
+
+### 11. dbt: best for SQL-first teams that model data in a warehouse
+
+
+
+*dbt home page, October 2026.*
+
+**At a glance**
+- Price: Developer plan free (1 seat, 3,000 models built a month); Starter $100 per user a month (5 developer seats, 15,000 models); Enterprise custom (dbt pricing page, October 2026)
+
+dbt's docs say it transforms data that is already in your warehouse using SQL select statements, and that it works alongside your ingestion tools. It doesn't extract or load data. The free dbt Core code is under the Apache 2.0 licence.
+
+
+
+*dbt's pricing page, October 2026.*
+
+**What's good**
+
+- dbt's docs say SQL select statements are enough to define models, and the free plan or open-source core lets you try it before paying.
+- The docs say dbt is free to use, with some capabilities unlocked when you sign in with any dbt platform account, and that dbt OSS is the free, fully open-source (Apache 2.0) distribution of its Rust-based engine (dbt docs).
+- Starter at $100 per user a month includes five developer seats, 15,000 successful models built a month and 5,000 queried metrics. Enterprise raises that to 100,000 models, 20,000 queried metrics and 30 projects ([pricing page](https://www.getdbt.com/pricing)).
+- The June 1, 2026 merger press release lists dbt State (preview), a caching layer that only builds what has changed, and dbt Wizard (beta) for model authoring and debugging. The 30% infrastructure saving it claims for dbt State is the vendor's own figure (the Fivetran and dbt Labs merger press release).
+
+**Where it falls short**
+
+- Nothing happens until data is already in the warehouse, so you have to add a loader. The docs say dbt works alongside your ingestion tools.
+- The free Developer plan allows only one seat, one project and 3,000 successful models built a month (pricing page).
+- Billing counts models that build successfully even if the overall run later fails: a job of 100 models that fails after 51 are built counts as 51 (dbt's billing docs).
+- The dbt State add-on is priced on its own, at $0.094 per daily active target table pay as you go (pricing page).
+- Starter allows 1 project, against 30 on Enterprise (pricing page).
+
+## Scheduling tools: make the steps run in order
+
+### 12. Apache Airflow: best for teams that want a code-first scheduler
+
+
+
+*Apache Airflow home page, October 2026.*
+
+**At a glance**
+- Price: Free; you pay for servers and staff, or for a managed host
+
+The Airflow docs call it a platform for developing, scheduling and monitoring workflows, such as batch-oriented data pipelines. The docs showed version 3.3.2 on October 8, 2026.
+
+**What's good**
+
+- It's free, and workflows are defined as code, so you control every step.
+- The docs say it runs from a single process on your laptop to a distributed system, you can backfill Dag runs to process historical data, and you can rerun only failed tasks to minimize cost and time (Airflow docs).
+- Each Dag states when the workflow should run, its tasks and the order and conditions under which they execute (Airflow docs).
+- On AWS, the managed host bills environments by the hour at one-second resolution with "no minimum fees or upfront commitments", plus any auto-scaling for extra worker or web server capacity ([MWAA pricing](https://aws.amazon.com/managed-workflows-for-apache-airflow/pricing/)).
+
+**Where it falls short**
+
+- The docs say "if you prefer clicking over coding, Airflow might not be the best fit". It's built for batch workflows with a clear start and end, not streaming, and it doesn't copy data itself.
+- A minimal install needs a scheduler, a metadata database (usually PostgreSQL or MySQL) and an API server, and workers and a triggerer are optional extras (Airflow's core concepts docs).
+- The docs index names no connector or provider count and does not mention change data capture or Apache Iceberg (Airflow docs).
+- Managed Airflow keeps charging while the environment is on: the Small environment rate in the meter table below works out to $357.70 for a 730-hour month before extra workers (MWAA pricing).
+
+Don't fancy running Airflow yourself? Amazon's managed host, Managed Workflows for Apache Airflow (MWAA), is billed by the hour, and its pricing page lists $0.49 an hour for a Small environment and $0.99 for a Large one in US East (N. Virginia), the region its worked examples use.
+
+
+
+*Amazon MWAA pricing page, October 2026. Rates change, so check the MWAA pricing page linked above.*
+
+### 13. Dagster: best for teams that think in tables and assets rather than tasks
+
+
+
+*Dagster home page, October 2026.*
+
+**At a glance**
+- Price: Solo $10 a month, Starter $100 a month, Pro by quote; pay-as-you-go $0.040 per credit on Solo and $0.035 on Starter; 30-day free trial (Dagster pricing page, October 2026)
+
+Dagster's docs call it "a data orchestrator built for data engineers". A credit on its pricing page is the sum of asset materializations and ops run. Dagster coordinates other tools and doesn't copy data itself.
+
+
+
+*Dagster's pricing page, October 2026.*
+
+**What's good**
+
+- Its docs list built-in lineage (a record of where data came from) and observability.
+- The docs also name a declarative programming model and testability, and describe Dagster+ as the managed offering with a hosted Dagster instance (Dagster docs).
+- Solo includes 1 user, 1 code location and 1 deployment, and Starter includes up to 3 users, 5 code locations and 1 deployment. Pro is priced by contacting sales ([pricing page](https://dagster.io/pricing)).
+- With the Hybrid option there is no compute charge from Dagster (pricing page).
+
+**Where it falls short**
+
+- Each asset materialization or op run is a credit, so lots of small runs add up quickly. The cheap plans also allow just 1 or 3 users.
+- Dagster+ Serverless compute is billed at $0.010 per minute on top of the plan and credits (pricing page).
+- Both cheap plans include a single deployment, and Solo has a single code location (pricing page).
+- Dagster's pricing page says "Dagster is now part of Prefect", so confirm what that means for plans and support before you buy.
+
+### 14. Prefect: best for Python teams that want to start with ordinary functions
+
+
+
+*Prefect home page, October 2026.*
+
+**At a glance**
+- Price: Hobby free forever; Starter $100 a month; Team $100 per user a month; Enterprise custom (Prefect pricing page, October 2026)
+
+Prefect's docs call it an open-source engine that turns Python functions into pipelines, and say you build workflows in pure Python with no DSLs or complex config files. The free Hobby plan allows 2 users, up to 5 deployments and 7 days of run history.
+
+
+
+*Prefect's pricing page, October 2026.*
+
+**What's good**
+
+- Workflows are ordinary Python functions, per its docs, and the free plan covers 2 users, which is enough for a first try.
+- Its docs say you can resume interrupted runs from the last successful point, trigger flows on schedules, external events or the API, pause a flow for human approval, and create tasks at runtime based on actual data or conditions (Prefect docs).
+- Hobby also includes 500 minutes of Prefect Serverless, and Starter at $100 a month adds 3 users, up to 20 deployments and 75 hours of Serverless on your own compute ([pricing page](https://www.prefect.io/pricing)).
+- Team at $100 per user a month covers 4 to 8 users, up to 100 deployments and 225 hours of Serverless (pricing page).
+- Enterprise adds SSO (SAML/OIDC), RBAC and ACLs, multiple workspaces, directory sync (SCIM), IP allowlisting and PrivateLink, and unlimited deployments (pricing page).
+
+**Where it falls short**
+
+- It's code only, and the free plan's caps (5 deployments, 7 days of history) start to bite as you grow.
+- Run history is kept for 7 days on Hobby and Starter and 14 days on Team (pricing page).
+- The comparison table shows no uptime SLA on Hobby, Starter or Team; 99.9% appears only on Enterprise (pricing page).
+- Support is community on Hobby and Starter and email on Team, and audit log retention is none on Hobby and Starter and 24 hours on Team (pricing page).
+- Team is billed per user, so a full 8-person team comes to $800 a month at list price (our arithmetic).
+- The get-started docs do not mention change data capture, so a loader is a separate tool.
+
+## How to choose, in five questions
+
+Answer these in order. The first answer usually knocks out half the list.
+
+1. **Which job is missing?** If data isn't landing anywhere yet, start with the ingestion group. If it lands but it's messy, look at dbt. If someone runs steps by hand, look at the schedulers.
+2. **How fast must changes show up?** Hourly or slower points to a loader or a batch tool. Seconds points to Debezium, Kafka and Flink, which you run yourself.
+3. **Who runs it?** If nobody wants to touch servers, choose a managed tool or a cloud bundle. If you have engineers, free software drops the licence bill and adds the work.
+4. **Which cloud are you on?** Inside AWS or Azure, the bundles save you from joining parts.
+5. **How does the tool count?** Rows, events, credits, models and compute hours grow in different ways. Work out your own number before you sign anything.
+
+### Worked examples of the meters
+
+Plain arithmetic on the published rates above, not quotes.
+
+| Meter | Published rate | Example | Result |
+| --- | --- | --- | --- |
+| Hevo events | Free to 1M events a month; an insert, update or delete is one event | 100,000 existing rows each updated 10 times in a month | 1,000,000 update events, exactly the free ceiling, before any new inserts |
+| AWS Glue | $0.44 per DPU-hour for Glue 5.1, $0.308 for Glue 6.0 (US East, Ohio) | A 10-DPU job running 1 hour | $4.40 on Glue 5.1, $3.08 on Glue 6.0 |
+| Amazon MWAA | $0.49 an hour, Small environment (US East, N. Virginia) | One Small environment left on for 730 hours | $357.70 a month before extra workers |
+| dbt Starter | $100 per user a month, 5 developer seats | 3 developers | $300 a month; 5 seats is $500 |
+
+Sources: AWS Glue price list, Amazon MWAA pricing, dbt pricing. Checked October 2026.
+
+An hourly meter keeps charging while you sit idle, and an event or row meter charges you for churn.
+
+### Check before you rely on a claim
+
+Speed, uptime, ease and savings claims on vendor pages are the vendor's own. Before you sign, ask for the status of the exact connector you need, a bill estimate from your own volumes in writing, and what the next price step is past the free limit.
+
+## Building a pipeline from parts: a decision table
+
+Most real pipelines combine one tool per job. This table shows common combinations, built only from what each vendor's page says the tool does.
+
+| Your situation | Ingest | Transform | Orchestrate | Watch out for |
+| --- | --- | --- | --- | --- |
+| Small team, no engineer, apps and databases into a warehouse | Fivetran or Hevo | dbt | The ingestion tool's own schedule, plus dbt's job scheduling | Row and event meters rise with change |
+| Python team that wants open source | Airbyte Core | dbt Core | Airflow, Dagster or Prefect | You run servers and upgrades |
+| Everything on AWS | Glue jobs | Glue or dbt | Managed Airflow on MWAA | Spark skills; no self-hosting |
+| Everything on Microsoft | Data Factory | Data Factory mapping data flows | Data Factory triggers | Batch only; look at Fabric |
+| Changes needed in seconds | Debezium | Flink | A scheduler for any batch jobs | You run Kafka and Flink |
+| Postgres, MySQL, MongoDB or SQL Server changes into Iceberg you run | [OLake Go](/docs/) | A query engine that reads Iceberg, such as Spark or Trino ([compatibility](/docs/understanding/compatibility-engines/)) | Your own scheduler | You run the servers and upgrades |
+| One vendor for load and transform on a warehouse | Matillion | Matillion | Matillion | No public dollar price |
+
+
+
+OLake Go | Writes Iceberg and Parquet itself, per its docs |
+| Postgres, MySQL, MongoDB or SQL Server changes into Iceberg you run | OLake Go | Writes Iceberg and Parquet itself ([our docs](/docs/)) |
| Redshift loads, a small team, posted prices | Hevo | Only Snowflake-managed Iceberg, in Early Access |
| Visual pipelines run inside Redshift | Matillion | Not covered here |
| A big governed estate | Informatica | A SageMaker Lakehouse connector reads and writes Iceberg |
@@ -463,7 +464,7 @@ Want a quicker answer? Try this short quiz.
## A note on Iceberg and open table formats
-Many teams now want Apache Iceberg tables on S3, so several engines can read the same data. AWS's DMS target list does not name Iceberg, AppFlow's overview does not mention it, and Firehose takes streams rather than databases as its source now. Glue, Airbyte, Informatica and Fivetran each document a route, and the limits we found are in their entries above.
+Many teams now want Apache Iceberg tables on S3, so several engines can read the same data. AWS's [DMS target list](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Targets.html) names S3, Kinesis, Kafka and others and not Iceberg, AppFlow's overview does not mention it, and Firehose takes streams rather than databases as its source now. Glue, Airbyte, Informatica and Fivetran each document a route, and the limits we found are in their entries above.
If that is your goal, OLake Go (entry 6) writes Iceberg or Parquet on storage you own, and the [OLake Go docs](/docs/) cover the rest.
@@ -483,7 +484,7 @@ Run one real table through two finalists and write down what you see. This check
OLake Go | Move data | You | Free, Apache 2.0; you run the servers | Yes |
+| Snowflake | Store and query | Vendor | Consumption-based; $2.00 to $4.00 per credit on AWS US East | Not checked |
+| Google BigQuery | Store and query | Vendor | Per TiB scanned, or reserved capacity | First 1 TiB of queries a month |
+| Databricks | Store and query | Vendor | Pay as you go, billed per second | Free Edition |
+| Microsoft Purview | Catalog and govern | Vendor | Per governed data asset, plus processing units | None shown |
+| Collibra | Catalog and govern | Vendor | Not published; request a demo | Not published |
+| OpenMetadata | Catalog and govern | You, unless you buy a managed version | Free, Apache-2.0 | Yes |
+| DataHub | Catalog and govern | You, or DataHub Cloud | Free core, Apache-2.0; no cloud price found | Yes |
+| Great Expectations | Check quality | You | Free library, Apache-2.0 | Yes |
+| Informatica | Master data and more | Vendor | Processing units; no dollar rate on its pricing page | Its pricing page menu lists Free Cloud Data Integration |
+| Semarchy xDM | Master data | Vendor | Not published; contact for a quote | Free trial offered |
+
+Sources, pages read in October 2026:
+
+- Prices: Fivetran pricing page and Service Consumption Table, Airbyte pricing page, Snowflake credit consumption table, BigQuery pricing page, Databricks pricing page, Azure Purview pricing page, Informatica pricing page.
+- No published price: Collibra governance page and Semarchy platform page.
+- Licences on GitHub: OpenMetadata, DataHub and Great Expectations.
+
+"Free option" means a free plan or free software, not free to run.
+
+## Start from your problem
+
+Most teams don't need all five jobs on day one. So ask which pain is real, then pick from that lane.
+
+
+
+If people can't get to the data, you need a pipeline and a place to store it. If they can reach it but argue about what it means, you need a catalog, rules and a master record.
+
+| What you hear in the team | The job | Tools in this guide | Skip it if |
+| --- | --- | --- | --- |
+| "I cannot get the data from our apps and databases." | Move | Fivetran, Airbyte, OLake Go | You already have a working pipeline |
+| "Reports are slow or the data sits in too many places." | Store and query | Snowflake, BigQuery, Databricks | You only need a free tool to start |
+| "Nobody knows which table is right or who owns it." | Catalog and govern | Microsoft Purview, Collibra, OpenMetadata, DataHub | Your data is small enough to document in one sheet |
+| "The numbers broke and we found out from a customer." | Check quality | Great Expectations | You have no engineer to write rules |
+| "Sales and finance each have a different customer list." | Master data | Informatica, Semarchy xDM | Duplicates are rare |
+
+Not sure where to start? Answer five quick questions and see which tools fit.
+
+
+
+
+
+*OLake home page, October 2026.*
+
+**At a glance:** open-source database replication under [Apache 2.0](https://github.com/datazip-inc/olake). It copies Postgres, MySQL/MariaDB, MongoDB and SQL Server with a full load and then change capture ([Postgres](/docs/connectors/postgres/), [MySQL](/docs/connectors/mysql/), [MongoDB](/docs/connectors/mongodb/), [SQL Server](/docs/connectors/mssql/)), and writes Apache Iceberg or Parquet.
+
+OLake Go moves data. It is not a catalog, quality or master data tool, so pair it with the tools in the later groups. For support, [contact the OLake team](/contact/).
+
+**What's good**
+
+- The [Postgres connector docs](/docs/connectors/postgres/) list full refresh, full refresh plus CDC, CDC only and full refresh plus incremental sync modes, so one tool does the first load and then the change capture.
+- [Oracle](/docs/connectors/oracle/), [Db2](/docs/connectors/db2/), [S3](/docs/connectors/s3/) and [Kafka](/docs/connectors/kafka/) also work as sources, without database change capture.
+- It writes Iceberg tables to catalogs such as [AWS Glue](/docs/writers/iceberg/catalog/glue/), with eleven catalog types on the [catalog compatibility page](/docs/understanding/compatibility-catalogs/), or Parquet files on object storage.
+- Schema discovery runs at the start of every sync, and a new source column is added to the Iceberg schema automatically ([schema evolution docs](/docs/features/schema/)).
+- You run it on your own infrastructure with [Docker](/docs/install/docker-cli/), [Helm on Kubernetes](/docs/install/kubernetes/) or the [OLake UI](/docs/install/olake-ui/).
+
+## Tools that store and query data
+
+A data warehouse is a database built for reports. Lakehouses keep data in cheap file storage but let you query it like a warehouse. All three tools below say they work with the open Apache Iceberg table format: see the Iceberg docs of Snowflake, BigQuery and Databricks.
+
+### 4. Snowflake: for teams that want a vendor-managed warehouse
+
+
+
+*Snowflake home page, October 2026.*
+
+**At a glance:** a managed cloud data platform with a consumption-based price.
+
+Snowflake's pricing page calls its model consumption-based (its own wording: it says it keeps pricing simple), and describes Standard as an entry-level, introductory offering. Its Iceberg docs say Iceberg tables combine Snowflake's query features with external cloud storage that you manage.
+
+**What's good**
+
+- Snowflake sells four editions: Standard, Enterprise, Business Critical and Virtual Private Snowflake (VPS), which the pricing page says includes all Business Critical features in a completely separate environment.
+- The pricing page says the monthly fee for stored data uses the average amount stored per month, after compression.
+- The Iceberg docs say Snowflake supports Iceberg tables that use Parquet files and versions 1, 2 and 3 of the Iceberg specification, excluding row-level equality deletes.
+- They list two catalog options: Snowflake as the Iceberg catalog, which gives full platform support with read and write access, or an external catalog such as a remote Iceberg REST catalog, including AWS Glue and Snowflake Open Catalog.
+- They also say Snowflake supports externally managed writes for Iceberg tables that use specification version 2 or 3.
+
+**Where it falls short**
+
+- The pricing page showed $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical) per credit for AWS, US East (Northern Virginia) when we read it in a browser in October 2026, and the [credit consumption table](https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf) (Table 2(a), effective October 2, 2026) lists the same rates. There is a platform and region picker, so your rate may differ, and a credit rate is not a monthly bill.
+- Rates move with region: Table 2(a) lists AWS EU Frankfurt at $2.60, $3.90 and $5.20 for the same three editions, and Table 3(a) lists AWS US East storage at $23.00 per terabyte per month on demand.
+- Pre-paid capacity is priced as the on-demand rate with a discount set in your order form, and the table does not publish the discount percentages.
+- Per the Iceberg docs, an Iceberg table that uses an external catalog gets limited Snowflake platform support. Row-level deletes work only as position deletes on version 2 tables and as deletion vectors on version 3 tables.
+- If your Snowflake account and external volume sit in different regions, your external cloud storage account incurs egress costs when you query the table.
+- No self-hosted option is stated.
+
+
+
+*Snowflake pricing page, October 2026.*
+
+### 5. Google BigQuery: for teams already on Google Cloud
+
+
+
+*Google BigQuery home page, October 2026.*
+
+**At a glance:** a serverless warehouse. A TiB, the unit it bills queries in by default, is about a trillion bytes.
+
+**What's good**
+
+- The [pricing page](https://cloud.google.com/bigquery/pricing) says queries are billed per TiB scanned by default and "The first 1 TiB of query data processed per month is free."
+- The first 10 GiB of storage is also free each month.
+- Long-term storage covers any table or partition not modified for 90 consecutive days, and its storage price drops automatically by about 50%.
+- Capacity pricing is available in 3 editions, Standard, Enterprise and Enterprise Plus, and the Standard edition lists slots at $0.04 per hour with no commitment (default view, and the price varies by region and edition).
+- Its Iceberg docs say Iceberg managed tables keep data in customer-owned storage buckets and support DML, schema evolution (add, drop and rename columns) and automatic storage optimization.
+- They also list an export of an Iceberg V2 snapshot, refreshed on each table mutation, for engines such as Spark.
+
+**Where it falls short**
+
+- On-demand queries get "generally ... access to up to 2,000 concurrent slots, shared among all queries in a single project," so busy projects share capacity.
+- On-demand charges round up to the nearest MB, with a minimum of 10 MB per table referenced and per query, and you pay for the data in the columns you select even when you set a LIMIT.
+- The rendered on-demand rate after the free TiB is $6.25 per TiB, but the table text does not name its region, so check yours.
+- Per the Iceberg docs, only one mutating DML statement (UPDATE, DELETE or MERGE) runs at a time for each Iceberg managed table, and others queue.
+- Iceberg managed tables do not support renaming operations, table copies or table clones.
+- It's a proprietary Google Cloud service, with no self-hosted option stated.
+
+
+
+*Google BigQuery pricing page, October 2026.*
+
+### 6. Databricks: for teams that mix analytics with machine learning
+
+
+
+*Databricks home page, October 2026.*
+
+**At a glance:** a data and AI platform on AWS, Azure and Google Cloud, billed pay as you go.
+
+**What's good**
+
+- The [pricing page](https://www.databricks.com/product/pricing) says "Only pay for the products you use at per second granularity," with no up-front costs.
+- It offers committed-use contracts, where larger commitments earn greater discounts, and lists a Free Edition.
+- Its Iceberg docs say it supports Iceberg tables that use Parquet files and versions 1, 2 and 3 of the Iceberg specification (the format's own version numbers).
+- They say it supports Iceberg tables managed by Unity Catalog and by foreign catalogs, such as AWS Glue, Hive metastore or Snowflake Horizon Catalog.
+- Iceberg tables you create in Unity Catalog are managed Iceberg tables, and you can create them with Databricks Runtime, Databricks SQL or external engines that support the Iceberg REST Catalog API, such as Apache Spark, Flink, Trino or Kafka.
+
+**Where it falls short**
+
+- Iceberg tables need Unity Catalog and Databricks Runtime 16.4 LTS (long-term support) or above, and managed tables also need serverless compute.
+- Foreign Iceberg tables are read-only in Databricks and have limited platform support.
+- Credential vending is not supported on workspaces that use default storage.
+- Partition evolution on managed Iceberg tables works through external Iceberg engines using the Iceberg REST Catalog, but not through Databricks SQL.
+- Serverless compute must have network connectivity to the cloud storage account that backs a managed Iceberg table.
+- The pricing page text we read does not list editions beyond Free Edition and pay as you go.
+
+## Tools that catalog and govern data
+
+A data catalog is a searchable inventory of your data, with owners and descriptions. Data governance is the set of rules about who may use data and how. Lineage is a record of where a piece of data came from.
+
+### 7. Microsoft Purview: for teams that already run on Microsoft
+
+
+
+*Microsoft Purview product page, October 2026.*
+
+**At a glance:** Microsoft's Learn page says Purview helps organizations govern, protect and manage data. It groups security, governance and compliance products, and its governance pieces are Data Map and Unified Catalog.
+
+**What's good**
+
+- The Learn page says the Purview portal provides access to data governance, data security and data compliance solutions in one place.
+- It lists the data security solutions as Data Loss Prevention, Data Security Investigations, Information Barriers, Information Protection and Insider Risk Management.
+- The [Azure pricing page](https://azure.microsoft.com/en-us/pricing/details/purview/) bills Unified Catalog per governed asset, and the monthly charge for a managed asset is prorated by the days it is governed in the billing cycle.
+- Governed assets can be tables, views, AI models and semantic models that are linked to governance concepts.
+- A Data Map starts at 1 Capacity Unit, which supports up to 25 operations per second and up to 10 GB of metadata.
+
+**Where it falls short**
+
+- Per-unit rates didn't show in the pricing page text we fetched. Its own example says 200 governed assets cost $100 for 30 days without discounts.
+- Only assets linked to governance concepts count as governed assets, and the Data Map needs an additional Capacity Unit for every 10 GB of metadata storage.
+- Governing data in other clouds such as AWS or GCP may incur data transfer and API call charges.
+- Purview provisions a storage account and an Azure Event Hubs account as managed resources in your subscription, which may carry separate charges that in most cases will not exceed 2% of scanning charges.
+- Microsoft's Learn page lists Data Security Posture Management as in preview.
+
+
+
+*Based on the worked example on the Azure Purview pricing page, October 2026.*
+
+### 8. Collibra: for large teams with formal governance programs
+
+
+
+*Collibra home page, October 2026.*
+
+**At a glance:** a governance product that, on its product page, says it automates workflows and centralizes policies to create a single source of truth.
+
+**What's good**
+
+- Its page lists a shared language: a consistent understanding of business terminology, rules and metrics in one location.
+- It lists assigning roles and responsibilities to protect data assets from unauthorized access.
+- It lists automated compliance checks that flag violations.
+- It says AI-driven automation and workflows can speed up curation, classification and governance tasks.
+- It says workflows can be embedded into Slack and Microsoft Teams for approvals.
+- It says data issues can be submitted, tracked and resolved with automated incident workflows.
+
+**Where it falls short**
+
+- No price is shown. The page points to a "Request a demo" button, and a pricing page we tried returned an error on October 1, 2026.
+- We didn't find the deployment model stated on that page.
+- We also found no connector count on that page.
+
+### 9. OpenMetadata: for teams that want an open-source catalog
+
+
+
+*OpenMetadata home page, October 2026.*
+
+**At a glance:** the OpenMetadata docs home says to document, discover and govern your data assets end to end. Its GitHub page lists the Apache-2.0 licence.
+
+**What's good**
+
+- The docs home lists setup guides for Discovery, Lineage, Observability, Insights, Data Contract and Governance.
+- It describes managing data quality checks with version-controlled configurations.
+- It lists an upgrade path with compatibility checks and safe rollback options.
+- Its Quick Start uses a hosted sandbox or a local Docker setup, so you can try it before you deploy.
+- The GitHub page says it comes with 130+ connectors, open metadata standards, semantic search, APIs, SDKs and an MCP server.
+
+**Where it falls short**
+
+- The docs home says to deploy for production using Kubernetes, cloud-managed services or bare metal, so you run and upgrade it yourself unless you buy a managed version.
+- We didn't check any managed offering or Iceberg support, so we say nothing about either.
+
+### 10. DataHub: for teams that want an open-source catalog with a cloud option
+
+
+
+*DataHub home page, October 2026.*
+
+**At a glance:** the DataHub docs call it a modern data catalog designed to streamline metadata management, data discovery and data governance, and list self-hosting on Kubernetes plus a managed DataHub Cloud. Its GitHub page lists the Apache-2.0 licence.
+
+**What's good**
+
+- The docs say discovery can search dashboards, datasets, ML models and raw files.
+- They list governance (define ownership and track PII) and quality and observability (automated anomaly detection, assertions and data contracts).
+- They say DataHub supports ingestion by UI and CLI.
+- The GitHub page says the self-hosted option has 150+ integrations and column-level lineage.
+- DataHub's Google Cloud trial page says a 21-day trial gives you a dedicated Cloud instance managed by the DataHub team, and it lists SLA-backed 99.5% availability for DataHub Cloud.
+
+**Where it falls short**
+
+- Self-hosting means running it yourself: the docs say to start with a CLI command or deploy to production with Docker or Helm charts, and the GitHub page calls self-hosting best for teams comfortable running their own stack.
+- The trial page lists enterprise lineage views, governance workflows and AI capabilities as not available in open source.
+- We found no DataHub Cloud price (its pricing path returned an error on October 1, 2026), so we state none.
+
+### 11. Great Expectations: for teams that write Python
+
+
+
+*Great Expectations home page, October 2026.*
+
+**At a glance:** the Great Expectations docs say its Core library is a Python library for building and running data validation workflows, built around rules called expectations. The repository licence is Apache 2.0.
+
+**What's good**
+
+- The docs define an Expectation as a verifiable assertion about data.
+- Every workflow follows the same four steps: set up an environment, connect to data, define Expectations and run Validations.
+- Actions can send validation results as email, Slack or Microsoft Teams alerts.
+- Data Docs are human-readable pages that hold your Expectation Suite definitions and Validation Results.
+- You can apply the same Expectation Suite to different Batches of data.
+
+**Where it falls short**
+
+- It validates data, so you write the rules yourself and run them in code.
+- The docs' scheduled-testing example needs an orchestrator to trigger the workflow.
+- Its blog post of May 2026 announced that GX Cloud would no longer be publicly available beginning June 1.
+- The same post says Fivetran will become the steward of the open-source community and GX Core, which it says will continue as an open-source, community-driven project.
+
+## Tools that manage master data
+
+Sales has one customer list and finance has another. Master data is the core list of things your business cares about, such as customers and products, and master data management (MDM) keeps one trusted record of each so nobody holds two versions.
+
+### 12. Informatica: for large companies that want many data jobs from one vendor
+
+**At a glance:** Informatica's MDM page describes MDM as the process of creating and maintaining a single, trusted record for customers, products and more. Salesforce completed its acquisition of Informatica on November 18, 2025, per Salesforce's press release.
+
+**What's good**
+
+- Its site lists catalog, integration, quality, MDM and governance products, so it spans several of the five jobs.
+- Its [pricing page](https://www.informatica.com/pricing.html) says Informatica Processing Units (IPUs) give access to all eligible cloud services listed in its Cloud and Product Description Schedule.
+- The page's menu lists Free Cloud Data Integration.
+- The Cloud and Product Description Schedule (version 39.2, effective September 11, 2026) shows Database Ingestion and Replication CDC falling from 6.5 IPU per million rows for the first 15 million to 0.55 up to 750 million and 0.05 above that.
+
+**Where it falls short**
+
+- The pricing page explains IPUs, a usage meter, and ends with a "Get Quote" button. No dollar price per unit appears, and the rates by service live in the separate schedule, so get a quote before you budget.
+- Under the IPU model, unused IPUs do not carry over to the next month, and under the Flex IPU model they are consumed over an annual period instead.
+- Flex IPU use above your subscription is charged at the overage fee in your order, and the fee is not published.
+- IPU and Flex IPU must be provisioned separately and may not be deployed in the same Org.
+- The schedule does not say whether a volume tier applies to your whole quantity or only to the units inside each band.
+
+### 13. Semarchy xDM: for teams that want a focused MDM tool
+
+
+
+*Semarchy xDM home page, October 2026.*
+
+**At a glance:** Semarchy's platform page says it handles customer, product, supplier, location and reference data, plus any domain you model, in one model.
+
+**What's good**
+
+- Deployment choice: the page lists SaaS, on-premises, cloud and Snowflake.
+- It says you can write custom match rules or use vector embeddings for entity resolution with SemQL, its SQL-like language.
+- It says lineage is tracked down to the individual record.
+- It says generated REST APIs connect ERP, BI, ML models or external targets.
+- The page offers a free trial.
+
+**Where it falls short**
+
+- No price is published, so you'll have to contact the company for a quote.
+
+## Where all-in-one suites fit
+
+Fivetran's guide on its site argues that unified tools reduce the work of running many point tools. Keep in mind that's a vendor view. The same page calls its own list "specialized data management tools you can assemble alongside others."
+
+Choose a suite if you already have one vendor, a budget for a contract and few engineers to wire tools together. Choose separate tools if you want open source, a lower starting cost or the freedom to swap one part later.
+
+## Other names worth knowing
+
+Atlan's pricing page shows no price and a "Start the conversation" prompt. Alation and Ataccama appear in several rival lists, but we didn't check their pages, so we say no more.
+
+Qlik's own Talend page says the open-source version of Talend Studio was retired on January 31, 2024, so be wary of lists that still recommend it as a free option.
+
+## A zero-licence starting stack (and what it still costs)
+
+Here's one way to start with no licence fees, using only prices shown above. Run Airbyte Core yourself, query in BigQuery under its free 1 TiB of queries a month, catalog with OpenMetadata, and check rules with Great Expectations. The software price is $0 for each of those pieces on the free terms listed above.
+
+But "no licence" isn't "free." You still pay for servers, for BigQuery storage and any queries past the free tier, and for the engineers who keep four tools working. A paid starting point is Airbyte Cloud Standard at $20 a month with 5 credits, which swaps the server work for a bill.
+
+## How to choose a data management platform
+
+1. **Name the pain.** Data people can't reach needs a pipeline and a store. Data they don't trust needs a catalog, quality checks or master data.
+2. **Check the meter.** Rows, credits, scanned bytes, governed assets and processing units all grow differently, and that's where bills surprise people.
+3. **Count who will run it.** Open-source tools are free to license, but they still need servers and people.
+4. **Ask for the missing price in writing.** Collibra, Semarchy and Informatica each want you to ask for a quote.
+5. **Test one tool before adding a second.** Add jobs once the first one works.
+
+Snowflake, BigQuery and Databricks all document support for Apache Iceberg, an open table format for data kept in your own storage (see the Iceberg docs of Snowflake, BigQuery and Databricks). If you keep data there, the copy step matters a lot, and OLake Go, covered in the first group above, is built for it.
+
+## The short version, again
+
+A data management platform isn't one product you buy once. It's a set of jobs: move the data, store it, find it, check it and agree on one master record. Match each job to a tool, and add them in the order your problems show up.
+
+The cheapest way to start is usually a free tier or an open-source tool for the first job. And if a vendor doesn't publish a price, ask for a written quote before you commit.
+
+OLake Go | Postgres, MySQL/MariaDB, MongoDB and SQL Server ([connector docs](/docs/connectors/overview/)) | Yes, [Iceberg](/docs/writers/iceberg/catalog/rest/) tables or [Parquet](/docs/writers/parquet/config/) files |
+| OLake Go | [Postgres](/docs/connectors/postgres/), [MySQL/MariaDB](/docs/connectors/mysql/), [MongoDB](/docs/connectors/mongodb/) and [SQL Server](/docs/connectors/mssql/) | Yes, [Iceberg](/docs/writers/iceberg/catalog/rest/) tables or [Parquet](/docs/writers/parquet/config/) files |
| dlt | Postgres replication source is the only log-based option we found in its open-source docs | Yes, through [PyIceberg](https://dlthub.com/docs/dlt-ecosystem/destinations/iceberg) |
| Matillion | Streaming CDC listed on the Scale edition ([pricing page](https://www.matillion.com/pricing)) | Not listed |
| Informatica | Yes for many database sources, per its [database ingestion docs](https://docs.informatica.com/integration-cloud/data-ingestion-and-replication/current-version/database-ingestion-and-replication/database-ingestion-and-replication-source-and-target-types.html) | Describes Snowflake-managed Iceberg tables for database ingestion; the formats behind its "Open Table" target are not stated |
diff --git a/compare/2026-10-09-postgres-vs-mysql.mdx b/compare/2026-10-09-postgres-vs-mysql.mdx
new file mode 100644
index 00000000..ede12962
--- /dev/null
+++ b/compare/2026-10-09-postgres-vs-mysql.mdx
@@ -0,0 +1,294 @@
+---
+slug: postgres-vs-mysql
+title: "Postgres vs MySQL: Differences and How to Choose"
+description: "Postgres vs MySQL compared from each project's own manuals: licence, transactions, SQL features, limits, versions and when each one is the better pick."
+tags: [comparison, postgresql, mysql, versus]
+authors: [priyansh]
+image: /img/blog/cover/postgres-vs-mysql-cover.webp
+---
+import BlogCTA from '@site/src/components/BlogCTA';
+import quiz from '@site/src/data/quizzes/postgres-vs-mysql.json';
+
+
+
+OLake Go | Iceberg-compatible Parquet files on ADLS Gen2, per its Azure guide | The guide uses the Lakekeeper REST catalog |
+| OLake Go | Iceberg-compatible Parquet files on ADLS Gen2, from [our Azure guide](/docs/writers/iceberg/azure/) | Our guide uses the Lakekeeper REST catalog |
-The Fabric and Iceberg details come from Microsoft's OneLake Iceberg page.
+The Fabric and Iceberg details come from Microsoft's [OneLake Iceberg page](https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables).
## So which one should you pick?
@@ -444,7 +443,7 @@ The Fabric and Iceberg details come from Microsoft's OneLake Iceberg page.
| --- | --- | --- |
| A live, read-only copy of a supported database for Fabric reports | Fabric mirroring | Free replication compute, no pipelines |
| Scheduled, incremental loads of raw tables into Fabric | Fabric Copy job | Incremental and CDC modes, no pipelines |
-| SSIS packages, private networks or an existing ADF estate | Azure Data Factory | Fabric lists both as "To be determined" |
+| SSIS packages, private networks or an existing ADF estate | Azure Data Factory | The [compare page](https://learn.microsoft.com/en-us/fabric/data-factory/compare-fabric-data-factory-and-azure-data-factory) lists SSIS runtimes and managed virtual networks as "To be determined" in Fabric |
| Your warehouse is already a Synapse workspace | Synapse pipelines | Sits beside the warehouse |
| CDC from MySQL, PostgreSQL, SQL Server or Oracle into Databricks | Lakeflow Connect | Built-in, Delta output |
| Hundreds of SaaS connectors and no servers | Fivetran or Hevo | Managed connector libraries |
@@ -452,7 +451,7 @@ The Fabric and Iceberg details come from Microsoft's OneLake Iceberg page.
| Analysts building pipelines visually | Matillion | Visual design |
| A big, governed estate | Informatica | Its docs list ADLS Gen2, Synapse and OneLake as replication targets |
| Iceberg tables on ADLS Gen2 from one tool | ADF Iceberg sink | Check catalog support first |
-| Postgres or MySQL on Azure into Iceberg on ADLS Gen2 you run | OLake Go | Its Azure guide writes Iceberg through Lakekeeper |
+| Postgres or MySQL on Azure into Iceberg on ADLS Gen2 you run | OLake Go | Our Azure guide writes Iceberg through Lakekeeper |
OLake Go | Datastream (Google Cloud Storage), Striim (Google Cloud Storage or Amazon S3 Tables, not plain S3), Fabric mirroring, AWS DMS without a second step |
+| Iceberg on S3 | Fivetran Managed Data Lake Service, Airbyte S3 Data Lake, OLake Go | Datastream (Google Cloud Storage), Striim (Google Cloud Storage or Amazon S3 Tables, not plain S3), Fabric mirroring, AWS DMS (its [target list](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Targets.html) does not name Iceberg) |
| Monthly schema changes | Estuary and OLake Go (switch to a second instance) | Fivetran needs a new capture instance each time |
No single tool covers every row, so each path needs a compromise.
@@ -571,7 +572,7 @@ If two rows fit, shortlist both and test them.
},
{
question: "Q6. Can I send SQL Server changes to Apache Iceberg?",
- answer: "With some tools. Fivetran and Airbyte document Iceberg destinations, Datastream writes Iceberg tables in BigQuery with data in your own buckets, and Striim writes to Google Cloud Storage or Amazon S3 Tables. AWS DMS has no Iceberg target."
+ answer: <>With some tools. Fivetran and Airbyte document Iceberg destinations, Datastream writes Iceberg tables in BigQuery with data in your own buckets, and Striim writes to Google Cloud Storage or Amazon S3 Tables. The AWS DMS target list names S3, Kinesis, Kafka and others, and not Iceberg.
},
{
question: "Q7. What permissions do I need to enable SQL Server CDC?",
diff --git a/compare/2026-10-13-best-data-replication-tools.mdx b/compare/2026-10-13-best-data-replication-tools.mdx
index 3ce6122f..8ad9e84e 100644
--- a/compare/2026-10-13-best-data-replication-tools.mdx
+++ b/compare/2026-10-13-best-data-replication-tools.mdx
@@ -63,7 +63,7 @@ Most replication is one-way. The source takes the writes and the target is a rea
-AWS DMS supports two-way replication on Oracle, SQL Server, MySQL, PostgreSQL and the two Aurora editions. Its docs are blunt about the limits. It is not a full multi-master solution, it has no conflict detection or resolution, and it suits data that is operationally segregated between the two nodes.
+AWS DMS supports two-way replication on Oracle, SQL Server, MySQL, PostgreSQL and the two Aurora editions. Its [docs](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html) are blunt about the limits. It is not a full multi-master solution, it does not include conflict detection or resolution, and it suits data that is operationally segregated between the two nodes.
@@ -89,7 +89,7 @@ Cells for the other vendors state what their own docs say, and we did not test t
| AWS DMS | Log-based | Oracle, SQL Server, MySQL, PostgreSQL, MongoDB | AWS managed |
| Google Datastream | Per source | MySQL, Oracle, SQL Server, PostgreSQL, MongoDB | Google Cloud serverless |
| Debezium | Log-based | MySQL, PostgreSQL, Oracle, MongoDB and more | Self-hosted |
-| [OLake Go](/docs/connectors/overview/) | Log-based | PostgreSQL, MySQL, MongoDB, SQL Server | Self-hosted |
+| [OLake Go](/docs/) | Log-based | PostgreSQL, MySQL, MongoDB, SQL Server | Self-hosted |
| Estuary | Log-based | PostgreSQL, MySQL, Oracle, SQL Server | Managed, private or your cloud |
| Airbyte | Log-based, on a schedule | PostgreSQL, MySQL, MongoDB, SQL Server | Core self-managed, or Cloud |
| Artie | Log-based | PostgreSQL, MySQL, MongoDB, Oracle, SQL Server | Managed, or your cloud on Enterprise |
@@ -101,7 +101,7 @@ Cells for the other vendors state what their own docs say, and we did not test t
| Qlik Replicate | Many targets; Iceberg via Qlik Open Lakehouse, or a Cloudera Iceberg target | Not published (October 2026) |
| Striim | Many targets; Iceberg on Google Cloud Storage or Amazon S3 Tables | Contact us; Developer tier free (October 2026) |
| Fivetran HVA | Fivetran destinations | No separate HVA price found; Enterprise plan or higher (October 2026) |
-| AWS DMS | Many targets, S3 files; no Iceberg target | Instance or DCU hours (October 2026) |
+| AWS DMS | Many targets, S3 files; [Iceberg is not on its target list](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Targets.html) | Instance or DCU hours (October 2026) |
| Google Datastream | BigQuery, Cloud Storage, Iceberg (append-only) | Per GiB processed (October 2026) |
| Debezium | Kafka; Iceberg through a community sink | Free licence, you pay for servers |
| [OLake Go](/docs/understanding/compatibility-catalogs/) | Apache Iceberg or Parquet | Free licence, you pay for servers |
@@ -273,7 +273,7 @@ Fivetran's page on High-Volume Agent (HVA) connectors lists Db2 for i, Oracle, S
*OLake home page, October 2026.*
**At a glance**
-- Method: Log-based change capture ([connectors](/docs/connectors/overview/))
+- Method: Log-based change capture
- Price: Free licence ([Apache 2.0](https://github.com/datazip-inc/olake)); you pay for infrastructure and staff
OLake Go is open source. It reads [PostgreSQL](/docs/connectors/postgres/) logical replication, the [MySQL](/docs/connectors/mysql/) and MariaDB binlog, [MongoDB](/docs/connectors/mongodb/) change streams and [SQL Server](/docs/connectors/mssql/)'s native CDC, then writes Apache Iceberg or Parquet to storage you control.
@@ -282,7 +282,7 @@ Engines such as Snowflake, BigQuery, Redshift, Athena, Spark and Trino can read
**What's good**
-- It does the first copy and then change capture in one tool ([connectors](/docs/connectors/overview/)).
+- It does the first copy and then change capture in one tool.
- It writes eleven Iceberg catalog types, and handles deletes with equality and positional deletes plus deletion vectors since v0.11.0 ([catalogs](/docs/understanding/compatibility-catalogs/)).
- It installs with Docker Compose and a web UI, a Docker command line, or Helm on Kubernetes ([Docker Compose](/docs/install/olake-ui/), [Docker CLI](/docs/install/docker-cli/) and [Kubernetes](/docs/install/kubernetes/) install pages).
- It follows schema changes: adds, drops and renames are detected, and widening type changes are documented ([schema evolution](/docs/features/schema/)).
@@ -290,7 +290,7 @@ Engines such as Snowflake, BigQuery, Redshift, Athena, Spark and Trino can read
- It uses a two-phase commit for the Iceberg destination in full refresh, incremental and CDC runs (MongoDB, PostgreSQL, MySQL and SQL Server), tracking per-chunk commit status so syncs avoid inconsistencies and duplicate writes ([capabilities](/docs/)).
**Good to know**
-- Change capture covers PostgreSQL, MySQL and MariaDB, MongoDB and SQL Server. Oracle, Db2 and S3 sources load by full refresh or incremental runs, and output is Apache Iceberg or Parquet on storage you choose ([connectors](/docs/connectors/overview/)).
+- Change capture covers PostgreSQL, MySQL and MariaDB, MongoDB and SQL Server. Oracle, Db2 and S3 sources load by full refresh or incremental runs, and output is Apache Iceberg or Parquet on storage you choose.
**Bottom line:** Pick OLake Go if you run PostgreSQL, MySQL, MariaDB, MongoDB or SQL Server and want Iceberg tables on storage you control ([docs](/docs/understanding/compatibility-catalogs/)).
@@ -304,7 +304,7 @@ Engines such as Snowflake, BigQuery, Redshift, Athena, Spark and Trino can read
**At a glance**
- Method: Log-based
-- Price: Replication instance hours or Serverless DCU hours, plus storage. AWS's DMS pricing page lists no separate CDC fee for DMS tasks. For example, dms.t3.medium lists at $0.0745 an hour in us-east-1, on-demand, Single-AZ ([AWS price list](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSDatabaseMigrationSvc/current/us-east-1/index.json), published September 2026, read 8 October 2026)
+- Price: Replication instance hours or Serverless DCU hours, plus storage (AWS DMS pricing page, October 2026)
AWS says DMS migrates databases with one-time moves or ongoing replication. A task can do a full load first and then switch to CDC. Under the hood, its CDC docs say it reads Oracle with LogMiner or a binary reader, SQL Server through Microsoft's CDC or replication, MySQL binary logs, PostgreSQL logical slots and the MongoDB oplog.
@@ -319,10 +319,10 @@ AWS says DMS migrates databases with one-time moves or ongoing replication. A ta
**Where it falls short**
- AWS's CDC page says DMS ongoing replication has no service-level agreement for latency, that it does not provide real-time replication, and that latency can rise to several minutes or longer.
-- The two-way mode has no conflict resolution. Loopback prevention tracks only DML, not DDL, and tasks that use it cannot commit changes in batches, so `BatchApplyEnabled` must be false (CDC page).
+- The two-way mode does not include conflict detection or resolution. Loopback prevention tracks only DML, not DDL, and tasks that use it cannot commit changes in batches, so `BatchApplyEnabled` must be false (CDC page).
- Views can be migrated in full-load tasks only. A CDC-only task or a full load that goes on to CDC includes tables only (CDC page).
- PostgreSQL as a source does not support a custom CDC start time, and the CDC start point is fixed when the task is created, so a different one needs a new task. For Oracle, starting from an SCN or timestamp misses open transactions unless you use the 3.5.1 setting above (CDC page).
-- Apache Iceberg is not among the target endpoints AWS's docs list for DMS, so Iceberg needs a separate step after the files land on S3.
+- Apache Iceberg is not among the [target endpoints AWS's docs list for DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Targets.html), so Iceberg needs a separate step after the files land on S3.
- On an S3 target, DMS records no ALTER statement when a column is added, dropped or renamed and does not alter earlier records, and the docs say batch apply to S3 might lose data (S3 target docs).
#### 8. Google Cloud Datastream: BigQuery on Google Cloud
@@ -496,8 +496,8 @@ Artie's architecture page says it reads the source transaction log and buffers c
| Iceberg lake, self-hosted | Debezium Server or [OLake Go](/docs/understanding/compatibility-catalogs/) | Debezium: you need a vendor-supported sink |
| Iceberg with a vendor in charge | Qlik Open Lakehouse or Fivetran | Your query engine needs position deletes (Fivetran) or you will not use Talend Cloud (Qlik) |
| Large Oracle or mainframe estate | GoldenGate, Qlik Replicate or Striim | Iceberg must land in a plain S3 bucket (Striim: Iceberg Writer is GCS only, S3 Tables Writer needs S3 Tables), or you want a public price |
-| One-off move into AWS | AWS DMS | You need Iceberg directly, or a promised latency |
-| Two-way copies | GoldenGate, or DMS | Both sides edit the same rows (DMS has no conflict rules) |
+| One-off move into AWS | AWS DMS | You need Iceberg directly ([not on its target list](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Targets.html)), or a promised latency |
+| Two-way copies | GoldenGate, or DMS | Both sides edit the same rows ([DMS two-way tasks do not include conflict detection or resolution](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html)) |
{description}
}
- {author && (
+ {showAuthor && author && (
a.name).join(', ')}>
{authorLink ? (
diff --git a/src/data/quizzes/aws-etl-tools.json b/src/data/quizzes/aws-etl-tools.json
index 49a19804..2d60e026 100644
--- a/src/data/quizzes/aws-etl-tools.json
+++ b/src/data/quizzes/aws-etl-tools.json
@@ -25,7 +25,7 @@
"firehose",
"olake"
],
- "why": "your source is a SaaS app, and DMS reads databases while Firehose needs a stream"
+ "why": "your source is a SaaS app: DMS reads databases, Firehose takes event streams, and OLake Go moves databases"
},
{
"label": "A live database",
@@ -42,7 +42,7 @@
"appflow",
"firehose"
],
- "why": "your source is a database, and AppFlow covers SaaS apps while the Firehose database preview ended"
+ "why": "your source is a database: AppFlow's overview names SaaS apps as sources, and the Firehose database preview ended"
},
{
"label": "Events already in Kafka or Kinesis",
@@ -59,7 +59,7 @@
"hevo": 2,
"zeroetl": 1
},
- "why": "you need a long connector list",
+ "why": "you need SaaS apps too, and OLake Go moves databases, not SaaS apps",
"exclude": [
"olake"
]
@@ -78,10 +78,12 @@
"fivetran": 2,
"firehose": 2,
"airbyte": 1,
- "dms": 1,
- "glue": 1
+ "dms": 1
},
- "why": "you load Redshift"
+ "why": "you load Redshift, and OLake Go writes Iceberg and Parquet",
+ "exclude": [
+ "olake"
+ ]
},
{
"label": "Files on S3",
@@ -138,7 +140,7 @@
"airbyte": 3,
"olake": 3
},
- "why": "Airbyte Core and OLake Go are free software you run yourself"
+ "why": "Airbyte Core is free to self-host and OLake Go is open source, and you run both yourself"
},
{
"label": "An AWS service is fine",
@@ -177,7 +179,7 @@
"firehose": 2,
"olake": 1
},
- "why": "you want changes to land quickly, and these read database logs or streams"
+ "why": "you want changes to land quickly, and these tools read database logs or streams, or list short sync intervals"
}
]
},
@@ -219,28 +221,28 @@
"name": "AWS Glue",
"summary": "Serverless Spark jobs with the Glue Data Catalog, billed in DPU-hours.",
"bestWhen": "your team writes Spark or Python and your data already lives in AWS.",
- "watchOut": "we found no native log-based CDC for self-managed databases, a job needs at least 2 DPUs, and there is no free tier for ETL jobs.",
+ "watchOut": "we found no native log-based CDC for self-managed databases in its docs, a job needs at least 2 DPUs, and its pricing page shows no free tier for ETL jobs.",
"href": "#1-aws-glue-best-for-spark-code-and-batch-jobs"
},
"dms": {
"name": "AWS DMS",
"summary": "A managed service that copies a database, then keeps it current from the database logs.",
"bestWhen": "you want to move or mirror a live database inside AWS.",
- "watchOut": "it has no Apache Iceberg target, and AWS says there are no SLAs for CDC latency.",
+ "watchOut": "its target list names S3, Kinesis, Kafka and others but not Iceberg, and AWS says there are no SLAs for CDC latency.",
"href": "#2-aws-dms-best-for-moving-a-live-database-into-aws"
},
"zeroetl": {
"name": "AWS Glue zero-ETL",
- "summary": "Managed integrations that replicate SaaS apps and DynamoDB with no pipeline code.",
+ "summary": "Managed integrations that replicate SaaS apps and DynamoDB, built to minimize pipeline code.",
"bestWhen": "your sources are supported SaaS apps or DynamoDB, or a self-managed database going to Redshift.",
- "watchOut": "self-managed databases can only go to Redshift, and the source list is short.",
+ "watchOut": "for self-managed databases the docs say you can replicate only to a Redshift data warehouse, and ingestion from applications bills $1.50 per GB (US East, Ohio).",
"href": "#3-aws-glue-zero-etl-best-for-saas-and-dynamodb-data-with-no-pipeline-code"
},
"appflow": {
"name": "Amazon AppFlow",
"summary": "Managed SaaS-to-AWS flows, billed per flow run and per gigabyte.",
"bestWhen": "a handful of SaaS apps need to land in S3 or Redshift.",
- "watchOut": "its documented sources are SaaS apps, not databases, and we found no Iceberg destination named.",
+ "watchOut": "its overview names SaaS apps as sources, not databases, and does not mention Iceberg.",
"href": "#4-amazon-appflow-best-for-small-saas-pulls"
},
"firehose": {
@@ -252,21 +254,21 @@
},
"olake": {
"name": "OLake Go",
- "summary": "Open-source replication of Postgres, MySQL/MariaDB, MongoDB and SQL Server (plus Oracle, Db2, S3 and Kafka sources) into Iceberg or Parquet.",
+ "summary": "Open-source replication with change capture for Postgres, MySQL/MariaDB, MongoDB and SQL Server (Oracle, Db2, S3 and Kafka load without it), writing Iceberg or Parquet.",
"bestWhen": "your sources are databases and you want Iceberg or Parquet on storage you run.",
"watchOut": "its output is Iceberg and Parquet, so it suits teams that want one of those on storage they run.",
"href": "#6-olake-go-best-for-database-changes-into-iceberg-on-storage-you-own"
},
"fivetran": {
"name": "Fivetran",
- "summary": "A fully managed ELT service with many connectors, billed on monthly active rows.",
+ "summary": "A managed ELT service that says it runs the pipelines for you, with many connectors, billed on monthly active rows.",
"bestWhen": "you want the vendor to run everything and your row counts are predictable.",
"watchOut": "busy tables push the bill up, and the rates sit in a separate Service Consumption Table.",
"href": "#7-fivetran-best-for-many-connectors-with-no-servers"
},
"airbyte": {
"name": "Airbyte",
- "summary": "Open-source ELT with a long connector list, plus a paid cloud.",
+ "summary": "ELT with a 700+ connector list, a self-hosted Core edition (free under the Elastic License 2.0) and a paid cloud.",
"bestWhen": "you want connector choice and can host it yourself, or you like a low entry price on Cloud.",
"watchOut": "database change capture runs as scheduled syncs, not a continuous stream, per Airbyte's docs, and Core needs servers and staff.",
"href": "#8-airbyte-best-for-connector-choice-and-self-hosting"
diff --git a/src/data/quizzes/azure-etl-tools.json b/src/data/quizzes/azure-etl-tools.json
index dbbd5347..f90f02b1 100644
--- a/src/data/quizzes/azure-etl-tools.json
+++ b/src/data/quizzes/azure-etl-tools.json
@@ -19,31 +19,32 @@
"informatica": 1
},
"exclude": [
- "lakeflow"
+ "lakeflow",
+ "hevo"
],
- "why": "you land in Fabric, and Lakeflow Connect writes into Databricks instead"
+ "why": "you land in Fabric, Lakeflow Connect writes into Databricks instead, and we found no Fabric destination in Hevo's docs"
},
{
"label": "Azure Databricks",
"scores": {
- "lakeflow": 3,
+ "lakeflow": 4,
"airbyte": 2
},
- "why": "Databricks is where your analysts work"
+ "why": "you land in Databricks, where Lakeflow Connect writes Delta tables"
},
{
"label": "Azure Synapse or Azure SQL",
"scores": {
"hevo": 2,
- "fivetran": 2,
"informatica": 2,
- "adf": 2,
- "airbyte": 1
+ "adf": 2
},
"exclude": [
- "lakeflow"
+ "lakeflow",
+ "olake",
+ "airbyte"
],
- "why": "your target is not Databricks, and Lakeflow Connect writes Delta streaming tables into Databricks"
+ "why": "your target is Synapse or Azure SQL: Lakeflow Connect writes Delta streaming tables into Databricks, Airbyte lists no Synapse page, and OLake Go writes Iceberg or Parquet files"
},
{
"label": "Files in my own Azure Data Lake Storage",
@@ -53,7 +54,10 @@
"informatica": 2,
"olake": 3
},
- "why": "ADF can write Iceberg to ADLS Gen2, Fivetran's docs say its lake service writes Parquet there, OLake Go's Azure guide writes Iceberg to ADLS Gen2, and Hevo lists no ADLS destination in the docs we read"
+ "why": "ADF has an Iceberg sink to ADLS Gen2, Fivetran's lake service writes Parquet there, Informatica lists ADLS Gen2 targets, our OLake Go guide writes Iceberg to ADLS Gen2, and we found no ADLS destination in Hevo's docs",
+ "exclude": [
+ "hevo"
+ ]
}
]
},
@@ -119,14 +123,14 @@
"airbyte": 1,
"olake": 2
},
- "why": "you want database changes copied over, and OLake Go's docs cover Azure Database for PostgreSQL and MySQL Flexible Server"
+ "why": "you want database changes copied over, and these tools document change capture"
},
{
"label": "SSIS packages or a private network",
"scores": {
"adf": 3
},
- "why": "Microsoft lists SSIS and managed virtual networks as To be determined in Fabric, and ADF supports them"
+ "why": "Microsoft's compare page lists SSIS runtimes and managed virtual networks as To be determined in Fabric, and ADF runs SSIS packages with a self-hosted runtime for your own network"
}
]
},
@@ -159,9 +163,10 @@
"informatica": 3
},
"exclude": [
- "mirroring"
+ "mirroring",
+ "olake"
],
- "why": "mirroring has a single read-only destination and no transforms"
+ "why": "mirroring has a single read-only destination and no custom scheduling or table and column management, and OLake Go does not transform data on the way"
}
]
},
@@ -191,7 +196,7 @@
"scores": {
"informatica": 3
},
- "why": "Informatica publishes no dollar price per unit, so expect a quote, and its docs list several Azure replication targets"
+ "why": "we found no dollar price per unit on Informatica's pages, so expect a quote, and its docs list several Azure replication targets"
}
]
}
@@ -201,7 +206,7 @@
"name": "Fabric Copy job",
"summary": "Scheduled full, incremental or CDC copies into OneLake without building pipelines.",
"bestWhen": "Fabric is your analytics home and you want scheduled, incremental loads.",
- "watchOut": "CDC is in preview and leaves out PostgreSQL, MySQL and MongoDB, and SSIS and managed private networks are not there yet.",
+ "watchOut": "CDC is in preview and leaves out PostgreSQL, MySQL and MongoDB, and Microsoft's compare page lists SSIS runtimes and managed virtual networks as To be determined in Fabric.",
"href": "#1-fabric-data-factory-copy-jobs-into-onelake"
},
"mirroring": {
@@ -227,14 +232,14 @@
},
"olake": {
"name": "OLake Go",
- "summary": "Open-source replication of Postgres, MySQL/MariaDB, MongoDB and SQL Server (plus Oracle, Db2, S3 and Kafka sources) into Iceberg or Parquet, with an Azure guide for ADLS Gen2.",
+ "summary": "Open-source change capture from Postgres, MySQL/MariaDB, MongoDB and SQL Server into Iceberg or Parquet, with an Azure guide for ADLS Gen2.",
"bestWhen": "your sources are databases and you want Iceberg or Parquet on Azure storage you run.",
"watchOut": "its output is Iceberg and Parquet, so it suits teams that want one of those on storage they run.",
"href": "#6-olake-go-best-for-azure-databases-into-iceberg-on-your-own-storage"
},
"fivetran": {
"name": "Fivetran",
- "summary": "A fully managed ELT service with many connectors and a free plan up to a row limit.",
+ "summary": "A managed ELT service that says it runs the pipelines for you, with many connectors and a free plan up to a row limit.",
"bestWhen": "you want a vendor to run everything and your row counts are predictable.",
"watchOut": "billing is by monthly active rows, deletes count as paid rows, and its lake service has no Iceberg position deletes.",
"href": "#7-fivetran-many-connectors-with-no-servers"
diff --git a/src/data/quizzes/best-change-data-capture-tools.json b/src/data/quizzes/best-change-data-capture-tools.json
index d99db6c9..ee01b17a 100644
--- a/src/data/quizzes/best-change-data-capture-tools.json
+++ b/src/data/quizzes/best-change-data-capture-tools.json
@@ -13,20 +13,37 @@
"options": [
{
"label": "A vendor runs it for me",
- "scores": { "fivetran": 3, "estuary": 3, "airbyte": 1, "enterprise": 1 },
- "exclude": ["olake"],
- "why": "you want a vendor to run it, and OLake Go runs on your own infrastructure"
+ "scores": {
+ "fivetran": 3,
+ "estuary": 3,
+ "airbyte": 1,
+ "enterprise": 1
+ },
+ "exclude": [
+ "olake"
+ ],
+ "why": "you want a vendor to run it, and you run OLake Go yourself"
},
{
"label": "My team runs it on our own servers",
- "scores": { "debezium": 3, "flink": 3, "olake": 3, "airbyte": 2, "enterprise": 1 },
+ "scores": {
+ "debezium": 3,
+ "flink": 3,
+ "olake": 3,
+ "airbyte": 2,
+ "enterprise": 1
+ },
"why": "you are happy to self-host"
},
{
"label": "My cloud's own service (AWS or Google)",
- "scores": { "cloud": 3, "fivetran": 1, "estuary": 1 },
- "exclude": ["olake"],
- "why": "you want a cloud service, and OLake Go runs on your own infrastructure"
+ "scores": {
+ "cloud": 3
+ },
+ "exclude": [
+ "olake"
+ ],
+ "why": "you want your cloud's own service, and you run OLake Go yourself"
}
]
},
@@ -35,19 +52,40 @@
"options": [
{
"label": "Apache Iceberg tables in my own storage",
- "scores": { "olake": 3, "flink": 2, "airbyte": 2, "fivetran": 2, "debezium": 1, "estuary": 1, "enterprise": 1 },
+ "scores": {
+ "olake": 3,
+ "flink": 2,
+ "airbyte": 2,
+ "fivetran": 2,
+ "debezium": 1,
+ "estuary": 1,
+ "enterprise": 1
+ },
"why": "you want Iceberg tables you control"
},
{
"label": "A warehouse such as BigQuery",
- "scores": { "fivetran": 3, "cloud": 2, "airbyte": 2, "estuary": 2, "enterprise": 1 },
- "exclude": ["olake"],
- "why": "you want a warehouse writer, and OLake Go writes Iceberg and Parquet (its docs say engines can read those tables)"
+ "scores": {
+ "fivetran": 3,
+ "cloud": 2,
+ "airbyte": 2,
+ "estuary": 2,
+ "enterprise": 1
+ },
+ "exclude": [
+ "olake"
+ ],
+ "why": "you want a warehouse writer, and OLake Go writes Iceberg and Parquet (warehouses can query those tables, per our compatibility page)"
},
{
"label": "Kafka topics, for many consumers",
- "scores": { "debezium": 3, "flink": 2 },
- "exclude": ["olake"],
+ "scores": {
+ "debezium": 3,
+ "flink": 2
+ },
+ "exclude": [
+ "olake"
+ ],
"why": "you want a stream into Kafka, and OLake Go writes Iceberg and Parquet"
}
]
@@ -57,19 +95,42 @@
"options": [
{
"label": "Postgres, MySQL, MongoDB or SQL Server",
- "scores": { "olake": 3, "debezium": 2, "airbyte": 2, "estuary": 2, "fivetran": 2, "flink": 1, "cloud": 1 },
- "why": "these four are covered by almost every tool here"
+ "scores": {
+ "olake": 3,
+ "debezium": 2,
+ "airbyte": 2,
+ "estuary": 2,
+ "fivetran": 2,
+ "flink": 1,
+ "cloud": 1
+ },
+ "why": "the tools in the post's table list these four databases"
},
{
"label": "Oracle at large scale, or a mainframe",
- "scores": { "enterprise": 3, "debezium": 1, "estuary": 1, "fivetran": 1, "cloud": 1 },
- "exclude": ["olake"],
+ "scores": {
+ "enterprise": 3,
+ "debezium": 1,
+ "estuary": 1,
+ "fivetran": 1,
+ "cloud": 1
+ },
+ "exclude": [
+ "olake"
+ ],
"why": "you need Oracle or mainframe capture, and OLake Go's change capture covers PostgreSQL, MySQL, MongoDB and SQL Server"
},
{
"label": "Databases plus many SaaS apps",
- "scores": { "fivetran": 3, "airbyte": 3, "estuary": 2 },
- "why": "you need a long connector list"
+ "scores": {
+ "fivetran": 3,
+ "airbyte": 3,
+ "estuary": 2
+ },
+ "why": "you need SaaS apps too, and OLake Go's sources are databases, Kafka and S3",
+ "exclude": [
+ "olake"
+ ]
}
]
},
@@ -78,18 +139,36 @@
"options": [
{
"label": "Within the hour is fine",
- "scores": { "airbyte": 3, "fivetran": 1, "olake": 1, "cloud": 1 },
+ "scores": {
+ "airbyte": 3,
+ "fivetran": 1,
+ "olake": 1,
+ "cloud": 1
+ },
"why": "scheduled syncs are enough"
},
{
"label": "Every few minutes",
- "scores": { "airbyte": 1, "fivetran": 2, "estuary": 2, "olake": 1, "flink": 1, "enterprise": 1, "cloud": 1 },
+ "scores": {
+ "airbyte": 1,
+ "fivetran": 1,
+ "estuary": 2,
+ "olake": 1,
+ "flink": 1,
+ "enterprise": 1,
+ "cloud": 1
+ },
"why": "you want a short delay"
},
{
"label": "As close to continuous as I can get",
- "scores": { "debezium": 3, "estuary": 3, "flink": 2, "enterprise": 1 },
- "why": "you want a continuous stream, not scheduled syncs"
+ "scores": {
+ "debezium": 3,
+ "estuary": 3,
+ "flink": 2,
+ "enterprise": 1
+ },
+ "why": "you want a continuous stream, not scheduled syncs: Debezium and Flink CDC stream events, and Estuary says millisecond latency (a vendor claim)"
}
]
},
@@ -98,18 +177,31 @@
"options": [
{
"label": "Kafka",
- "scores": { "debezium": 3, "flink": 1 },
+ "scores": {
+ "debezium": 3,
+ "flink": 1
+ },
"why": "Debezium can run inside Kafka Connect"
},
{
"label": "Flink",
- "scores": { "flink": 3 },
+ "scores": {
+ "flink": 3
+ },
"why": "Flink CDC fits teams that already run Flink"
},
{
"label": "Neither, and I would rather not add them",
- "scores": { "olake": 2, "airbyte": 2, "fivetran": 2, "estuary": 2, "cloud": 2, "enterprise": 1 },
- "why": "you do not want to run a streaming stack just for this"
+ "scores": {
+ "olake": 2,
+ "airbyte": 2,
+ "fivetran": 2,
+ "estuary": 2,
+ "cloud": 2,
+ "enterprise": 1,
+ "debezium": 1
+ },
+ "why": "you do not want to run a streaming stack just for this (Debezium Server runs without Kafka)"
}
]
}
@@ -129,7 +221,7 @@
},
"olake": {
"name": "OLake Go",
- "summary": "Free, log-based CDC that copies database changes into Apache Iceberg or Parquet on your own storage.",
+ "summary": "Open-source, log-based CDC that copies PostgreSQL, MySQL, MongoDB and SQL Server changes into Apache Iceberg or Parquet on your own storage.",
"bestWhen": "you run PostgreSQL, MySQL, MongoDB or SQL Server and want Iceberg tables you control.",
"watchOut": "it runs on your own infrastructure and writes Iceberg and Parquet."
},
@@ -153,15 +245,15 @@
},
"enterprise": {
"name": "Striim, Qlik Replicate or GoldenGate",
- "summary": "Enterprise platforms with deep Oracle support. Qlik Replicate also lists mainframe sources.",
+ "summary": "Enterprise platforms with documented Oracle log readers. Qlik Replicate also lists mainframe sources.",
"bestWhen": "Oracle is your main source and you are fine talking to sales.",
- "watchOut": "prices are hidden or complex, and their Iceberg support has limits, such as Striim's Iceberg Writer supporting only Google Cloud Storage, and Qlik's coming mainly from a separate feature."
+ "watchOut": "Striim and Qlik publish no prices and GoldenGate is priced per processor or user, and their Iceberg support has limits: Striim's Iceberg Writer supports only Google Cloud Storage (it documents a separate S3 Tables writer), and Qlik's comes mainly from a separate feature."
},
"cloud": {
"name": "AWS DMS or Datastream",
"summary": "Cloud-native services for teams that live on one cloud.",
- "bestWhen": "you are on AWS or Google Cloud and need migration or simple replication.",
- "watchOut": "DMS has no Iceberg target and no latency guarantee, and Datastream's Iceberg output is append-only."
+ "bestWhen": "you are on AWS or Google Cloud and want migration or replication on one cloud.",
+ "watchOut": "AWS's DMS target list does not include Iceberg and its CDC page says there is no latency guarantee, and Datastream's Iceberg output is append-only."
}
}
}
diff --git a/src/data/quizzes/best-data-integration-tools.json b/src/data/quizzes/best-data-integration-tools.json
index 02a6e32c..494cfba6 100644
--- a/src/data/quizzes/best-data-integration-tools.json
+++ b/src/data/quizzes/best-data-integration-tools.json
@@ -9,42 +9,176 @@
"questions": [
{
"q": "Who will run it?",
- "hint": "The post's trade: a vendor-run tool costs money and saves staff time, while a self-run tool costs staff time.",
+ "hint": "Self-run tools have no software fee, but you pay for servers and staff time.",
"options": [
- { "label": "A vendor runs it for me", "scores": { "fivetran": 3, "hevo": 3, "estuary": 2, "airbyte": 1, "informatica": 1, "qlik": 1 }, "why": "you want a vendor to run the pipelines" },
- { "label": "My team runs it on our own servers", "scores": { "airbyte": 3, "code": 2, "olake": 2, "estuary": 1, "informatica": 1, "qlik": 1 }, "why": "your engineers will run it" },
- { "label": "Pipelines should live as code in Git", "scores": { "code": 3, "airbyte": 1 }, "why": "you want pipelines kept as code" },
- { "label": "I live inside one cloud", "scores": { "glue": 3 }, "why": "you work inside one cloud" }
+ {
+ "label": "A vendor runs it for me",
+ "scores": {
+ "fivetran": 3,
+ "hevo": 3,
+ "estuary": 2,
+ "airbyte": 1,
+ "informatica": 1,
+ "qlik": 1
+ },
+ "why": "you want a vendor to run the pipelines"
+ },
+ {
+ "label": "My team runs it on our own servers",
+ "scores": {
+ "airbyte": 3,
+ "code": 2,
+ "olake": 2,
+ "estuary": 1,
+ "informatica": 1,
+ "qlik": 1
+ },
+ "why": "your engineers will run it"
+ },
+ {
+ "label": "Pipelines should live as code in Git",
+ "scores": {
+ "code": 3,
+ "airbyte": 1
+ },
+ "why": "you want pipelines kept as code"
+ },
+ {
+ "label": "I live inside AWS or Azure",
+ "scores": {
+ "glue": 3,
+ "adf": 3
+ },
+ "why": "you work inside AWS or Azure, where the post lists Glue and Azure Data Factory"
+ }
]
},
{
"q": "What are most of your sources?",
"options": [
- { "label": "SaaS apps such as Salesforce", "scores": { "fivetran": 3, "hevo": 3, "airbyte": 3, "code": 1 }, "why": "you load many SaaS apps" },
- { "label": "Databases that need fresh data", "scores": { "estuary": 3, "qlik": 3, "fivetran": 2, "olake": 2 }, "why": "you need database changes with fresh data" },
- { "label": "Old systems under governance rules", "scores": { "informatica": 3, "qlik": 3 }, "why": "you have mainframes or governance rules" },
- { "label": "Mostly SQL, REST APIs and files", "scores": { "code": 3, "glue": 1, "airbyte": 1 }, "why": "you load SQL databases, APIs and files" }
+ {
+ "label": "SaaS apps such as Salesforce",
+ "scores": {
+ "fivetran": 3,
+ "hevo": 3,
+ "airbyte": 3,
+ "code": 1
+ },
+ "why": "you load many SaaS apps, and OLake Go reads databases, Kafka and S3",
+ "exclude": [
+ "olake"
+ ]
+ },
+ {
+ "label": "Databases that need fresh data",
+ "scores": {
+ "estuary": 3,
+ "qlik": 3,
+ "fivetran": 2,
+ "olake": 3
+ },
+ "why": "you need database changes with fresh data"
+ },
+ {
+ "label": "Old systems under governance rules",
+ "scores": {
+ "informatica": 4,
+ "qlik": 3
+ },
+ "why": "you have older systems or governance rules, and Qlik lists mainframe connectivity on Enterprise"
+ },
+ {
+ "label": "Mostly SQL, REST APIs and files",
+ "scores": {
+ "code": 3,
+ "airbyte": 1
+ },
+ "why": "you load SQL databases, APIs and files"
+ }
]
},
{
"q": "How fresh must the data be?",
"options": [
- { "label": "Hourly is fine", "scores": { "airbyte": 2, "hevo": 1, "code": 2, "glue": 1 }, "why": "scheduled syncs are enough" },
- { "label": "Within minutes", "scores": { "fivetran": 2, "estuary": 3, "qlik": 2, "informatica": 1 }, "why": "you need database changes within minutes" }
+ {
+ "label": "Hourly is fine",
+ "scores": {
+ "airbyte": 2,
+ "hevo": 1,
+ "code": 2,
+ "glue": 1,
+ "adf": 2
+ },
+ "why": "scheduled syncs are enough"
+ },
+ {
+ "label": "Within minutes",
+ "scores": {
+ "fivetran": 2,
+ "estuary": 3,
+ "qlik": 2
+ },
+ "why": "you need database changes within minutes"
+ }
]
},
{
"q": "Where must the data land?",
"options": [
- { "label": "A warehouse", "scores": { "fivetran": 2, "hevo": 2, "airbyte": 2, "informatica": 1, "code": 1 }, "why": "a warehouse is easy almost everywhere" },
- { "label": "Iceberg tables on my own catalog", "scores": { "qlik": 3, "glue": 3, "olake": 3, "airbyte": 2, "fivetran": 2, "estuary": 2, "code": 1, "informatica": 1 }, "why": "you want Iceberg tables, which shrinks the field" }
+ {
+ "label": "A warehouse",
+ "scores": {
+ "fivetran": 2,
+ "hevo": 2,
+ "airbyte": 2,
+ "informatica": 1,
+ "code": 1
+ },
+ "why": "you load a warehouse, and OLake Go writes Iceberg and Parquet",
+ "exclude": [
+ "olake"
+ ]
+ },
+ {
+ "label": "Iceberg tables",
+ "scores": {
+ "qlik": 3,
+ "glue": 3,
+ "olake": 3,
+ "airbyte": 2,
+ "fivetran": 1,
+ "estuary": 2,
+ "code": 1,
+ "informatica": 1
+ },
+ "why": "you want Iceberg tables, which shrinks the field to the writers in the post's second table"
+ }
]
},
{
"q": "Does your data have to stay in your network?",
"options": [
- { "label": "Yes, I need to host it myself", "scores": { "airbyte": 2, "code": 2, "olake": 2, "informatica": 1, "qlik": 1, "estuary": 1 }, "why": "you want to host it yourself, and the post marks Airbyte Core, Meltano, dlt and OLake Go as self-run" },
- { "label": "No, a hosted service is fine", "scores": { "fivetran": 2, "hevo": 2, "estuary": 1 }, "why": "a hosted service is fine for you, so vendor-run tools stay in play" }
+ {
+ "label": "Yes, I need to host it myself",
+ "scores": {
+ "airbyte": 2,
+ "code": 2,
+ "olake": 2,
+ "informatica": 1,
+ "qlik": 1,
+ "estuary": 1
+ },
+ "why": "you want to host it yourself: Airbyte Core, Meltano, dlt and OLake Go run on your own servers, and Informatica, Qlik and Estuary list self-run options"
+ },
+ {
+ "label": "No, a hosted service is fine",
+ "scores": {
+ "fivetran": 2,
+ "hevo": 2,
+ "estuary": 1
+ },
+ "why": "a hosted service is fine for you, so vendor-run tools stay in play"
+ }
]
}
],
@@ -53,12 +187,12 @@
"name": "Fivetran",
"summary": "A vendor-run tool with a big connector list, billed on monthly active rows per connection.",
"bestWhen": "nobody wants to run pipelines and your sources are mostly SaaS.",
- "watchOut": "Oracle, SAP and Db2 sources are only on Enterprise and Business Critical, and since January 2026 deleted rows count as paid MAR.",
+ "watchOut": "Oracle, SAP and Db2 (Beta) sources are only on Enterprise and Business Critical, and since January 2026 deleted rows count as paid MAR.",
"href": "#1-fivetran-best-for-saas-sources-and-a-hands-off-setup"
},
"hevo": {
"name": "Hevo",
- "summary": "A managed tool with a permanent free plan and entry prices you can see.",
+ "summary": "A managed tool with a free plan and entry prices you can see.",
"bestWhen": "you want a managed tool with a small, published price.",
"watchOut": "streaming must be enabled through sales, Iceberg is Early Access and Snowflake-managed only, and we found no self-hosted edition.",
"href": "#3-hevo-best-for-a-low-entry-price-and-a-free-plan"
@@ -67,7 +201,7 @@
"name": "Airbyte",
"summary": "A big connector list you can self-host as Core or rent as Cloud.",
"bestWhen": "you want lots of connectors and the option to self-host.",
- "watchOut": "syncs are scheduled, not continuous (hourly at best apart from 15-minute syncs on Cloud Plus), and Core uses the Elastic License 2.0, which does not let you sell a cloud service that gives users direct access to Airbyte.",
+ "watchOut": "syncs are scheduled, not continuous (its pricing page lists 1 hour on Standard, 15 minutes on Plus and 5 minutes on Pro and Enterprise Flex), and Core uses the Elastic License 2.0, which does not let you sell a cloud service that gives users direct access to Airbyte.",
"href": "#2-airbyte-best-for-a-big-connector-list-you-can-self-host-or-rent"
},
"estuary": {
@@ -87,7 +221,7 @@
"informatica": {
"name": "Informatica",
"summary": "A large suite for governance and big, mixed estates.",
- "bestWhen": "you need governance, mainframes and a big mixed estate.",
+ "bestWhen": "you need governance and a big mixed estate.",
"watchOut": "we found no dollar price per IPU on its site, so you need a quote, and its database ingestion docs describe Iceberg only as Snowflake-managed tables.",
"href": "#13-informatica-best-for-governance-and-large-mixed-estates"
},
@@ -95,12 +229,12 @@
"name": "AWS Glue",
"summary": "A serverless AWS service for Spark jobs that writes Iceberg to S3.",
"bestWhen": "your team writes Spark and lives on AWS.",
- "watchOut": "we found no native log-based CDC engine, and jobs need storage and a catalog as well.",
+ "watchOut": "we found no log-based CDC engine documented for Glue ETL jobs, and jobs need storage and a catalog as well.",
"href": "#15-aws-glue-best-for-spark-jobs-inside-aws"
},
"olake": {
"name": "OLake Go",
- "summary": "An open-source tool that copies Postgres, MySQL, MongoDB and SQL Server changes into Iceberg or Parquet on storage you own.",
+ "summary": "An open-source tool that captures Postgres, MySQL, MongoDB and SQL Server changes into Iceberg or Parquet on storage you own.",
"bestWhen": "you run your own infrastructure and want database changes in Iceberg tables.",
"watchOut": "it reads databases, Kafka and S3 rather than SaaS apps, and it does not transform data.",
"href": "#10-olake-go-best-for-database-changes-into-iceberg-or-parquet-on-your-own-storage"
@@ -109,8 +243,15 @@
"name": "Meltano or dlt",
"summary": "Free code-first tools: Meltano keeps pipelines in Git, and dlt is a Python library.",
"bestWhen": "your team likes code and wants control.",
- "watchOut": "you own scheduling, upgrades and on-call work, we found no official Meltano Iceberg loader, and dlt's open-source CDC is Postgres only.",
+ "watchOut": "you own scheduling, upgrades and on-call work, we found no official Meltano Iceberg loader, and the log-based option we found in dlt's open-source docs is Postgres.",
"href": "#9-meltano-best-for-pipelines-kept-as-code"
+ },
+ "adf": {
+ "name": "Azure Data Factory",
+ "summary": "Microsoft's managed ETL and ELT service, billed on activity runs, copy hours and data flow hours.",
+ "bestWhen": "you build pipelines inside Azure.",
+ "watchOut": "native CDC covers SAP, Azure SQL, SQL Server, Cosmos DB and Snowflake, and its Copy activity writes Iceberg only to Azure Data Lake Storage Gen2.",
+ "href": "#16-azure-data-factory-best-for-pipelines-inside-azure"
}
}
}
diff --git a/src/data/quizzes/best-data-lake-tools.json b/src/data/quizzes/best-data-lake-tools.json
index d7c52fd5..39afbea9 100644
--- a/src/data/quizzes/best-data-lake-tools.json
+++ b/src/data/quizzes/best-data-lake-tools.json
@@ -15,7 +15,9 @@
"label": "Amazon Web Services",
"scores": {
"athena": 3,
- "dremio": 1
+ "dremio": 1,
+ "snowflake": 1,
+ "databricks": 1
},
"why": "you are on AWS"
},
@@ -66,7 +68,7 @@
"why": "you want one vendor and one bill"
},
{
- "label": "We must self-host",
+ "label": "We would rather host it ourselves",
"scores": {
"trino": 3,
"dremio": 2,
@@ -101,7 +103,7 @@
"scores": {
"fabric": 3
},
- "why": "you use Power BI and want governance in one place"
+ "why": "you use Power BI and Microsoft tools, and want one lake"
},
{
"label": "Engineers who like open source",
@@ -118,13 +120,14 @@
"hint": "\"Supports Iceberg\" can mean read only.",
"options": [
{
- "label": "Many engines must write Iceberg",
+ "label": "Many engines must work on the same Iceberg tables",
"scores": {
"trino": 3,
"snowflake": 1,
- "olake": 2
+ "olake": 2,
+ "databricks": 1
},
- "why": "you want to switch or mix engines later"
+ "why": "you want to mix engines later: Trino reads and writes Iceberg, and OLake Go writes tables other engines read"
},
{
"label": "One engine does the writing",
@@ -208,14 +211,14 @@
"athena": {
"name": "Amazon S3 plus Athena",
"summary": "Object storage with a query engine you do not run, billed on data scanned. Lake Formation can add access rules.",
- "bestWhen": "you are on AWS, ask one-off questions and have low, spiky usage.",
- "watchOut": "Athena works on Iceberg v2 tables only and only with the AWS Glue catalog.",
+ "bestWhen": "you are on AWS and run occasional SQL over files in S3.",
+ "watchOut": "Athena's docs say it creates and operates only on Iceberg v2 tables, and only tables created against the AWS Glue catalog work.",
"href": "#8-amazon-athena-best-for-occasional-sql-over-files-in-s3"
},
"fabric": {
"name": "Microsoft Fabric OneLake",
"summary": "One lake per Fabric tenant, built on Azure Data Lake Storage, with capacity billed per workspace.",
- "bestWhen": "you use Power BI and want governance in one place.",
+ "bestWhen": "you use Power BI and want one lake.",
"watchOut": "Fabric engines store data in Delta Parquet, so Iceberg is bridged rather than the default.",
"href": "#14-microsoft-fabric-onelake-best-for-microsoft-teams-that-want-one-lake"
},
@@ -223,7 +226,7 @@
"name": "Google BigQuery",
"summary": "A Google Cloud platform with Iceberg managed tables kept in your own buckets, billed per TiB scanned or by slots.",
"bestWhen": "your analysts use BigQuery and you want Iceberg files in your own Cloud Storage.",
- "watchOut": "only one UPDATE, DELETE or MERGE runs on a table at a time, and renaming or cloning is not supported.",
+ "watchOut": "the docs say only one UPDATE, DELETE or MERGE runs on a table at a time, and renaming, copying and cloning are not supported.",
"href": "#15-google-bigquery-best-for-sql-teams-on-google-cloud-who-want-iceberg-tables-in-their-own-buckets"
},
"trino": {
@@ -256,7 +259,7 @@
},
"olake": {
"name": "OLake Go plus a catalog and an engine",
- "summary": "Open-source copy of Postgres, MySQL and MariaDB, MongoDB and SQL Server data (plus Oracle, Db2, S3 and Kafka sources) into Iceberg or Parquet on storage you run.",
+ "summary": "Open-source tool that captures changes from Postgres, MySQL and MariaDB, MongoDB and SQL Server (Oracle, Db2, S3 and Kafka load without database change capture) into Iceberg or Parquet on storage you run.",
"bestWhen": "your data starts in a database and you want Iceberg tables that Trino, Athena or another engine can read.",
"watchOut": "OLake Go loads the lake, so you still pick a catalog and a query engine.",
"href": "#10-olake-go-best-for-getting-database-changes-into-an-iceberg-lake"
diff --git a/src/data/quizzes/best-data-management-platforms.json b/src/data/quizzes/best-data-management-platforms.json
new file mode 100644
index 00000000..26dc11db
--- /dev/null
+++ b/src/data/quizzes/best-data-management-platforms.json
@@ -0,0 +1,357 @@
+{
+ "id": "best-data-management-platforms",
+ "mode": "recommend",
+ "title": "Which data management tool fits you?",
+ "updated": "October 2026",
+ "guideHref": "#start-from-your-problem",
+ "guideLabel": "See the problem-to-tool table",
+ "topN": 3,
+ "questions": [
+ {
+ "q": "What is the pain you have today?",
+ "hint": "The post says to start from the pain, because no single tool does every job.",
+ "options": [
+ {
+ "label": "I cannot get the data from our apps and databases",
+ "scores": {
+ "fivetran": 3,
+ "airbyte": 3,
+ "olake": 3,
+ "informatica": 1
+ },
+ "why": "your pain is getting data out of apps and databases, which the post gives to Fivetran, Airbyte and OLake Go",
+ "exclude": [
+ "bigquery",
+ "snowflake",
+ "databricks",
+ "purview",
+ "openmetadata",
+ "datahub",
+ "collibra",
+ "greatexpectations",
+ "semarchy"
+ ]
+ },
+ {
+ "label": "Reports are slow or the data sits in too many places",
+ "scores": {
+ "bigquery": 3,
+ "snowflake": 3,
+ "databricks": 3,
+ "informatica": 1
+ },
+ "why": "your pain is slow reports and scattered data, which the post gives to Snowflake, BigQuery and Databricks",
+ "exclude": [
+ "fivetran",
+ "airbyte",
+ "olake",
+ "purview",
+ "openmetadata",
+ "datahub",
+ "collibra",
+ "greatexpectations",
+ "semarchy"
+ ]
+ },
+ {
+ "label": "Nobody knows which table is right or who owns it",
+ "scores": {
+ "purview": 3,
+ "openmetadata": 3,
+ "datahub": 3,
+ "collibra": 3,
+ "informatica": 1
+ },
+ "why": "your pain is not knowing which table is right, which the post gives to Purview, Collibra, OpenMetadata and DataHub",
+ "exclude": [
+ "fivetran",
+ "airbyte",
+ "olake",
+ "bigquery",
+ "snowflake",
+ "databricks",
+ "greatexpectations",
+ "semarchy"
+ ]
+ },
+ {
+ "label": "The numbers broke and we heard it from a customer",
+ "scores": {
+ "greatexpectations": 3,
+ "informatica": 1
+ },
+ "why": "your pain is broken numbers, which the post gives to Great Expectations",
+ "exclude": [
+ "fivetran",
+ "airbyte",
+ "olake",
+ "bigquery",
+ "snowflake",
+ "databricks",
+ "purview",
+ "openmetadata",
+ "datahub",
+ "collibra",
+ "semarchy"
+ ]
+ },
+ {
+ "label": "Sales and finance have different customer lists",
+ "scores": {
+ "informatica": 3,
+ "semarchy": 3
+ },
+ "why": "your pain is two customer lists, which the post gives to Informatica and Semarchy",
+ "exclude": [
+ "fivetran",
+ "airbyte",
+ "olake",
+ "bigquery",
+ "snowflake",
+ "databricks",
+ "purview",
+ "openmetadata",
+ "datahub",
+ "collibra",
+ "greatexpectations"
+ ]
+ }
+ ]
+ },
+ {
+ "q": "Who should run it?",
+ "options": [
+ {
+ "label": "A vendor runs it for me",
+ "scores": {
+ "fivetran": 3,
+ "bigquery": 3,
+ "purview": 3,
+ "informatica": 2,
+ "datahub": 1,
+ "snowflake": 3,
+ "databricks": 3,
+ "semarchy": 2
+ },
+ "why": "you want the vendor to run it"
+ },
+ {
+ "label": "My team runs it",
+ "scores": {
+ "airbyte": 3,
+ "openmetadata": 3,
+ "datahub": 3,
+ "greatexpectations": 3,
+ "olake": 3
+ },
+ "why": "you can run it yourself"
+ },
+ {
+ "label": "Either, I want a managed option and a self-run option",
+ "scores": {
+ "airbyte": 3,
+ "datahub": 3,
+ "semarchy": 3
+ },
+ "why": "it offers a vendor-run route and a self-run route"
+ }
+ ]
+ },
+ {
+ "q": "What does your budget look like?",
+ "options": [
+ {
+ "label": "Free software, we will pay in servers and people",
+ "scores": {
+ "openmetadata": 3,
+ "greatexpectations": 3,
+ "datahub": 2,
+ "airbyte": 2,
+ "olake": 3
+ },
+ "why": "it is free, Apache-2.0 or free to run yourself"
+ },
+ {
+ "label": "A free tier or published price I can check",
+ "scores": {
+ "bigquery": 3,
+ "fivetran": 2,
+ "airbyte": 2,
+ "databricks": 2,
+ "purview": 1,
+ "snowflake": 2
+ },
+ "why": "it has a free plan or a price you can read"
+ },
+ {
+ "label": "A quote and a contract are fine",
+ "scores": {
+ "informatica": 3,
+ "collibra": 3,
+ "semarchy": 3
+ },
+ "why": "its price comes from a quote"
+ }
+ ]
+ },
+ {
+ "q": "Where does your data have to live?",
+ "options": [
+ {
+ "label": "On Google Cloud",
+ "scores": {
+ "bigquery": 3
+ },
+ "why": "you are already on Google Cloud"
+ },
+ {
+ "label": "On Microsoft",
+ "scores": {
+ "purview": 3
+ },
+ "why": "Microsoft is already your home base"
+ },
+ {
+ "label": "No preference",
+ "scores": {
+ "fivetran": 1,
+ "airbyte": 1,
+ "openmetadata": 1,
+ "datahub": 1,
+ "olake": 1
+ },
+ "why": "you have no cloud preference"
+ },
+ {
+ "label": "I would rather keep it in our own environment",
+ "scores": {
+ "airbyte": 2,
+ "openmetadata": 3,
+ "datahub": 2,
+ "greatexpectations": 3,
+ "semarchy": 2,
+ "olake": 3
+ },
+ "why": "you can run it in your own environment"
+ }
+ ]
+ },
+ {
+ "q": "What skills does your team have?",
+ "options": [
+ {
+ "label": "Engineers who write Python",
+ "scores": {
+ "greatexpectations": 3,
+ "airbyte": 2
+ },
+ "why": "your engineers can write the rules and code"
+ },
+ {
+ "label": "Platform engineers who run services",
+ "scores": {
+ "openmetadata": 3,
+ "datahub": 3,
+ "airbyte": 1,
+ "olake": 2
+ },
+ "why": "you can run services on Kubernetes"
+ },
+ {
+ "label": "Few engineers",
+ "scores": {
+ "fivetran": 3,
+ "informatica": 2,
+ "purview": 2,
+ "bigquery": 2,
+ "snowflake": 2,
+ "databricks": 1,
+ "collibra": 1,
+ "semarchy": 2
+ },
+ "why": "you want a vendor to carry the work"
+ }
+ ]
+ }
+ ],
+ "results": {
+ "fivetran": {
+ "name": "Fivetran",
+ "summary": "A managed service that moves your data using ready-made connectors.",
+ "bestWhen": "You want a managed pipeline service and can accept per-row billing.",
+ "watchOut": "Paid per-row rates sit in a separate table, so your bill depends on your row counts and connections."
+ },
+ "airbyte": {
+ "name": "Airbyte",
+ "summary": "Open-source connectors you can self-host, or pay for as Cloud.",
+ "bestWhen": "You have engineers and want control or a lower starting price.",
+ "watchOut": "Its CDC docs say syncs are scheduled and that CDC incremental needs primary keys for most sources, and Core uses the ELv2 licence, which restricts resale as a hosted service."
+ },
+ "olake": {
+ "name": "OLake Go",
+ "summary": "Open-source tool that captures Postgres, MySQL/MariaDB, MongoDB and SQL Server changes into Iceberg or Parquet (Oracle, Db2, S3 and Kafka load without database change capture).",
+ "bestWhen": "Your data sits in databases and you want Iceberg or Parquet on storage you run.",
+ "watchOut": "It moves data, so catalog, quality and master data are other tools' jobs."
+ },
+ "bigquery": {
+ "name": "Google BigQuery",
+ "summary": "A serverless warehouse billed per TiB scanned, with the first 1 TiB of queries a month free.",
+ "bestWhen": "You are on Google Cloud and want pay-per-query.",
+ "watchOut": "It is a Google Cloud service with no self-hosted option stated, and busy projects share query capacity."
+ },
+ "purview": {
+ "name": "Microsoft Purview",
+ "summary": "A catalog and governance product that sits next to security and compliance tools.",
+ "bestWhen": "Microsoft is already your home base.",
+ "watchOut": "Billing is per governed asset, and its Learn page lists Data Security Posture Management as in preview."
+ },
+ "openmetadata": {
+ "name": "OpenMetadata",
+ "summary": "A free, Apache-2.0 catalog for discovery, lineage, quality checks and governance.",
+ "bestWhen": "You have platform engineers and want no licence fee.",
+ "watchOut": "The docs say to deploy for production using Kubernetes, cloud-managed services or bare metal, so you run and upgrade it yourself."
+ },
+ "datahub": {
+ "name": "DataHub",
+ "summary": "An Apache-2.0 catalog you can self-host, with a managed DataHub Cloud route.",
+ "bestWhen": "You want open source now and a managed option later.",
+ "watchOut": "We found no DataHub Cloud price, and self-hosting means running it yourself, with Docker or Helm charts."
+ },
+ "greatexpectations": {
+ "name": "Great Expectations",
+ "summary": "A free Python library that checks your data against rules you write.",
+ "bestWhen": "Engineers own your pipelines.",
+ "watchOut": "It is a Python library for data validation, so you write the rules and run them in code."
+ },
+ "informatica": {
+ "name": "Informatica",
+ "summary": "A large vendor with master data, catalog, integration and quality products.",
+ "bestWhen": "You want a single large vendor and a negotiated contract.",
+ "watchOut": "No dollar price per unit appears on its pricing page, so you need a quote."
+ },
+ "snowflake": {
+ "name": "Snowflake",
+ "summary": "A managed cloud data platform with consumption-based pricing.",
+ "bestWhen": "You want a managed warehouse.",
+ "watchOut": "Credit rates vary by edition and region, and no self-hosted option is stated."
+ },
+ "databricks": {
+ "name": "Databricks",
+ "summary": "A data and AI platform on AWS, Azure and Google Cloud, billed pay as you go.",
+ "bestWhen": "Your team mixes analytics with machine learning.",
+ "watchOut": "Managed Iceberg tables need Unity Catalog, Databricks Runtime 16.4 LTS or newer and serverless compute."
+ },
+ "collibra": {
+ "name": "Collibra",
+ "summary": "A governance product with shared business terms, assigned roles, automated compliance checks and incident workflows.",
+ "bestWhen": "You are a large team with a formal governance program.",
+ "watchOut": "No price is shown and the page points to a demo request, so ask for a quote."
+ },
+ "semarchy": {
+ "name": "Semarchy xDM",
+ "summary": "A focused master data tool offered as SaaS or on-premises.",
+ "bestWhen": "Master data is your main problem.",
+ "watchOut": "No price is published, so you need to ask for a quote."
+ }
+ }
+}
diff --git a/src/data/quizzes/best-data-pipeline-tools.json b/src/data/quizzes/best-data-pipeline-tools.json
new file mode 100644
index 00000000..e888dc23
--- /dev/null
+++ b/src/data/quizzes/best-data-pipeline-tools.json
@@ -0,0 +1,357 @@
+{
+ "id": "best-data-pipeline-tools",
+ "mode": "recommend",
+ "title": "Which pipeline tool fits you?",
+ "updated": "October 2026",
+ "guideHref": "#building-a-pipeline-from-parts-a-decision-table",
+ "guideLabel": "See the full decision table",
+ "topN": 3,
+ "questions": [
+ {
+ "q": "Which job in your pipeline is missing?",
+ "hint": "A pipeline has four jobs: ingest, store, transform and orchestrate. Most tools only do one.",
+ "options": [
+ {
+ "label": "Copying data in from databases or apps",
+ "scores": {
+ "fivetran": 3,
+ "airbyte": 3,
+ "debezium": 2,
+ "matillion": 2,
+ "glue": 2,
+ "adf": 2,
+ "hevo": 3,
+ "olake": 3
+ },
+ "why": "you need to copy data in, and the post's table lists dbt, Airflow, Dagster and Prefect as transform or scheduling tools",
+ "exclude": [
+ "dbt",
+ "airflow",
+ "dagster",
+ "prefect"
+ ]
+ },
+ {
+ "label": "Reshaping data that is already in a warehouse",
+ "scores": {
+ "dbt": 3,
+ "matillion": 2,
+ "glue": 1,
+ "adf": 1
+ },
+ "why": "your data has landed and needs reshaping, and the post's table lists these tools' main job as ingest or scheduling",
+ "exclude": [
+ "fivetran",
+ "airbyte",
+ "hevo",
+ "olake",
+ "debezium",
+ "airflow",
+ "dagster",
+ "prefect"
+ ]
+ },
+ {
+ "label": "Running the steps in order and retrying failures",
+ "scores": {
+ "airflow": 3,
+ "adf": 1,
+ "matillion": 1,
+ "dagster": 3,
+ "prefect": 3
+ },
+ "why": "you need something to run the steps in order, and the post's table lists these tools' main job as ingest or transform",
+ "exclude": [
+ "fivetran",
+ "airbyte",
+ "hevo",
+ "olake",
+ "debezium",
+ "dbt"
+ ]
+ },
+ {
+ "label": "All of it, from one vendor",
+ "scores": {
+ "matillion": 3,
+ "glue": 3,
+ "adf": 3
+ },
+ "why": "you want several jobs from one vendor, and the post lists Matillion, AWS Glue and Azure Data Factory as the bundles",
+ "exclude": [
+ "fivetran",
+ "airbyte",
+ "hevo",
+ "olake",
+ "debezium",
+ "dbt",
+ "airflow",
+ "dagster",
+ "prefect"
+ ]
+ }
+ ]
+ },
+ {
+ "q": "Who will run it?",
+ "hint": "Free software still has a bill: servers and people.",
+ "options": [
+ {
+ "label": "A vendor runs it for us",
+ "scores": {
+ "fivetran": 3,
+ "matillion": 2,
+ "glue": 2,
+ "adf": 2,
+ "airbyte": 1,
+ "hevo": 3,
+ "dagster": 1,
+ "prefect": 1
+ },
+ "why": "you want someone else to run it"
+ },
+ {
+ "label": "We run it ourselves, on our own servers",
+ "scores": {
+ "debezium": 3,
+ "airbyte": 3,
+ "airflow": 3,
+ "dbt": 2,
+ "dagster": 2,
+ "prefect": 2,
+ "olake": 3
+ },
+ "exclude": [
+ "glue"
+ ],
+ "why": "you want to self-host, and AWS Glue runs only on AWS"
+ }
+ ]
+ },
+ {
+ "q": "How fast must changes show up?",
+ "options": [
+ {
+ "label": "Hourly or slower is fine",
+ "scores": {
+ "airbyte": 2,
+ "fivetran": 2,
+ "adf": 2,
+ "glue": 2,
+ "airflow": 1,
+ "dbt": 1,
+ "hevo": 2,
+ "dagster": 1,
+ "prefect": 1
+ },
+ "why": "hourly or slower is enough"
+ },
+ {
+ "label": "Every few minutes",
+ "scores": {
+ "fivetran": 2,
+ "matillion": 1,
+ "airbyte": 1
+ },
+ "why": "you need syncs every few minutes"
+ },
+ {
+ "label": "Within seconds",
+ "scores": {
+ "debezium": 3
+ },
+ "exclude": [
+ "fivetran",
+ "airbyte",
+ "matillion",
+ "glue",
+ "adf",
+ "dbt",
+ "airflow",
+ "hevo",
+ "dagster",
+ "prefect",
+ "olake"
+ ],
+ "why": "you need seconds, and the post points seconds to Debezium, Kafka and Flink, which you run yourself"
+ }
+ ]
+ },
+ {
+ "q": "Where does your data live?",
+ "options": [
+ {
+ "label": "Everything is on AWS",
+ "scores": {
+ "glue": 3,
+ "airflow": 1
+ },
+ "why": "your data and skills are on AWS"
+ },
+ {
+ "label": "Everything is on Azure",
+ "scores": {
+ "adf": 3
+ },
+ "why": "you live on Azure"
+ },
+ {
+ "label": "In a cloud warehouse such as Snowflake or BigQuery",
+ "scores": {
+ "matillion": 2,
+ "dbt": 2,
+ "fivetran": 1,
+ "hevo": 2
+ },
+ "why": "you work inside a cloud warehouse"
+ },
+ {
+ "label": "Not tied to one cloud",
+ "scores": {
+ "airbyte": 1,
+ "debezium": 1,
+ "airflow": 1,
+ "dagster": 1,
+ "prefect": 1,
+ "olake": 2
+ },
+ "exclude": [
+ "glue"
+ ],
+ "why": "you want to stay cloud-neutral, and AWS Glue runs only on AWS"
+ }
+ ]
+ },
+ {
+ "q": "What kind of bill do you want?",
+ "hint": "Rows, events, credits, models and compute hours all grow in different ways.",
+ "options": [
+ {
+ "label": "Free software, and we pay in staff time",
+ "scores": {
+ "debezium": 2,
+ "airbyte": 2,
+ "airflow": 2,
+ "dbt": 2,
+ "prefect": 2,
+ "olake": 3,
+ "dagster": 2
+ },
+ "why": "you prefer free software over a licence bill"
+ },
+ {
+ "label": "A public price I can plan around",
+ "scores": {
+ "airbyte": 2,
+ "dbt": 2,
+ "glue": 2,
+ "adf": 2,
+ "airflow": 1,
+ "hevo": 2,
+ "dagster": 2,
+ "prefect": 2
+ },
+ "exclude": [
+ "matillion"
+ ],
+ "why": "you need a public price, and Matillion's pricing page shows no dollar price"
+ },
+ {
+ "label": "I will pay for hands-off loading",
+ "scores": {
+ "fivetran": 2,
+ "matillion": 1,
+ "hevo": 2
+ },
+ "why": "you will budget for hands-off loading"
+ }
+ ]
+ }
+ ],
+ "results": {
+ "fivetran": {
+ "name": "Fivetran",
+ "summary": "A managed loader with a free plan, 700+ connectors and 15-minute syncs on Standard.",
+ "bestWhen": "You want someone else to run loading and you can budget for the meter.",
+ "watchOut": "Paid rates sit in a separate table and each connection has its own curve, so you need rows per connection to predict a bill, and deleted rows count as paid rows.",
+ "href": "#1-fivetran-best-for-hands-off-loading-from-many-sources"
+ },
+ "airbyte": {
+ "name": "Airbyte",
+ "summary": "A long connector list with a free self-hosted core or a paid cloud.",
+ "bestWhen": "You want lots of connectors and can live with minutes to an hour of delay.",
+ "watchOut": "It is not a continuous streaming tool, and the core uses the Elastic License 2.0, which stops you offering it to others as a managed service.",
+ "href": "#2-airbyte-best-for-a-long-connector-list-with-a-free-self-hosted-core"
+ },
+ "debezium": {
+ "name": "Debezium",
+ "summary": "Free and open source. Its docs say it records database changes as events, and describe delay in the millisecond range for MySQL and PostgreSQL (the project's claim, not a measurement).",
+ "bestWhen": "You already run Kafka or have engineers to spare, and need changes in seconds.",
+ "watchOut": "It only captures changes. You still need a place to send them and someone to keep it healthy.",
+ "href": "#5-debezium-best-for-engineers-who-need-every-database-change-as-an-event"
+ },
+ "matillion": {
+ "name": "Matillion",
+ "summary": "One tool for loading, transforming and scheduling inside a cloud warehouse.",
+ "bestWhen": "You want one vendor for load and transform on a warehouse.",
+ "watchOut": "No dollar price is shown on its pricing page, and we found no Apache Iceberg destination in its docs.",
+ "href": "#8-matillion-best-for-teams-that-load-and-transform-inside-a-cloud-warehouse"
+ },
+ "glue": {
+ "name": "AWS Glue",
+ "summary": "A managed Spark service on AWS with a built-in data catalog, billed by compute time.",
+ "bestWhen": "Your data and skills already live on AWS.",
+ "watchOut": "You write Spark code or compose jobs in Glue's visual editor, and it runs only on AWS.",
+ "href": "#9-aws-glue-best-for-teams-that-already-write-spark-jobs-on-aws"
+ },
+ "adf": {
+ "name": "Azure Data Factory",
+ "summary": "A managed Microsoft service for ETL, ELT and data integration.",
+ "bestWhen": "You live on Azure and run batch jobs.",
+ "watchOut": "Pipelines are batch only, and Microsoft says new users should start with Fabric Data Factory.",
+ "href": "#10-azure-data-factory-best-for-microsoft-centred-batch-pipelines"
+ },
+ "dbt": {
+ "name": "dbt",
+ "summary": "Reshapes data already in your warehouse using SQL, with a free plan and open-source core.",
+ "bestWhen": "Analysts will write the transformations in SQL.",
+ "watchOut": "It does not extract or load anything, so you need a loader first.",
+ "href": "#11-dbt-best-for-sql-first-teams-that-model-data-in-a-warehouse"
+ },
+ "airflow": {
+ "name": "Apache Airflow",
+ "summary": "A free, code-first scheduler that runs your pipeline steps in order.",
+ "bestWhen": "Your team writes Python and wants a free scheduler.",
+ "watchOut": "It does not copy data itself, it is built for batch rather than streaming, and you pay for servers and staff.",
+ "href": "#12-apache-airflow-best-for-teams-that-want-a-code-first-scheduler"
+ },
+ "hevo": {
+ "name": "Hevo",
+ "summary": "A managed, no-code loader into a warehouse, with a free plan up to 1M events a month.",
+ "bestWhen": "You are a small team that wants a warehouse fed without writing code.",
+ "watchOut": "Every insert, update or delete is an event, so busy tables use the quota fast. Streaming sits on higher plans.",
+ "href": "#3-hevo-best-for-small-teams-that-want-no-code-pipelines-into-a-warehouse"
+ },
+ "dagster": {
+ "name": "Dagster",
+ "summary": "A code-first orchestrator that shows your pipeline as the datasets it makes, with built-in lineage.",
+ "bestWhen": "You like to see your pipeline as tables and assets rather than tasks.",
+ "watchOut": "Each asset run or op is a credit, so many small runs add up. The cheaper plans allow only 1 or 3 users.",
+ "href": "#13-dagster-best-for-teams-that-think-in-tables-and-assets-rather-than-tasks"
+ },
+ "prefect": {
+ "name": "Prefect",
+ "summary": "An open-source engine that turns plain Python functions into pipelines, with a free Hobby plan.",
+ "bestWhen": "Your team already writes Python and wants to start with ordinary functions.",
+ "watchOut": "It is code only, and the free plan caps at 5 deployments and 7 days of history.",
+ "href": "#14-prefect-best-for-python-teams-that-want-to-start-with-ordinary-functions"
+ },
+ "olake": {
+ "name": "OLake Go",
+ "summary": "Open-source tool that captures Postgres, MySQL/MariaDB, MongoDB and SQL Server changes into Iceberg or Parquet (Oracle, Db2, S3 and Kafka load without database change capture).",
+ "bestWhen": "Your sources are databases and you want Iceberg or Parquet on storage you run.",
+ "watchOut": "Its output is Iceberg and Parquet, so it suits teams that want one of those on storage they run.",
+ "href": "#4-olake-go-best-for-copying-postgres-mysql-mongodb-or-sql-server-into-iceberg"
+ }
+ }
+}
diff --git a/src/data/quizzes/best-data-quality-tools.json b/src/data/quizzes/best-data-quality-tools.json
new file mode 100644
index 00000000..19d2a1b5
--- /dev/null
+++ b/src/data/quizzes/best-data-quality-tools.json
@@ -0,0 +1,61 @@
+{
+ "id": "best-data-quality-tools",
+ "mode": "recommend",
+ "title": "Which data quality tool fits you?",
+ "updated": "October 2026",
+ "guideHref": "#the-three-jobs-and-which-tool-fits",
+ "guideLabel": "See the job table",
+ "topN": 3,
+ "questions": [
+ {
+ "q": "What job do you want the tool to do?",
+ "hint": "The post says data quality tools do three different jobs, so start here.",
+ "options": [
+ {"label": "Run rules I write in a pipeline", "scores": {"gx": 2, "soda": 2, "dbt": 2, "deequ": 2}, "why": "you want rules you write and run in a pipeline"},
+ {"label": "Watch production tables for surprises", "scores": {"elementary": 3, "montecarlo": 3, "soda": 1}, "why": "you want something to watch tables for drift"},
+ {"label": "A free catalog with quality tests too", "scores": {"openmetadata": 3}, "why": "you want a catalog with tests built in"},
+ {"label": "Cleansing and governance across a big company", "scores": {"enterprise": 3}, "why": "you need profiling, cleansing and governance"}
+ ]
+ },
+ {
+ "q": "What does your data stack look like?",
+ "options": [
+ {"label": "We already transform data with dbt", "scores": {"dbt": 3, "elementary": 3}, "why": "you already use dbt"},
+ {"label": "Our data runs on Apache Spark", "scores": {"deequ": 3}, "why": "your data is on Spark"},
+ {"label": "No dbt and no Spark, we write Python", "scores": {"gx": 3, "soda": 3, "openmetadata": 1, "montecarlo": 1, "enterprise": 1}, "exclude": ["deequ", "dbt"], "why": "dbt tests need dbt and Deequ needs Spark"}
+ ]
+ },
+ {
+ "q": "What is the budget shape?",
+ "options": [
+ {"label": "$0 for now", "scores": {"gx": 2, "soda": 2, "dbt": 2, "elementary": 2, "deequ": 2, "openmetadata": 2}, "why": "it has a free plan or free software"},
+ {"label": "I can pay, but I want a price I can see", "scores": {"soda": 3, "dbt": 2}, "why": "its paid price is published"},
+ {"label": "A sales quote is fine", "scores": {"montecarlo": 2, "enterprise": 2, "elementary": 1}, "why": "its paid plans are quote based"}
+ ]
+ },
+ {
+ "q": "Who should run the tool?",
+ "options": [
+ {"label": "A vendor hosts it", "scores": {"soda": 1, "montecarlo": 3, "enterprise": 2, "elementary": 1}, "why": "it is sold as a hosted platform or cloud service"},
+ {"label": "We run it ourselves", "scores": {"openmetadata": 2, "deequ": 2, "gx": 2, "elementary": 2, "soda": 2, "dbt": 2}, "why": "you can run it yourself"}
+ ]
+ },
+ {
+ "q": "How big is your company?",
+ "options": [
+ {"label": "A small team that wants to start fast", "scores": {"dbt": 2, "gx": 2, "soda": 2, "elementary": 2}, "why": "our read is that small teams gain most from a few rules, then alerts"},
+ {"label": "Large or regulated", "scores": {"enterprise": 3, "montecarlo": 2, "openmetadata": 1}, "why": "our read is that large or regulated companies often begin on the catalog and governance side or need many tables watched"}
+ ]
+ }
+ ],
+ "results": {
+ "gx": {"name": "Great Expectations", "summary": "A free Python library for data tests (GX Core).", "bestWhen": "You want rules written in Python that you can review and version like code.", "watchOut": "You write every rule yourself in Python. GX Cloud is no longer publicly available, per GX's May 2026 post, and the same post says Fivetran will become the new steward of GX Core.", "href": "#1-great-expectations-rules-written-in-python"},
+ "soda": {"name": "Soda", "summary": "Checks written in YAML, with a free plan and a published Team price.", "bestWhen": "You want an openly priced paid plan or a free hosted tier.", "watchOut": "Soda Core uses the Elastic License 2.0, not Apache or MIT, and Soda's docs say Soda Core has no observability features. Those come from the hosted product.", "href": "#2-soda-data-contracts-and-a-free-hosted-tier"},
+ "dbt": {"name": "dbt tests", "summary": "Tests built into dbt, such as unique, not_null, accepted_values and relationships.", "bestWhen": "You already transform your data with dbt.", "watchOut": "It only tests what dbt manages, and the docs we read do not describe cleansing or learned alerts.", "href": "#3-dbt-tests-data-already-transformed-with-dbt"},
+ "elementary": {"name": "Elementary", "summary": "An open-source tool that turns dbt test results into a report and alerts.", "bestWhen": "You use dbt and want a report and alerts on your dbt test results.", "watchOut": "The docs say the open-source tool covers a single dbt project, and it depends on dbt. Cloud plans say \"Talk to us\" instead of a price.", "href": "#4-elementary-adding-alerts-to-a-dbt-project"},
+ "deequ": {"name": "Deequ", "summary": "A free AWS Labs library that runs unit tests for data on Spark.", "bestWhen": "Your data is very large and already on Apache Spark.", "watchOut": "You need a Spark setup and a matching Java version. It is a library from AWS Labs, not an AWS managed service.", "href": "#5-deequ-very-large-data-on-apache-spark"},
+ "openmetadata": {"name": "OpenMetadata", "summary": "A free open-source catalog with no-code quality tests and alerts.", "bestWhen": "You want one free tool for the catalog and the tests.", "watchOut": "You run and upgrade it yourself unless you use a managed offering, and it is more than a test library.", "href": "#6-openmetadata-quality-tests-inside-a-free-catalog"},
+ "montecarlo": {"name": "Monte Carlo", "summary": "A paid platform that monitors tables and traces issues.", "bestWhen": "Many tables across a company need watching and you can pay.", "watchOut": "No price is published, no free trial is mentioned and its pricing and platform pages show no open-source edition.", "href": "#7-monte-carlo-broad-production-monitoring-with-a-paid-platform"},
+ "enterprise": {"name": "Informatica or Ataccama", "summary": "Enterprise suites for profiling, cleansing and governance.", "bestWhen": "You need cleansing and governance across many teams in an enterprise suite.", "watchOut": "Neither shows a dollar price on its own pages, and both are proprietary enterprise suites.", "href": "#8-informatica-data-quality-and-observability-enterprise-cleansing"}
+ }
+}
diff --git a/src/data/quizzes/best-data-replication-tools.json b/src/data/quizzes/best-data-replication-tools.json
index bbafbb8d..580245ce 100644
--- a/src/data/quizzes/best-data-replication-tools.json
+++ b/src/data/quizzes/best-data-replication-tools.json
@@ -72,7 +72,8 @@
"debezium": 2,
"airbyte": 2,
"dms": 2,
- "datastream": 2
+ "datastream": 2,
+ "goldengate": 1
},
"why": "your source is MongoDB"
},
@@ -80,7 +81,6 @@
"label": "SQL Server",
"scores": {
"olake": 2,
- "debezium": 1,
"airbyte": 2,
"estuary": 2,
"dms": 2,
@@ -113,9 +113,6 @@
"dms": 2,
"airbyte": 2
},
- "exclude": [
- "olake"
- ],
"why": "you want a hosted service"
},
{
@@ -129,7 +126,7 @@
"exclude": [
"datastream"
],
- "why": "you are happy to self-host"
+ "why": "you want to run it yourself; Google's docs describe no self-hosted option for Datastream"
},
{
"label": "It should be part of the database",
@@ -153,9 +150,7 @@
{
"label": "BigQuery on Google Cloud",
"scores": {
- "datastream": 3,
- "estuary": 2,
- "airbyte": 2
+ "datastream": 3
},
"why": "you land data in BigQuery"
},
@@ -165,8 +160,7 @@
"estuary": 2,
"airbyte": 2,
"dms": 2,
- "goldengate": 2,
- "datastream": 1
+ "goldengate": 2
},
"why": "you want many possible targets"
},
@@ -211,7 +205,7 @@
"scores": {
"goldengate": 3
},
- "why": "you can pay for an enterprise licence"
+ "why": "it is an enterprise platform priced per OCPU on Oracle Cloud"
}
]
},
@@ -238,14 +232,12 @@
"estuary": 2,
"debezium": 2,
"goldengate": 2,
- "datastream": 1,
- "olake": 1,
- "native": 1
+ "olake": 1
},
"exclude": [
"airbyte"
],
- "why": "you need continuous streaming"
+ "why": "you want changes read from the log with low delay"
}
]
}
@@ -259,21 +251,21 @@
},
"goldengate": {
"name": "Oracle GoldenGate",
- "summary": "An enterprise platform for one-way, two-way and multi-directional copies, strong on Oracle.",
+ "summary": "An enterprise platform for one-way, two-way and multi-directional copies, with Oracle among its supported databases.",
"bestWhen": "you are an Oracle shop with the budget and may need two-way copies.",
"watchOut": "its FAQ lists Iceberg, but the write details are in separate GoldenGate DAA docs, and we do not quote an on-premises price."
},
"dms": {
"name": "AWS DMS",
"summary": "AWS's managed service for one-time moves or ongoing replication, billed on instance or DCU hours.",
- "bestWhen": "you are moving a database into AWS or doing simple replication inside AWS.",
- "watchOut": "Iceberg is not a DMS target, there is no latency promise, and its two-way mode has no conflict resolution."
+ "bestWhen": "you are moving a database into AWS, with a full load first and then change capture.",
+ "watchOut": "Iceberg is not on the DMS target list, AWS's CDC page says there is no latency SLA, and DMS two-way tasks do not include conflict detection or resolution."
},
"datastream": {
"name": "Google Cloud Datastream",
"summary": "A serverless CDC service on Google Cloud, billed per GiB processed.",
"bestWhen": "you are on Google Cloud and land data in BigQuery.",
- "watchOut": "there is no self-hosted option, and Iceberg streaming is append-only, so no updates and deletes."
+ "watchOut": "Google's docs describe no self-hosted option, and streaming to its Iceberg tables is supported only in append-only mode, where every row is an INSERT."
},
"debezium": {
"name": "Debezium",
@@ -289,8 +281,8 @@
},
"estuary": {
"name": "Estuary",
- "summary": "A managed platform for streaming and batch, billed at $0.50 per GB plus connector fees.",
- "bestWhen": "you want one managed platform that does streaming and batch.",
+ "summary": "A platform that says it does streaming and batch, priced at $0.50 per GB moved plus connector fees.",
+ "bestWhen": "you want one platform that, by Estuary's own description, does streaming and batch.",
"watchOut": "every source and destination is billed, and its Iceberg connector needs an AWS EMR Serverless Spark application."
},
"airbyte": {
diff --git a/src/data/quizzes/best-reverse-etl-tools.json b/src/data/quizzes/best-reverse-etl-tools.json
index b2759b92..471f78a0 100644
--- a/src/data/quizzes/best-reverse-etl-tools.json
+++ b/src/data/quizzes/best-reverse-etl-tools.json
@@ -20,11 +20,7 @@
"segment": 3,
"rudderstack": 3
},
- "exclude": [
- "multiwoven",
- "script"
- ],
- "why": "you want a vendor-run service, and Multiwoven and a script are yours to run"
+ "why": "you want a vendor-run service"
},
{
"label": "We run it ourselves",
@@ -33,7 +29,7 @@
"script": 2,
"polytomic": 1
},
- "why": "you are happy to deploy and upgrade it (Polytomic offers on-premises only on Enterprise)"
+ "why": "you are happy to deploy and upgrade it (Polytomic lists on-premises under Enterprise)"
},
{
"label": "Just a small script",
@@ -51,9 +47,7 @@
{
"label": "One or two tools",
"scores": {
- "script": 3,
- "multiwoven": 1,
- "rudderstack": 1
+ "script": 3
},
"why": "with one or two destinations a short script is often cheaper"
},
@@ -72,12 +66,9 @@
{
"label": "Many, owned by marketing or revenue teams",
"scores": {
- "hightouch": 3,
- "segment": 1,
- "polytomic": 1,
- "script": 0
+ "hightouch": 3
},
- "why": "a script gets heavy as destinations pile up, and Hightouch has no cap on destinations"
+ "why": "Hightouch's self-serve plans have no limit on destinations, and a script gets heavy as destinations pile up"
}
]
},
@@ -123,14 +114,14 @@
"script": 1,
"multiwoven": 1
},
- "why": "Hightouch's lower plans run hourly at most, which is fine here"
+ "why": "most syncs run on a batch schedule, and Hightouch's free and self-serve plans are hourly at most"
},
{
"label": "Every 15 minutes or faster",
"scores": {
"segment": 3
},
- "why": "Segment's shortest published gap is 15 minutes"
+ "why": "Segment publishes 15 minutes as its shortest gap"
}
]
},
@@ -140,12 +131,7 @@
"options": [
{
"label": "Yes, that is fine",
- "scores": {
- "fivetran": 1,
- "segment": 1,
- "rudderstack": 1
- },
- "why": "Segment, RudderStack and Fivetran Advanced all keep state in your warehouse, so they need write access"
+ "scores": {}
},
{
"label": "No, read-only access only",
@@ -169,7 +155,6 @@
"rudderstack": 3,
"hightouch": 3,
"multiwoven": 3,
- "script": 2,
"fivetran": 1
},
"why": "RudderStack Free, Hightouch's 2 free active syncs and Multiwoven's free licence are all published"
@@ -190,7 +175,7 @@
"segment": 2,
"polytomic": 1
},
- "why": "Hightouch self-serve and Segment show no dollar price on the pages we read"
+ "why": "Hightouch self-serve and Segment show no dollar price on the pages we read, and Polytomic's Standard and Enterprise say \"Let's chat\""
}
]
}
@@ -198,7 +183,7 @@
"results": {
"hightouch": {
"name": "Hightouch",
- "summary": "A vendor-run tool billed by active syncs, with no cap on destinations or seats.",
+ "summary": "A vendor-run tool billed by active syncs, with no limit on destinations or seats on its self-serve plans.",
"bestWhen": "marketing or revenue teams will own the syncs and you need many destinations.",
"watchOut": "free and Self-serve plans are limited to hourly syncs, and the self-serve dollar price is not published.",
"href": "#1-hightouch-best-for-marketing-led-teams-that-want-many-destinations"
@@ -226,8 +211,8 @@
},
"rudderstack": {
"name": "RudderStack",
- "summary": "Event collection with reverse ETL on every plan, and a free tier.",
- "bestWhen": "you also want event collection and a free start with a published price after.",
+ "summary": "Billed by events, with reverse ETL on every plan and a free tier.",
+ "bestWhen": "you want a free start with a published price after.",
"watchOut": "failed syncs cannot be replayed, a source cannot go to a warehouse destination, and it stores sync state in your warehouse.",
"href": "#5-rudderstack-best-for-a-free-start-with-reverse-etl-included"
},
@@ -242,8 +227,8 @@
"name": "A short script or iPaaS",
"summary": "Custom API code or a general automation tool instead of a dedicated product.",
"bestWhen": "you send data to only one or two tools and an engineer can own it.",
- "watchOut": "every API change, schema change and new destination means more work, and iPaaS tools are not designed to use a warehouse as the source.",
+ "watchOut": "per Fivetran's guide (Fivetran sells a reverse ETL product), every API change, schema change and new destination means more work, and iPaaS tools are not designed to use a warehouse as the source.",
"href": "#do-you-need-a-reverse-etl-tool-at-all"
}
}
-}
\ No newline at end of file
+}
diff --git a/src/data/quizzes/data-migration-strategy.json b/src/data/quizzes/data-migration-strategy.json
index fb20f8dc..568dad68 100644
--- a/src/data/quizzes/data-migration-strategy.json
+++ b/src/data/quizzes/data-migration-strategy.json
@@ -9,7 +9,7 @@
"q": "Your business can switch everything off for the whole copy. Which cutover approach fits best?",
"options": ["Phased, one group at a time", "Big bang, copy it all in one go", "Trickle, a little at a time", "Replication-based, with changes kept in sync"],
"correct": 1,
- "explain": "Big bang copies everything in one operation, then switches. MongoDB says it is faster but needs downtime, so it only suits systems that can stop."
+ "explain": "Big bang copies everything in one operation, then switches. MongoDB says it is faster but needs downtime, so it fits a system that can be offline for the whole copy and whose data is small enough."
},
{
"q": "What does Google's architecture guide say about moving a database with zero downtime for clients?",
@@ -50,7 +50,7 @@
"The old system is always deleted at switchover",
"Backups stop working after a switch",
"New writes land on the new system, so the old one goes stale",
- "Old systems cannot be turned back on"
+ "Clients keep writing to both systems after the switch"
],
"correct": 2,
"explain": "After switchover the source stops receiving changes. Google's fallback phase copies target changes back to it, so a reverse move stays possible."
diff --git a/src/data/quizzes/data-migration-tools.json b/src/data/quizzes/data-migration-tools.json
index cc591675..fdfa4493 100644
--- a/src/data/quizzes/data-migration-tools.json
+++ b/src/data/quizzes/data-migration-tools.json
@@ -29,7 +29,7 @@
"scores": {
"datasync": 3,
"fivetran": 1,
- "airbyte": 1
+ "olake": 1
}
},
{
@@ -57,7 +57,7 @@
"azuredms",
"googledms"
],
- "why": "Azure DMS targets Azure SQL and Google DMS targets Google Cloud databases"
+ "why": "the Azure DMS overview names Azure SQL targets and the Google DMS overview names Google Cloud databases"
},
{
"label": "A cloud database on Azure or Google Cloud",
@@ -65,20 +65,18 @@
"azuredms": 2,
"googledms": 2
},
- "why": "Azure DMS targets Azure SQL and Google DMS targets Google Cloud databases"
+ "why": "the Azure DMS overview names Azure SQL targets and the Google DMS overview names Google Cloud databases"
},
{
"label": "BigQuery",
"scores": {
- "datastream": 3,
- "fivetran": 1,
- "airbyte": 1
+ "datastream": 3
},
"exclude": [
"azuredms",
"googledms"
],
- "why": "Azure DMS and Google DMS do not list BigQuery as a target"
+ "why": "the Google DMS overview lists Google Cloud databases and no warehouse, and the Azure DMS overview names Azure SQL targets"
},
{
"label": "An Iceberg or Parquet lake I run",
@@ -92,7 +90,7 @@
"azuredms",
"googledms"
],
- "why": "Azure DMS and Google DMS land in their own cloud's databases only"
+ "why": "the Azure DMS overview names Azure SQL targets and the Google DMS overview names Google Cloud databases"
},
{
"label": "A warehouse or lake somewhere else",
@@ -104,7 +102,7 @@
"azuredms",
"googledms"
],
- "why": "Azure DMS and Google DMS land in their own cloud's databases only"
+ "why": "the Azure DMS overview names Azure SQL targets and the Google DMS overview names Google Cloud databases"
}
]
},
@@ -145,8 +143,7 @@
"label": "Scheduled copies",
"scores": {
"datasync": 2,
- "airbyte": 1,
- "fivetran": 1
+ "airbyte": 1
},
"why": "DataSync does scheduled copies"
}
@@ -159,12 +156,12 @@
"label": "Free or close to it",
"scores": {
"googledms": 2,
- "azuredms": 2,
"airbyte": 2,
"goldengate": 1,
- "fivetran": 1
+ "fivetran": 1,
+ "olake": 2
},
- "why": "some of these have a free route, such as Core, Standard or a free plan"
+ "why": "some have a free route: Airbyte Core, Fivetran's free plan, GoldenGate Free for Oracle databases up to 20 GB, Google's no-charge native moves and OLake Go's open-source licence (you still pay for servers)"
},
{
"label": "A price I can work out from the vendor page",
@@ -177,11 +174,11 @@
"why": "DataSync is a flat per-GB fee and AWS DMS lists an hourly rate"
},
{
- "label": "A negotiated enterprise licence is fine",
+ "label": "A paid enterprise licence is fine",
"scores": {
"goldengate": 3
},
- "why": "GoldenGate licences are usually negotiated"
+ "why": "GoldenGate is sold by licence, listed per processor or Named User Plus"
}
]
},
@@ -207,7 +204,7 @@
"airbyte": 3,
"olake": 3
},
- "why": "Airbyte Core and OLake Go are the self-run options in the post, and Datastream is documented as Google-run only"
+ "why": "Airbyte Core is self-managed and you run OLake Go on your own infrastructure"
}
]
}
@@ -217,25 +214,26 @@
"name": "AWS Database Migration Service",
"summary": "One-time moves or ongoing replication between databases, billed by instance hours or Serverless DCU hours.",
"bestWhen": "you are moving a database into AWS.",
- "watchOut": "AWS offers no SLA for change latency, and Apache Iceberg is not a target."
+ "watchOut": "AWS says there is no SLA for CDC latency, and its target list names S3, Kinesis, Kafka and others but not Iceberg (the guide's AWS DMS section links the page).",
+ "href": "#1-aws-database-migration-service-best-for-moving-databases-into-aws"
},
"azuredms": {
"name": "Azure Database Migration Service",
- "summary": "Moves SQL Server databases to Azure SQL targets, with a Standard tier that the classic pricing page lists as free for offline moves.",
+ "summary": "Moves SQL Server databases to Azure SQL targets; the only pricing page we found is for the classic service.",
"bestWhen": "you are moving SQL Server into Azure SQL.",
- "watchOut": "the classic version for SQL Server lost support on 15 March 2026, the portal version has no assessment or login migration, and it does not suit other sources or targets outside Azure SQL."
+ "watchOut": "Microsoft's notice says the classic version for SQL Server scenarios was not supported after 15 March 2026, the overview table does not list assessment or login migration for the portal version, and it names Azure SQL targets only, so confirm other sources and targets with Microsoft."
},
"googledms": {
"name": "Google Database Migration Service",
- "summary": "One-time or continuous moves into Google Cloud databases, with no additional charge for same-engine native moves to Cloud SQL or AlloyDB.",
+ "summary": "One-time or continuous moves into Google Cloud databases, with no additional charge for native same-engine moves to Cloud SQL or AlloyDB for PostgreSQL.",
"bestWhen": "the target is Cloud SQL or AlloyDB.",
"watchOut": "destinations are Google Cloud databases only, and no lake or warehouse is listed."
},
"goldengate": {
"name": "Oracle GoldenGate",
- "summary": "Real-time copies between many systems, including one-way and two-way setups.",
+ "summary": "Copies between many systems, including one-way and two-way setups; Oracle's FAQ says it delivers changes in real time.",
"bestWhen": "your estate is Oracle-heavy, or a copy has to run both ways.",
- "watchOut": "licences are priced per processor or Named User Plus and usually negotiated, and Iceberg writing sits in a separate product."
+ "watchOut": "Oracle's price list charges per processor or Named User Plus, and Iceberg writing sits in a separate product."
},
"datasync": {
"name": "AWS DataSync",
@@ -246,27 +244,27 @@
"fivetran": {
"name": "Fivetran",
"summary": "Managed pipelines from databases, apps and files into warehouses and lakes, billed on Monthly Active Rows.",
- "bestWhen": "you want a warehouse fed with little upkeep.",
- "watchOut": "it is built for ongoing syncs, not a one-time cutover, and its Iceberg service does not support position deletes."
+ "bestWhen": "you want a vendor to run the pipelines into a warehouse.",
+ "watchOut": "the decision table rules it out for a one-time database cutover, and its Iceberg service does not support position deletes."
},
"airbyte": {
"name": "Airbyte",
"summary": "Syncs from many sources into warehouses, databases and lakes, with a free self-run Core edition.",
- "bestWhen": "you want many connectors and are fine with minutes between syncs.",
- "watchOut": "CDC runs on a schedule, and Cloud Standard syncs at most hourly, so skip it for a live cutover."
+ "bestWhen": "you want many connectors and are fine with scheduled syncs.",
+ "watchOut": "CDC runs on a schedule and Cloud Standard syncs at most hourly, so the decision table rules it out for a one-time cutover."
},
"datastream": {
"name": "Google Cloud Datastream",
"summary": "A serverless service that backfills, then captures changes into BigQuery, Cloud Storage or Iceberg tables in BigQuery.",
"bestWhen": "you are on Google Cloud and landing data in BigQuery.",
- "watchOut": "a processed byte can be 2 to 5 times the real data size, its Iceberg mode is append-only, and there is no self-hosted option."
+ "watchOut": "a processed byte can be 2 to 5 times the real data size, its Iceberg mode is append-only, and Google documents no self-hosted option."
},
"olake": {
- "name": "OLake Go (ours)",
+ "name": "OLake Go",
"summary": "Open-source copies of databases, S3 files and Kafka topics into Apache Iceberg or Parquet on storage you run, with change capture for PostgreSQL, MySQL/MariaDB, MongoDB and SQL Server.",
"bestWhen": "you want an Iceberg or Parquet lake and want to run the tool on your own infrastructure.",
- "watchOut": "it is built for Iceberg and Parquet output, and Oracle and Db2 load by full refresh or incremental copy.",
- "href": "/docs/connectors/overview/"
+ "watchOut": "output is Apache Iceberg or Parquet, and Oracle, Db2 and S3 sources load by full refresh or incremental copy.",
+ "href": "#7-olake-go-best-for-copying-databases-into-iceberg-or-parquet-on-storage-you-run"
}
}
}
diff --git a/src/data/quizzes/database-to-iceberg-without-kafka.json b/src/data/quizzes/database-to-iceberg-without-kafka.json
index 1b0a6882..f6b3d267 100644
--- a/src/data/quizzes/database-to-iceberg-without-kafka.json
+++ b/src/data/quizzes/database-to-iceberg-without-kafka.json
@@ -14,8 +14,7 @@
{
"label": "Yes, and several systems read it",
"scores": {
- "kafka": 3,
- "flink": 1
+ "kafka": 3
},
"why": "other systems already read your Kafka topics"
},
@@ -83,20 +82,18 @@
"label": "Yes, joins or filters in flight",
"scores": {
"flink": 3,
- "risingwave": 3,
- "kafka": 1
+ "risingwave": 3
},
"exclude": [
"olake"
],
- "why": "you transform in flight, which Flink CDC and RisingWave are built for"
+ "why": "Flink CDC's docs describe reshaping data in the same job and RisingWave offers SQL for shaping data; OLake Go is an extract-and-load tool with no transformation step"
},
{
"label": "Only small renames or tweaks",
"scores": {
"flink": 2,
- "risingwave": 1,
- "dlt": 1
+ "risingwave": 1
},
"why": "you need light changes only"
},
@@ -123,12 +120,11 @@
"dbzserver": 2,
"kafka": 2,
"flink": 1,
- "risingwave": 2,
+ "risingwave": 1,
"estuary": 2,
- "fivetran": 1,
- "dlt": 1
+ "fivetran": 1
},
- "why": "your sources have CDC in most tools"
+ "why": "your sources have log-based CDC in most of these tools; check MongoDB and SQL Server by name"
},
{
"label": "Oracle, with change capture",
@@ -141,7 +137,7 @@
"exclude": [
"olake"
],
- "why": "you need Oracle CDC, which Debezium Server, Estuary and Flink CDC cover"
+ "why": "you need Oracle change capture, which Debezium Server, Estuary and Flink CDC list; OLake Go's change capture covers PostgreSQL, MySQL/MariaDB, MongoDB and SQL Server"
},
{
"label": "A mix of databases and SaaS apps",
@@ -151,7 +147,7 @@
"exclude": [
"olake"
],
- "why": "you need SaaS sources, which Fivetran covers"
+ "why": "you need SaaS sources, which Fivetran sells connectors for; OLake Go reads databases, S3 and Kafka"
}
]
},
@@ -209,7 +205,7 @@
"name": "Flink CDC",
"summary": "Reads database changes inside Apache Flink and writes Iceberg, with room to reshape data in the same job.",
"bestWhen": "Flink already runs and you join or clean data before it lands.",
- "watchOut": "you need a Flink cluster, source tables need primary keys and the Iceberg sink does not offer exactly-once."
+ "watchOut": "you need a Flink cluster, source tables need primary keys and the pipeline page says exactly-once is not supported."
},
"risingwave": {
"name": "RisingWave",
@@ -225,21 +221,21 @@
},
"fivetran": {
"name": "Fivetran Managed Data Lake Service",
- "summary": "A hosted service that lands data as Iceberg tables, next to Fivetran's large connector list.",
- "bestWhen": "Fivetran already moves your data, or you also need SaaS sources, and you want to run nothing.",
- "watchOut": "it is hosted only, has no position deletes, and compaction is not listed."
+ "summary": "A hosted service that lands data as Iceberg tables, next to Fivetran's database and SaaS connectors.",
+ "bestWhen": "Fivetran already moves your data, or you also need SaaS sources, and you want no brokers, cluster or sink to run.",
+ "watchOut": "the destination page lists the SaaS deployment only, position deletes are not supported, and compaction is not listed."
},
"dlt": {
"name": "dlt",
"summary": "An open-source Python library that writes Iceberg through PyIceberg.",
"bestWhen": "your team writes Python and the source is Postgres or a batch load.",
- "watchOut": "log-based CDC is Postgres only, Parquet only, and you write and watch the code yourself."
+ "watchOut": "log-based CDC in the open-source library is PostgreSQL only, it writes Parquet only, and you write and watch the code yourself."
},
"olake": {
"name": "OLake Go",
"summary": "Open-source software (Apache 2.0) that copies database changes into Apache Iceberg or Parquet with no Kafka or Debezium to run.",
- "bestWhen": "your sources are Postgres, MySQL, MongoDB or SQL Server and you want a small setup.",
- "watchOut": "jobs run on a schedule you set, and change capture covers PostgreSQL, MySQL/MariaDB, MongoDB and SQL Server (Oracle, Db2 and S3 load by full refresh or incremental)."
+ "bestWhen": "your sources are Postgres, MySQL, MongoDB or SQL Server and you want one process to run.",
+ "watchOut": "you choose the job schedule (every minute, hourly, daily, weekly or cron), so freshness follows it; change capture covers PostgreSQL, MySQL/MariaDB, MongoDB and SQL Server, and Oracle, Db2 and S3 sources load by full refresh or incremental sync."
}
}
-}
\ No newline at end of file
+}
diff --git a/src/data/quizzes/etl-pipelines.json b/src/data/quizzes/etl-pipelines.json
index 9de4befe..43916eb6 100644
--- a/src/data/quizzes/etl-pipelines.json
+++ b/src/data/quizzes/etl-pipelines.json
@@ -17,7 +17,7 @@
"explain": "That's the whole idea. A pipeline extracts the data from each source, transforms it, and loads it into one destination, so nobody has to babysit it each day."
},
{
- "q": "What is the one real difference between ETL and ELT?",
+ "q": "Per Microsoft, what is the only difference between ETL and ELT?",
"options": [
"ETL is for small data and ELT is for big data",
"ETL needs a person to run it and ELT runs itself",
@@ -28,7 +28,7 @@
"explain": "Microsoft puts it simply: the only difference is where the transformation happens. In ELT it runs inside the destination, using that system's own power."
},
{
- "q": "A pipeline copies only rows changed since its last run. Someone then deletes an order in the source. What happens?",
+ "q": "A pipeline copies only rows whose last-updated time is newer than its last run (a watermark). Someone then deletes an order in the source. What happens?",
"options": [
"The pipeline deletes it from the destination too",
"The pipeline stops and sends an alert",
@@ -36,7 +36,7 @@
"The pipeline copies the order again"
],
"correct": 2,
- "explain": "A deleted row leaves nothing behind for the query to find. If deletes matter to you, the pipeline needs a different method for spotting changes."
+ "explain": "A deleted row leaves nothing behind for the query to find. Our read: if deletes matter to you, the pipeline needs a different method for spotting changes, such as SQL Server Change Tracking or reading the database's change log."
},
{
"q": "A run fails halfway and the scheduler retries it. What do you want the retry to do?",
diff --git a/src/data/quizzes/mysql-to-iceberg-tools.json b/src/data/quizzes/mysql-to-iceberg-tools.json
index f10b8f5f..fce965d8 100644
--- a/src/data/quizzes/mysql-to-iceberg-tools.json
+++ b/src/data/quizzes/mysql-to-iceberg-tools.json
@@ -13,19 +13,35 @@
"options": [
{
"label": "A vendor runs it for me",
- "scores": { "fivetran": 3, "estuary": 3, "airbyte": 2 },
+ "scores": {
+ "fivetran": 3,
+ "estuary": 3,
+ "airbyte": 2
+ },
"why": "you want a vendor to run it"
},
{
"label": "My team runs it, with a UI or command line",
- "scores": { "olake": 3, "airbyte": 2, "supermetal": 2 },
- "exclude": ["fivetran"],
+ "scores": {
+ "olake": 3,
+ "airbyte": 2,
+ "supermetal": 2
+ },
+ "exclude": [
+ "fivetran"
+ ],
"why": "you want to self-host, and Fivetran's lake destination runs as SaaS only"
},
{
"label": "My team runs it, as Java services or code",
- "scores": { "debezium": 3, "flink": 2, "kafka": 2 },
- "exclude": ["fivetran"],
+ "scores": {
+ "debezium": 3,
+ "flink": 2,
+ "kafka": 2
+ },
+ "exclude": [
+ "fivetran"
+ ],
"why": "you want to self-host, and Fivetran's lake destination runs as SaaS only"
}
]
@@ -35,17 +51,29 @@
"options": [
{
"label": "Kafka, and several teams read the changes",
- "scores": { "kafka": 3, "debezium": 1 },
+ "scores": {
+ "kafka": 3,
+ "debezium": 1
+ },
"why": "Kafka is already your backbone"
},
{
"label": "Apache Flink in production",
- "scores": { "flink": 3 },
+ "scores": {
+ "flink": 3
+ },
"why": "Flink already runs in production"
},
{
"label": "Neither, I want few moving parts",
- "scores": { "olake": 2, "debezium": 2, "supermetal": 2, "fivetran": 1, "estuary": 1, "airbyte": 1 },
+ "scores": {
+ "olake": 2,
+ "debezium": 2,
+ "supermetal": 2,
+ "fivetran": 1,
+ "estuary": 1,
+ "airbyte": 1
+ },
"why": "you want few moving parts"
}
]
@@ -55,18 +83,34 @@
"options": [
{
"label": "Amazon S3",
- "scores": { "estuary": 2, "airbyte": 2, "olake": 1, "supermetal": 1, "fivetran": 1 },
+ "scores": {
+ "estuary": 2,
+ "airbyte": 2,
+ "olake": 1,
+ "supermetal": 1,
+ "fivetran": 1
+ },
"why": "your tables live on S3"
},
{
"label": "Google Cloud or Azure storage",
- "scores": { "olake": 2, "fivetran": 2, "supermetal": 1 },
- "exclude": ["estuary"],
- "why": "Estuary's Iceberg connector supports only Amazon S3 today"
+ "scores": {
+ "olake": 2,
+ "fivetran": 2,
+ "supermetal": 1
+ },
+ "exclude": [
+ "estuary"
+ ],
+ "why": "Estuary's Iceberg connector docs list REST catalogs on Amazon S3 only, with GCP and Azure planned"
},
{
"label": "Not decided yet",
- "scores": { "olake": 1, "fivetran": 1, "supermetal": 1 },
+ "scores": {
+ "olake": 1,
+ "fivetran": 1,
+ "supermetal": 1
+ },
"why": "you want a wide choice of storage"
}
]
@@ -76,18 +120,36 @@
"options": [
{
"label": "It must have an open-source licence",
- "scores": { "debezium": 2, "olake": 2, "flink": 2, "kafka": 1 },
- "exclude": ["airbyte", "estuary"],
- "why": "Airbyte Core uses the Elastic License 2.0 and Estuary uses the Business Source License"
+ "scores": {
+ "debezium": 2,
+ "olake": 2,
+ "flink": 2,
+ "kafka": 1
+ },
+ "exclude": [
+ "airbyte"
+ ],
+ "why": "Airbyte Core uses the Elastic License 2.0, which is not an open-source licence"
},
{
"label": "No licence fee, I will pay for servers",
- "scores": { "debezium": 2, "olake": 2, "flink": 2, "kafka": 2, "airbyte": 1 },
+ "scores": {
+ "debezium": 2,
+ "olake": 2,
+ "flink": 2,
+ "kafka": 2,
+ "airbyte": 1
+ },
"why": "you want to avoid a licence fee"
},
{
"label": "I can pay a vendor fee",
- "scores": { "fivetran": 2, "estuary": 2, "supermetal": 2, "airbyte": 2 },
+ "scores": {
+ "fivetran": 2,
+ "estuary": 2,
+ "supermetal": 2,
+ "airbyte": 2
+ },
"why": "you can budget a paid plan"
}
]
@@ -97,14 +159,24 @@
"options": [
{
"label": "Just databases like MySQL",
- "scores": { "olake": 1, "debezium": 1, "estuary": 1, "supermetal": 1, "flink": 1 },
+ "scores": {
+ "olake": 1,
+ "debezium": 1,
+ "estuary": 1,
+ "supermetal": 1,
+ "flink": 1
+ },
"why": "your sources are mostly databases"
},
{
"label": "Many other sources, including SaaS apps",
- "scores": { "airbyte": 3, "fivetran": 1 },
- "exclude": ["olake"],
- "why": "you need SaaS app sources, which Airbyte and Fivetran cover"
+ "scores": {
+ "airbyte": 3
+ },
+ "exclude": [
+ "olake"
+ ],
+ "why": "you need SaaS app sources, which Airbyte covers; OLake Go reads databases, S3 and Kafka"
}
]
}
@@ -119,7 +191,7 @@
"olake": {
"name": "OLake Go",
"summary": "Free software that copies MySQL into Iceberg tables with a UI or command line, on your own infrastructure.",
- "bestWhen": "you want a ready-made job, many catalogs and no licence fee.",
+ "bestWhen": "you want a UI, a documented catalog list and no licence fee.",
"watchOut": "it runs on your own infrastructure, writes Iceberg and Parquet, and OLake Fusion handles compaction.",
"href": "/docs/connectors/mysql/"
},
@@ -143,9 +215,9 @@
},
"fivetran": {
"name": "Fivetran Managed Data Lake Service",
- "summary": "A vendor-run service that keeps your data in your own bucket and handles table upkeep.",
+ "summary": "A vendor-run service that, per its docs, keeps your data in your own bucket and runs snapshot cleanup every day.",
"bestWhen": "you want someone else to run it.",
- "watchOut": "no support for Iceberg position deletes, SaaS only, and priced on monthly active rows."
+ "watchOut": "we found no page that pairs its MySQL connector with the lake service, so ask Fivetran to confirm; position deletes are not supported, it runs as SaaS only, and it is priced on monthly active rows."
},
"estuary": {
"name": "Estuary",
@@ -155,9 +227,9 @@
},
"supermetal": {
"name": "Supermetal",
- "summary": "A single self-hosted agent that writes Iceberg V3 or V2 and is priced on compute.",
+ "summary": "A self-hosted agent (Supermetal calls it a single binary) that writes Iceberg V3 or V2 and is priced on compute.",
"bestWhen": "you want a lean agent and can budget the fee.",
- "watchOut": "no data type promotion yet, compaction is on you, and it is paid after the trial and the licence is not stated."
+ "watchOut": "data type promotion is not supported, compaction is on you, it is paid after the trial and the pages we read state no licence."
}
}
}
diff --git a/src/data/quizzes/oltp-vs-olap.json b/src/data/quizzes/oltp-vs-olap.json
new file mode 100644
index 00000000..c9824350
--- /dev/null
+++ b/src/data/quizzes/oltp-vs-olap.json
@@ -0,0 +1,65 @@
+{
+ "id": "oltp-vs-olap",
+ "mode": "knowledge",
+ "title": "Test your OLTP vs OLAP basics",
+ "updated": "October 2026",
+ "guideHref": "#when-to-use-oltp-olap-both-or-neither",
+ "guideLabel": "Re-read when to use which",
+ "questions": [
+ {
+ "q": "A shop's till and website record each sale the moment it happens. Which job is that?",
+ "options": [
+ "OLAP, because it deals with sales data",
+ "OLTP, because it records one transaction at a time",
+ "Neither, because only a warehouse can store sales",
+ "OLAP, because it needs to be fast"
+ ],
+ "correct": 1,
+ "explain": "That's OLTP, the database behind your app. It records orders, payments and sign-ups one at a time, fast and correctly."
+ },
+ {
+ "q": "Your finance team wants a report that adds up millions of past orders. Why is running it on the live shop database a bad idea?",
+ "options": [
+ "It would delete the old orders",
+ "The report would be too inaccurate to trust",
+ "It can be slow and can block other transactions, like customers paying",
+ "Live databases refuse to answer any report at all"
+ ],
+ "correct": 2,
+ "explain": "Microsoft says reports that add up millions of transactions are very resource-intensive for an OLTP system. They can be slow and can block other transactions, though it notes exceptions such as a well-planned schema."
+ },
+ {
+ "q": "The dashboard is fed by an overnight copy, so it is a day behind. Which question should still go to the live OLTP side?",
+ "options": [
+ "Is this item in stock right now?",
+ "Which region sold the most this year?",
+ "How did revenue trend over the last 12 months?",
+ "Which products sold best last quarter?"
+ ],
+ "correct": 0,
+ "explain": "A day-old copy is fine for monthly reports. For something like \"is this in stock right now?\", it's wrong, so that question goes to the OLTP side."
+ },
+ {
+ "q": "Why can a column store add up one field across millions of records so quickly?",
+ "options": [
+ "It keeps every full record together, so it reads whole rows at once",
+ "It reads the whole table, then discards the fields it does not need",
+ "It copies every row to a second system before it answers",
+ "It keeps each field together, so it only reads the columns it needs"
+ ],
+ "correct": 3,
+ "explain": "A column store keeps each field together, so adding up one field is quick. A row store keeps each record together, so finding one order is quick."
+ },
+ {
+ "q": "So which should a growing company pick, OLTP or OLAP?",
+ "options": [
+ "OLAP alone, since it can answer every question",
+ "Usually both, with data copied from the first into the second",
+ "OLTP alone, since big reports are safe to run on the live app",
+ "Whichever has the bigger brand name"
+ ],
+ "correct": 1,
+ "explain": "AWS says most companies use both together. The better question is usually how to connect them, because they do different jobs."
+ }
+ ]
+}
diff --git a/src/data/quizzes/postgres-vs-mysql.json b/src/data/quizzes/postgres-vs-mysql.json
new file mode 100644
index 00000000..0dddf312
--- /dev/null
+++ b/src/data/quizzes/postgres-vs-mysql.json
@@ -0,0 +1,158 @@
+{
+ "id": "postgres-vs-mysql",
+ "mode": "recommend",
+ "title": "Which database fits you?",
+ "updated": "October 2026",
+ "guideHref": "#where-each-fits",
+ "guideLabel": "See the full decision table",
+ "topN": 2,
+ "questions": [
+ {
+ "q": "Where are you starting from?",
+ "hint": "Moving a working system has real costs, so your current setup counts.",
+ "options": [
+ {
+ "label": "A new project with no database yet",
+ "scores": {
+ "postgresql": 2,
+ "mysql": 1
+ },
+ "why": "you are starting fresh"
+ },
+ {
+ "label": "We already run MySQL well",
+ "scores": {
+ "mysql": 3
+ },
+ "why": "switching costs more than it looks"
+ },
+ {
+ "label": "A vendor product we need is built for MySQL",
+ "scores": {
+ "mysql": 3
+ },
+ "why": "your tools target MySQL"
+ }
+ ]
+ },
+ {
+ "q": "How often will you change how your data is laid out?",
+ "options": [
+ {
+ "label": "Often. We tweak our tables almost every week",
+ "scores": {
+ "postgresql": 3
+ },
+ "why": "schema changes can sit inside a transaction and roll back"
+ },
+ {
+ "label": "Rarely. We set it up once and leave it",
+ "scores": {
+ "either": 2,
+ "postgresql": 1,
+ "mysql": 1
+ },
+ "why": "you can plan around MySQL's schema change rules"
+ }
+ ],
+ "hint": "Think adding a column or a new table to a live app."
+ },
+ {
+ "q": "What do your reports need to do?",
+ "options": [
+ {
+ "label": "Save the answer to a slow report so it opens fast (a materialized view)",
+ "scores": {
+ "postgresql": 3
+ },
+ "exclude": [
+ "mysql"
+ ],
+ "why": "the MySQL 8.4 views FAQ answers No on materialized views"
+ },
+ {
+ "label": "Join two lists and keep unmatched rows from both (a full outer join)",
+ "scores": {
+ "postgresql": 3
+ },
+ "why": "PostgreSQL supports it, and MySQL's join syntax lists no full outer join"
+ },
+ {
+ "label": "Nothing fancy, just everyday lookups and summaries",
+ "scores": {
+ "either": 2,
+ "mysql": 1,
+ "postgresql": 1
+ },
+ "why": "both cover everyday SQL"
+ }
+ ]
+ },
+ {
+ "q": "When something breaks, who do you want to call?",
+ "options": [
+ {
+ "label": "A big vendor that sells a commercial edition",
+ "scores": {
+ "mysql": 3
+ },
+ "why": "Oracle sells commercial MySQL editions"
+ },
+ {
+ "label": "My hosting company or my own team",
+ "scores": {
+ "either": 2,
+ "postgresql": 1,
+ "mysql": 1
+ },
+ "why": "pick what your hosting and teammates already know"
+ }
+ ]
+ },
+ {
+ "q": "How much flexible, free-form data (JSON) will you keep?",
+ "options": [
+ {
+ "label": "A lot, and I need to search inside it",
+ "scores": {
+ "either": 3,
+ "postgresql": 1
+ },
+ "why": "jsonb is indexable directly, and MySQL indexes through generated columns, so test both"
+ },
+ {
+ "label": "Little or none",
+ "scores": {
+ "either": 1,
+ "postgresql": 1,
+ "mysql": 1
+ },
+ "why": "JSON will not decide it"
+ }
+ ]
+ }
+ ],
+ "results": {
+ "postgresql": {
+ "name": "PostgreSQL",
+ "summary": "A free, community-run database with transactional schema changes, full outer join, materialized views and six index types.",
+ "bestWhen": "You are starting fresh, ship many migrations, or need materialized views or its six documented index types.",
+ "watchOut": "It needs VACUUM housekeeping, though autovacuum is on by default. If your team already runs MySQL well, switching costs real time.",
+ "href": "#where-each-fits"
+ },
+ "mysql": {
+ "name": "MySQL",
+ "summary": "A mature database developed by Oracle. In 8.4 the default engine is InnoDB, which follows ACID.",
+ "bestWhen": "Your team, tools and hosting already run on it, or you want a commercial edition from Oracle.",
+ "watchOut": "Its 8.4 join syntax lists no full outer join, its views FAQ answers No on materialized views, and a schema change ends any open transaction. MySQL 8.0 is in Sustaining Support.",
+ "href": "#schema-changes-the-difference-that-bites-in-production"
+ },
+ "either": {
+ "name": "Either one works",
+ "summary": "Both are free, mature and can run most web apps. The post found no neutral, linked benchmark that crowns a winner.",
+ "bestWhen": "Your needs are plain and no feature above is a must. Test your own slowest queries on both.",
+ "watchOut": "Do not pick on speed claims. Check your version too: MySQL 8.0 is in Sustaining Support and PostgreSQL 14 gets its final release on November 12, 2026.",
+ "href": "#how-to-test-both-on-your-own-data"
+ }
+ }
+}
\ No newline at end of file
diff --git a/src/data/quizzes/sql-server-cdc-tools.json b/src/data/quizzes/sql-server-cdc-tools.json
index 8765d708..1267fbc7 100644
--- a/src/data/quizzes/sql-server-cdc-tools.json
+++ b/src/data/quizzes/sql-server-cdc-tools.json
@@ -33,7 +33,7 @@
{
"label": "Microsoft Fabric runs it for me",
"scores": {
- "fabric": 3
+ "fabric": 4
},
"why": "you want Microsoft's platform to run it"
}
@@ -54,30 +54,36 @@
"scores": {
"fivetran": 3,
"airbyte": 2,
- "estuary": 2,
- "striim": 2
+ "estuary": 2
},
- "why": "these tools document warehouse loads"
+ "why": "warehouses are among the targets these tools list"
},
{
"label": "Iceberg tables on Amazon S3",
"scores": {
"fivetran": 2,
"airbyte": 2,
- "olake": 3,
- "estuary": 1
+ "olake": 3
},
"exclude": [
"fabric"
],
- "why": "you want Iceberg on S3"
+ "why": "your target is Iceberg on S3 (Fabric mirroring writes to OneLake)"
},
{
"label": "Microsoft Fabric",
"scores": {
- "fabric": 3
+ "fabric": 5
},
"why": "Fabric is where analysts work"
+ },
+ {
+ "label": "Iceberg on Google Cloud Storage or Amazon S3 Tables",
+ "scores": {
+ "striim": 3,
+ "fivetran": 2
+ },
+ "why": "you want Iceberg on Google Cloud Storage or S3 Tables, which Striim's Iceberg writers cover"
}
]
},
@@ -88,9 +94,9 @@
"label": "Many have no primary key",
"scores": {
"estuary": 3,
- "fivetran": 2
+ "fivetran": 1
},
- "why": "keyless tables need a tool that handles them"
+ "why": "Estuary's CDC connector handles tables without a primary key, though you set a key by hand"
},
{
"label": "CDC is not allowed on the server",
@@ -101,7 +107,7 @@
"exclude": [
"debezium"
],
- "why": "Teleport Sync does not use CDC, and Estuary's batch connector polls tables, per its docs"
+ "why": "CDC is off the table, so you need a method that skips it: Fivetran Teleport Sync, or Estuary's batch connector, which its docs say polls"
},
{
"label": "They have keys and CDC is fine",
@@ -145,14 +151,14 @@
"estuary": 3,
"olake": 3
},
- "why": "Estuary and OLake switch to a second capture instance"
+ "why": "Estuary and OLake Go both switch to a second capture instance"
},
{
"label": "Rarely",
"scores": {
"fivetran": 2
},
- "why": "schema changes are not a worry"
+ "why": "new columns are rare, so Fivetran's single capture instance rule matters less"
}
]
},
@@ -164,9 +170,10 @@
"scores": {
"striim": 3,
"olake": 2,
- "estuary": 1
+ "debezium": 2,
+ "fivetran": 1
},
- "why": "lower load or a replica read matters to you"
+ "why": "lower load (Striim's claim for MSJet) or a read from a replica matters to you"
},
{
"label": "Not much",
@@ -183,47 +190,47 @@
"results": {
"debezium": {
"name": "Debezium",
- "summary": "An open-source CDC engine that reads SQL Server change tables and writes to Kafka topics, one per table.",
+ "summary": "A CDC engine with a free licence that reads SQL Server change tables and writes to Kafka topics, one per table.",
"bestWhen": "Kafka is already in your stack.",
- "watchOut": "after an outage the docs warn of duplicate events, and you run Kafka plus a schema history topic yourself."
+ "watchOut": "after an outage its docs warn of duplicate events, so your loads must cope with repeats."
},
"fivetran": {
"name": "Fivetran",
"summary": "A vendor-run service that documents four ways to find SQL Server changes, including Teleport Sync for when CDC is off.",
- "bestWhen": "you want a vendor to run the pipeline and you are fine with row-based billing.",
- "watchOut": "with CDC it syncs only tables with a single capture instance, so a new column means a new instance."
+ "bestWhen": "you want a vendor to run the pipeline and you are fine with billing on monthly active rows.",
+ "watchOut": "with CDC it syncs only tables with a single capture instance, so a new column means a new instance and removing the old one."
},
"airbyte": {
"name": "Airbyte",
- "summary": "An open-source option whose SQL Server connector uses CDC in scheduled syncs, with Iceberg on S3 through its S3 Data Lake destination.",
+ "summary": "Its Core edition is free to self-run, and Airbyte calls it open source. The SQL Server connector uses CDC in scheduled syncs, with Iceberg on S3 through its S3 Data Lake destination.",
"bestWhen": "scheduled syncs are fine and you want something you can run yourself.",
- "watchOut": "incremental CDC needs a primary key, delivery is at least once, and its docs do not treat CDC as continuous streaming."
+ "watchOut": "incremental CDC needs a primary key, delivery is at least once, and its CDC docs say it does not treat CDC sources as infinite streaming sources."
},
"estuary": {
"name": "Estuary",
- "summary": "Three SQL Server connectors: CDC, change tracking and batch, so you can pick per table.",
- "bestWhen": "your tables are a mix, some with keys and some without.",
- "watchOut": "the CDC worker must run on the primary, and its platform uses the Business Source License."
+ "summary": "Three SQL Server connectors: CDC, change tracking and batch, so you can pick the one that fits.",
+ "bestWhen": "you want to choose between CDC and change tracking, or you have tables without a primary key.",
+ "watchOut": "the CDC worker must run on the primary, and the batch connector polls, so expect latency of minutes to hours."
},
"striim": {
"name": "Striim",
- "summary": "Two readers: MS SQL Reader for CDC tables and MSJet for log backups, which has lower use of the source server.",
- "bestWhen": "source load is your main worry and you run big, busy servers.",
- "watchOut": "its Iceberg writers cover Google Cloud Storage or Amazon S3 Tables, not plain S3."
+ "summary": "Two readers: MS SQL Reader for CDC tables and MSJet for log backups, which Striim says uses less of the source server.",
+ "bestWhen": "you want to read log backups, and Striim's claim of lower load on the source matters to you.",
+ "watchOut": "its Iceberg Writer docs cover Google Cloud Storage, and its S3 Tables Writer covers Amazon S3 Tables."
},
"fabric": {
"name": "Microsoft Fabric mirroring",
"summary": "Microsoft's own way to copy a SQL Server database into Fabric.",
"bestWhen": "Fabric is where your analysts work.",
"watchOut": "data lands in Fabric OneLake as Delta tables, and a schema change breaks mirroring on 2016 to 2022 until you re-enable CDC for the table.",
- "href": "#10-microsoft-fabric-mirroring-best-for-teams-whose-target-is-microsoft-fabric"
+ "href": "#11-microsoft-fabric-mirroring-best-for-teams-whose-target-is-microsoft-fabric"
},
"olake": {
- "name": "OLake Go (ours)",
- "summary": "Our open-source tool. Per its docs it reads SQL Server CDC capture instances (2017 or later) and writes Apache Iceberg or Parquet.",
+ "name": "OLake Go",
+ "summary": "Our open-source tool. It reads SQL Server CDC capture instances (2017 or later) and writes Apache Iceberg or Parquet.",
"bestWhen": "you want SQL Server changes in Iceberg tables on storage you own and can run a container.",
- "watchOut": "it writes Iceberg and Parquet, you run it on your own infrastructure, and CDC needs to be enabled on each table first.",
- "href": "#11-olake-go-best-for-sql-server-changes-into-iceberg-or-parquet-on-your-own-storage"
+ "watchOut": "output is Apache Iceberg or Parquet, you run it on your own infrastructure, and it needs to keep reading inside SQL Server's three-day change table window (Microsoft's default).",
+ "href": "#6-olake-go-best-for-sql-server-changes-into-iceberg-or-parquet-on-your-own-storage"
}
}
}
diff --git a/src/data/quizzes/what-is-a-data-pipeline.json b/src/data/quizzes/what-is-a-data-pipeline.json
index 8efddf4b..65e021f0 100644
--- a/src/data/quizzes/what-is-a-data-pipeline.json
+++ b/src/data/quizzes/what-is-a-data-pipeline.json
@@ -14,7 +14,7 @@
"Only when the data ends up in cheap file storage"
],
"correct": 1,
- "explain": "IBM says a data pipeline may simply move data from one system to another. ETL is one kind of pipeline, so every ETL pipeline is a data pipeline but not the other way round."
+ "explain": "IBM says a data pipeline may simply move data from one system to another. On IBM's reading, ETL is one kind of pipeline, so every ETL pipeline is a data pipeline but not the other way round."
},
{
"q": "A pipeline loads data in groups at set times, say once a night. How fresh is the data in the report?",
@@ -22,7 +22,7 @@
"Updated the second something changes",
"Updated only when someone clicks refresh",
"Always a few seconds old",
- "As old as the last run"
+ "As old as the last successful run"
],
"correct": 3,
"explain": "That is a batch pipeline. It is simple, and it fits reports that can be a few hours or a day old. If you need seconds, that is when streaming earns its keep."
diff --git a/src/data/quizzes/what-is-fivetran.json b/src/data/quizzes/what-is-fivetran.json
index b4347ce8..195cff79 100644
--- a/src/data/quizzes/what-is-fivetran.json
+++ b/src/data/quizzes/what-is-fivetran.json
@@ -10,11 +10,11 @@
"options": [
"It builds your dashboards and reports",
"It stores your only copy of the data for safekeeping",
- "It copies data from your apps and databases into a warehouse or lake and keeps it fresh",
+ "It copies data from your apps and databases into a warehouse or lake on a schedule",
"It cleans and models the data so your team never has to write SQL"
],
"correct": 2,
- "explain": "You pick a source, a destination and a schedule, and Fivetran runs the connector. The reshaping afterwards, usually in SQL, is still your job."
+ "explain": "Per Fivetran, you pick a source, a destination and a schedule, and Fivetran runs the connector. The reshaping afterwards happens in your destination, with pre-built models or dbt, and is still your job."
},
{
"q": "A new connection finishes its first big copy. What does it do from then on?",
@@ -28,7 +28,7 @@
"explain": "The first run is the initial sync and copies everything you selected. After that it switches to incremental mode and moves only the changes."
},
{
- "q": "In the post's orders example, one order is deleted in the source database. What does the destination show afterwards?",
+ "q": "In the post's orders example (a PostgreSQL source using logical replication), one order is deleted in the source database. What does the destination show afterwards?",
"options": [
"The order is gone, so reports match the source on their own",
"The whole table is copied again from scratch",
@@ -36,7 +36,7 @@
"The order is still there, just marked as deleted"
],
"correct": 3,
- "explain": "Order 3 stays in the destination, flagged as deleted. Your queries need to filter flagged rows out, or deleted orders sneak into your reports."
+ "explain": "Per Fivetran's PostgreSQL connector docs, the destination row for order 3 is updated with _fivetran_deleted = TRUE, so it stays in the destination, flagged. Your queries need to filter flagged rows out, or deleted orders sneak into your reports."
},
{
"q": "A database keeps its change log for a limited time. What happens if Fivetran falls behind for longer than that?",
@@ -47,7 +47,7 @@
"It switches your plan to a cheaper tier"
],
"correct": 0,
- "explain": "Think of the log as a diary the database wipes after a while. If Fivetran misses the window, the changes are gone, so the fix is a full re-copy. Big tables mean extra load and extra waiting."
+ "explain": "Fivetran's docs say expired change logs require a full source re-sync. Think of the log as a diary the database wipes after a while: if Fivetran misses the window, the changes are gone. On a big table, plan for extra load and extra waiting."
},
{
"q": "Fivetran bills by monthly active rows, counted per connection. Which setup costs more at the same list rates?",
@@ -58,7 +58,7 @@
"It depends only on how many users you add"
],
"correct": 2,
- "explain": "Each connection is priced on its own curve and you can't pool them. The post's Standard-plan example is about $1,010 for one 4 million row connection, against $2,000 for four 1 million row ones."
+ "explain": "Each connection is priced on its own curve and you cannot pool them. In the post's Standard-plan example, from Fivetran's Service Consumption Table at list rates, one 4 million row connection costs about $1,010, against $2,000 for four 1 million row ones."
}
]
}
diff --git a/src/data/quizzes/what-is-matillion.json b/src/data/quizzes/what-is-matillion.json
index f1f86c60..b598a287 100644
--- a/src/data/quizzes/what-is-matillion.json
+++ b/src/data/quizzes/what-is-matillion.json
@@ -25,7 +25,7 @@
"On the laptop of whoever built the pipeline"
],
"correct": 2,
- "explain": "The transforms run as SQL inside your warehouse, which is called push-down. So the warehouse does the heavy lifting, and yes, your warehouse bill grows with it."
+ "explain": "Matillion's own comparison post says Maia Foundation runs transformations as native SQL pushed down into the warehouse, which is called push-down. So the warehouse does the heavy lifting, and your warehouse bill grows with it."
},
{
"q": "You open Matillion's pricing page to see what Developer, Teams and Scale cost. What do you find?",
@@ -36,7 +36,7 @@
"Features for each plan, but no dollar prices"
],
"correct": 3,
- "explain": "You pay in credits, and the page shows no dollar amount for any plan. Matillion also does not publish the price of one credit, so ask for it in writing before you sign."
+ "explain": "You pay in credits, and the pricing page shows no dollar amount for any plan. The post found no price per credit on Matillion's official pages either, so ask for it in writing before you sign."
},
{
"q": "Someone forgets to switch off an older Matillion ETL instance over the weekend. What happens?",
@@ -47,7 +47,7 @@
"The pipelines fail until someone restarts it"
],
"correct": 1,
- "explain": "Matillion ETL counts credits for every hour the software is on in your cloud account. In the post's example, the same instance cost 3 times as much when left on all month."
+ "explain": "Per Matillion's credit terms, Matillion ETL counts credits for every hour the software is on in your cloud account. In the post's invented example, the same 4-core instance used 3 times as many credits when left on all month."
},
{
"q": "On the current Maia Foundation platform, which kind of pipeline uses more credits for each hour it runs?",
@@ -58,7 +58,7 @@
"Neither one, because both are free"
],
"correct": 0,
- "explain": "A transformation hour costs 3 credits, while a loading hour costs 1. So trimming an hour of transformation saves more than trimming an hour of loading."
+ "explain": "Per Matillion's credit terms, a transformation hour costs 3 credits, while an orchestration (loading) hour costs 1. So trimming an hour of transformation saves more than trimming an hour of loading."
}
]
}
diff --git a/src/theme/BlogPostItems/index.js b/src/theme/BlogPostItems/index.js
index 8fa3a76b..150a16d5 100644
--- a/src/theme/BlogPostItems/index.js
+++ b/src/theme/BlogPostItems/index.js
@@ -30,6 +30,7 @@ export default function BlogPostItems({ items, featured = false }) {
priority={index === 0}
featured={featured && index === 0}
half={index >= halfFrom}
+ showAuthor={false}
formatDate={formatDate}
/>
))}
diff --git a/static/img/blog/2026/10/aws-olap-oltp-comparison-table.webp b/static/img/blog/2026/10/aws-olap-oltp-comparison-table.webp
new file mode 100644
index 00000000..931a5c9d
Binary files /dev/null and b/static/img/blog/2026/10/aws-olap-oltp-comparison-table.webp differ
diff --git a/static/img/blog/2026/10/best-data-management-platforms--airbyte-pricing-plans.webp b/static/img/blog/2026/10/best-data-management-platforms--airbyte-pricing-plans.webp
new file mode 100644
index 00000000..ac6894c9
Binary files /dev/null and b/static/img/blog/2026/10/best-data-management-platforms--airbyte-pricing-plans.webp differ
diff --git a/static/img/blog/2026/10/best-data-management-platforms--bigquery-on-demand-free-tib.webp b/static/img/blog/2026/10/best-data-management-platforms--bigquery-on-demand-free-tib.webp
new file mode 100644
index 00000000..cd798a3e
Binary files /dev/null and b/static/img/blog/2026/10/best-data-management-platforms--bigquery-on-demand-free-tib.webp differ
diff --git a/static/img/blog/2026/10/best-data-management-platforms--snowflake-pricing-options.webp b/static/img/blog/2026/10/best-data-management-platforms--snowflake-pricing-options.webp
new file mode 100644
index 00000000..387ad5d9
Binary files /dev/null and b/static/img/blog/2026/10/best-data-management-platforms--snowflake-pricing-options.webp differ
diff --git a/static/img/blog/2026/10/best-data-management-platforms-purview-governed-assets.webp b/static/img/blog/2026/10/best-data-management-platforms-purview-governed-assets.webp
new file mode 100644
index 00000000..433fb7ab
Binary files /dev/null and b/static/img/blog/2026/10/best-data-management-platforms-purview-governed-assets.webp differ
diff --git a/static/img/blog/2026/10/best-data-management-platforms-start-from-problem.webp b/static/img/blog/2026/10/best-data-management-platforms-start-from-problem.webp
new file mode 100644
index 00000000..dd58c5e7
Binary files /dev/null and b/static/img/blog/2026/10/best-data-management-platforms-start-from-problem.webp differ
diff --git a/static/img/blog/2026/10/best-data-management-platforms-two-meanings.webp b/static/img/blog/2026/10/best-data-management-platforms-two-meanings.webp
new file mode 100644
index 00000000..417b2b78
Binary files /dev/null and b/static/img/blog/2026/10/best-data-management-platforms-two-meanings.webp differ
diff --git a/static/img/blog/2026/10/best-data-pipeline-tools--dagster-pricing-plans.webp b/static/img/blog/2026/10/best-data-pipeline-tools--dagster-pricing-plans.webp
new file mode 100644
index 00000000..b1b6fe59
Binary files /dev/null and b/static/img/blog/2026/10/best-data-pipeline-tools--dagster-pricing-plans.webp differ
diff --git a/static/img/blog/2026/10/best-data-pipeline-tools--dbt-pricing-plans.webp b/static/img/blog/2026/10/best-data-pipeline-tools--dbt-pricing-plans.webp
new file mode 100644
index 00000000..8d4e860f
Binary files /dev/null and b/static/img/blog/2026/10/best-data-pipeline-tools--dbt-pricing-plans.webp differ
diff --git a/static/img/blog/2026/10/best-data-pipeline-tools--hevo-pricing-plans.webp b/static/img/blog/2026/10/best-data-pipeline-tools--hevo-pricing-plans.webp
new file mode 100644
index 00000000..7ab79e88
Binary files /dev/null and b/static/img/blog/2026/10/best-data-pipeline-tools--hevo-pricing-plans.webp differ
diff --git a/static/img/blog/2026/10/best-data-pipeline-tools--mwaa-pricing-rates.webp b/static/img/blog/2026/10/best-data-pipeline-tools--mwaa-pricing-rates.webp
new file mode 100644
index 00000000..c5a3413a
Binary files /dev/null and b/static/img/blog/2026/10/best-data-pipeline-tools--mwaa-pricing-rates.webp differ
diff --git a/static/img/blog/2026/10/best-data-pipeline-tools--prefect-pricing-plans.webp b/static/img/blog/2026/10/best-data-pipeline-tools--prefect-pricing-plans.webp
new file mode 100644
index 00000000..128e8a9d
Binary files /dev/null and b/static/img/blog/2026/10/best-data-pipeline-tools--prefect-pricing-plans.webp differ
diff --git a/static/img/blog/2026/10/best-data-pipeline-tools-build-from-parts.webp b/static/img/blog/2026/10/best-data-pipeline-tools-build-from-parts.webp
new file mode 100644
index 00000000..04041396
Binary files /dev/null and b/static/img/blog/2026/10/best-data-pipeline-tools-build-from-parts.webp differ
diff --git a/static/img/blog/2026/10/best-data-pipeline-tools-four-jobs.webp b/static/img/blog/2026/10/best-data-pipeline-tools-four-jobs.webp
new file mode 100644
index 00000000..4b26867b
Binary files /dev/null and b/static/img/blog/2026/10/best-data-pipeline-tools-four-jobs.webp differ
diff --git a/static/img/blog/2026/10/best-data-quality-tools--dbt-data-tests-plain-english.webp b/static/img/blog/2026/10/best-data-quality-tools--dbt-data-tests-plain-english.webp
new file mode 100644
index 00000000..1c2f5c87
Binary files /dev/null and b/static/img/blog/2026/10/best-data-quality-tools--dbt-data-tests-plain-english.webp differ
diff --git a/static/img/blog/2026/10/best-data-quality-tools--dbt-pricing-plans-quality.webp b/static/img/blog/2026/10/best-data-quality-tools--dbt-pricing-plans-quality.webp
new file mode 100644
index 00000000..8d4e860f
Binary files /dev/null and b/static/img/blog/2026/10/best-data-quality-tools--dbt-pricing-plans-quality.webp differ
diff --git a/static/img/blog/2026/10/best-data-quality-tools--soda-pricing-plans.webp b/static/img/blog/2026/10/best-data-quality-tools--soda-pricing-plans.webp
new file mode 100644
index 00000000..ca258d8b
Binary files /dev/null and b/static/img/blog/2026/10/best-data-quality-tools--soda-pricing-plans.webp differ
diff --git a/static/img/blog/2026/10/best-data-quality-tools-adoption-path.webp b/static/img/blog/2026/10/best-data-quality-tools-adoption-path.webp
new file mode 100644
index 00000000..49be6b47
Binary files /dev/null and b/static/img/blog/2026/10/best-data-quality-tools-adoption-path.webp differ
diff --git a/static/img/blog/2026/10/best-data-quality-tools-test-flow.webp b/static/img/blog/2026/10/best-data-quality-tools-test-flow.webp
new file mode 100644
index 00000000..39079d1c
Binary files /dev/null and b/static/img/blog/2026/10/best-data-quality-tools-test-flow.webp differ
diff --git a/static/img/blog/2026/10/best-data-quality-tools-two-jobs.webp b/static/img/blog/2026/10/best-data-quality-tools-two-jobs.webp
new file mode 100644
index 00000000..5e5a4ff8
Binary files /dev/null and b/static/img/blog/2026/10/best-data-quality-tools-two-jobs.webp differ
diff --git a/static/img/blog/2026/10/elementary-cloud-pricing.webp b/static/img/blog/2026/10/elementary-cloud-pricing.webp
new file mode 100644
index 00000000..93194489
Binary files /dev/null and b/static/img/blog/2026/10/elementary-cloud-pricing.webp differ
diff --git a/static/img/blog/2026/10/landing-apache-airflow.webp b/static/img/blog/2026/10/landing-apache-airflow.webp
new file mode 100644
index 00000000..596f2b4d
Binary files /dev/null and b/static/img/blog/2026/10/landing-apache-airflow.webp differ
diff --git a/static/img/blog/2026/10/landing-apache-flink.webp b/static/img/blog/2026/10/landing-apache-flink.webp
new file mode 100644
index 00000000..8919cd63
Binary files /dev/null and b/static/img/blog/2026/10/landing-apache-flink.webp differ
diff --git a/static/img/blog/2026/10/landing-ataccama.webp b/static/img/blog/2026/10/landing-ataccama.webp
new file mode 100644
index 00000000..ead302ff
Binary files /dev/null and b/static/img/blog/2026/10/landing-ataccama.webp differ
diff --git a/static/img/blog/2026/10/landing-collibra.webp b/static/img/blog/2026/10/landing-collibra.webp
new file mode 100644
index 00000000..8ed03afe
Binary files /dev/null and b/static/img/blog/2026/10/landing-collibra.webp differ
diff --git a/static/img/blog/2026/10/landing-dagster.webp b/static/img/blog/2026/10/landing-dagster.webp
new file mode 100644
index 00000000..ee8e6753
Binary files /dev/null and b/static/img/blog/2026/10/landing-dagster.webp differ
diff --git a/static/img/blog/2026/10/landing-datahub.webp b/static/img/blog/2026/10/landing-datahub.webp
new file mode 100644
index 00000000..52beaf74
Binary files /dev/null and b/static/img/blog/2026/10/landing-datahub.webp differ
diff --git a/static/img/blog/2026/10/landing-dbt.webp b/static/img/blog/2026/10/landing-dbt.webp
new file mode 100644
index 00000000..537e809e
Binary files /dev/null and b/static/img/blog/2026/10/landing-dbt.webp differ
diff --git a/static/img/blog/2026/10/landing-deequ.webp b/static/img/blog/2026/10/landing-deequ.webp
new file mode 100644
index 00000000..e5d7ed0b
Binary files /dev/null and b/static/img/blog/2026/10/landing-deequ.webp differ
diff --git a/static/img/blog/2026/10/landing-elementary.webp b/static/img/blog/2026/10/landing-elementary.webp
new file mode 100644
index 00000000..85a5a27c
Binary files /dev/null and b/static/img/blog/2026/10/landing-elementary.webp differ
diff --git a/static/img/blog/2026/10/landing-great-expectations.webp b/static/img/blog/2026/10/landing-great-expectations.webp
new file mode 100644
index 00000000..7dad5e77
Binary files /dev/null and b/static/img/blog/2026/10/landing-great-expectations.webp differ
diff --git a/static/img/blog/2026/10/landing-kafka.webp b/static/img/blog/2026/10/landing-kafka.webp
new file mode 100644
index 00000000..772489f6
Binary files /dev/null and b/static/img/blog/2026/10/landing-kafka.webp differ
diff --git a/static/img/blog/2026/10/landing-microsoft-purview.webp b/static/img/blog/2026/10/landing-microsoft-purview.webp
new file mode 100644
index 00000000..6e15c7db
Binary files /dev/null and b/static/img/blog/2026/10/landing-microsoft-purview.webp differ
diff --git a/static/img/blog/2026/10/landing-openmetadata.webp b/static/img/blog/2026/10/landing-openmetadata.webp
new file mode 100644
index 00000000..bb422e39
Binary files /dev/null and b/static/img/blog/2026/10/landing-openmetadata.webp differ
diff --git a/static/img/blog/2026/10/landing-prefect.webp b/static/img/blog/2026/10/landing-prefect.webp
new file mode 100644
index 00000000..4e9f7b14
Binary files /dev/null and b/static/img/blog/2026/10/landing-prefect.webp differ
diff --git a/static/img/blog/2026/10/landing-semarchy.webp b/static/img/blog/2026/10/landing-semarchy.webp
new file mode 100644
index 00000000..64daa87c
Binary files /dev/null and b/static/img/blog/2026/10/landing-semarchy.webp differ
diff --git a/static/img/blog/2026/10/landing-soda.webp b/static/img/blog/2026/10/landing-soda.webp
new file mode 100644
index 00000000..005108dc
Binary files /dev/null and b/static/img/blog/2026/10/landing-soda.webp differ
diff --git a/static/img/blog/2026/10/mysql-atomic-ddl-not-transactional.webp b/static/img/blog/2026/10/mysql-atomic-ddl-not-transactional.webp
new file mode 100644
index 00000000..db97cbd7
Binary files /dev/null and b/static/img/blog/2026/10/mysql-atomic-ddl-not-transactional.webp differ
diff --git a/static/img/blog/2026/10/mysql-faq-no-materialized-views.webp b/static/img/blog/2026/10/mysql-faq-no-materialized-views.webp
new file mode 100644
index 00000000..9e0900bb
Binary files /dev/null and b/static/img/blog/2026/10/mysql-faq-no-materialized-views.webp differ
diff --git a/static/img/blog/2026/10/oltp-vs-olap--microsoft-olap-d.webp b/static/img/blog/2026/10/oltp-vs-olap--microsoft-olap-d.webp
new file mode 100644
index 00000000..c748ee25
Binary files /dev/null and b/static/img/blog/2026/10/oltp-vs-olap--microsoft-olap-d.webp differ
diff --git a/static/img/blog/2026/10/oltp-vs-olap--microsoft-oltp-d.webp b/static/img/blog/2026/10/oltp-vs-olap--microsoft-oltp-d.webp
new file mode 100644
index 00000000..22055585
Binary files /dev/null and b/static/img/blog/2026/10/oltp-vs-olap--microsoft-oltp-d.webp differ
diff --git a/static/img/blog/2026/10/oltp-vs-olap--snowflake-hybrid-a.webp b/static/img/blog/2026/10/oltp-vs-olap--snowflake-hybrid-a.webp
new file mode 100644
index 00000000..5c4dd873
Binary files /dev/null and b/static/img/blog/2026/10/oltp-vs-olap--snowflake-hybrid-a.webp differ
diff --git a/static/img/blog/2026/10/oltp-vs-olap-two-systems.webp b/static/img/blog/2026/10/oltp-vs-olap-two-systems.webp
new file mode 100644
index 00000000..daa1301d
Binary files /dev/null and b/static/img/blog/2026/10/oltp-vs-olap-two-systems.webp differ
diff --git a/static/img/blog/2026/10/oltp-vs-olap-which-fits.webp b/static/img/blog/2026/10/oltp-vs-olap-which-fits.webp
new file mode 100644
index 00000000..ac723ac7
Binary files /dev/null and b/static/img/blog/2026/10/oltp-vs-olap-which-fits.webp differ
diff --git a/static/img/blog/2026/10/postgres-index-types-list.webp b/static/img/blog/2026/10/postgres-index-types-list.webp
new file mode 100644
index 00000000..5e2866e6
Binary files /dev/null and b/static/img/blog/2026/10/postgres-index-types-list.webp differ
diff --git a/static/img/blog/2026/10/postgres-jsonb-preferred.webp b/static/img/blog/2026/10/postgres-jsonb-preferred.webp
new file mode 100644
index 00000000..c1358107
Binary files /dev/null and b/static/img/blog/2026/10/postgres-jsonb-preferred.webp differ
diff --git a/static/img/blog/2026/10/postgres-sql-conformance.webp b/static/img/blog/2026/10/postgres-sql-conformance.webp
new file mode 100644
index 00000000..0a817226
Binary files /dev/null and b/static/img/blog/2026/10/postgres-sql-conformance.webp differ
diff --git a/static/img/blog/2026/10/postgres-vs-mysql-ddl.webp b/static/img/blog/2026/10/postgres-vs-mysql-ddl.webp
new file mode 100644
index 00000000..a0ecc9cc
Binary files /dev/null and b/static/img/blog/2026/10/postgres-vs-mysql-ddl.webp differ
diff --git a/static/img/blog/2026/10/postgres-vs-mysql-which-fits.webp b/static/img/blog/2026/10/postgres-vs-mysql-which-fits.webp
new file mode 100644
index 00000000..d83ffaba
Binary files /dev/null and b/static/img/blog/2026/10/postgres-vs-mysql-which-fits.webp differ
diff --git a/static/img/blog/cover/best-data-management-platforms-cover.webp b/static/img/blog/cover/best-data-management-platforms-cover.webp
new file mode 100644
index 00000000..1958eb9e
Binary files /dev/null and b/static/img/blog/cover/best-data-management-platforms-cover.webp differ
diff --git a/static/img/blog/cover/best-data-pipeline-tools-cover.webp b/static/img/blog/cover/best-data-pipeline-tools-cover.webp
new file mode 100644
index 00000000..ffebcc84
Binary files /dev/null and b/static/img/blog/cover/best-data-pipeline-tools-cover.webp differ
diff --git a/static/img/blog/cover/best-data-quality-tools-cover.webp b/static/img/blog/cover/best-data-quality-tools-cover.webp
new file mode 100644
index 00000000..5f47ad0e
Binary files /dev/null and b/static/img/blog/cover/best-data-quality-tools-cover.webp differ
diff --git a/static/img/blog/cover/oltp-vs-olap-cover.webp b/static/img/blog/cover/oltp-vs-olap-cover.webp
new file mode 100644
index 00000000..73ba0f53
Binary files /dev/null and b/static/img/blog/cover/oltp-vs-olap-cover.webp differ
diff --git a/static/img/blog/cover/postgres-vs-mysql-cover.webp b/static/img/blog/cover/postgres-vs-mysql-cover.webp
new file mode 100644
index 00000000..fb08fcac
Binary files /dev/null and b/static/img/blog/cover/postgres-vs-mysql-cover.webp differ
diff --git a/static/img/og/blog/cover/best-data-management-platforms-cover.jpg b/static/img/og/blog/cover/best-data-management-platforms-cover.jpg
new file mode 100644
index 00000000..e795d1f2
Binary files /dev/null and b/static/img/og/blog/cover/best-data-management-platforms-cover.jpg differ
diff --git a/static/img/og/blog/cover/best-data-pipeline-tools-cover.jpg b/static/img/og/blog/cover/best-data-pipeline-tools-cover.jpg
new file mode 100644
index 00000000..3b42abe1
Binary files /dev/null and b/static/img/og/blog/cover/best-data-pipeline-tools-cover.jpg differ
diff --git a/static/img/og/blog/cover/best-data-quality-tools-cover.jpg b/static/img/og/blog/cover/best-data-quality-tools-cover.jpg
new file mode 100644
index 00000000..63575a2f
Binary files /dev/null and b/static/img/og/blog/cover/best-data-quality-tools-cover.jpg differ
diff --git a/static/img/og/blog/cover/oltp-vs-olap-cover.jpg b/static/img/og/blog/cover/oltp-vs-olap-cover.jpg
new file mode 100644
index 00000000..a68d2c52
Binary files /dev/null and b/static/img/og/blog/cover/oltp-vs-olap-cover.jpg differ
diff --git a/static/img/og/blog/cover/postgres-vs-mysql-cover.jpg b/static/img/og/blog/cover/postgres-vs-mysql-cover.jpg
new file mode 100644
index 00000000..f2cbb559
Binary files /dev/null and b/static/img/og/blog/cover/postgres-vs-mysql-cover.jpg differ