feat(mysql): binary column support and backwards compatibility tests by mihir-datazip · Pull Request #1137 · datazip-inc/olake

mihir-datazip · 2026-08-21T05:41:48Z

Description

MySQL BINARY, VARBINARY and BLOB columns holding non-UTF-8 bytes could not sync. OLake cast them to string, and the Iceberg writer failed with string field contains invalid UTF-8. They now stay bytes end to end. BINARY(n) lands as Iceberg fixed[n], and the other types land as binary. State version 8, pinned to v0.12.2, gates the change, so syncs on older state keep string.

This PR also adds the backward-compatibility suite and splits the test harness to host it. The suite syncs each scenario twice, once on a baseline image and once with a switch to this build after the initial load, then compares the destinations. Separately, LoadedStateVersion read 0 in commands without a state file and is now set in init().

Fixes #1031

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
This change requires a documentation update

How Has This Been Tested?

Every driver's integration suite and the backward-compatibility suite passed in CI on 86711085 (run 38717757700). The run shows red only because the performance matrix is empty on PRs.
make test.unit, make test.build.all and both lint targets passed after merging the latest staging.
A manual MySQL full refresh and CDC of BINARY(16), BINARY(4), VARBINARY(64) and BLOB columns landed byte-exact in Iceberg. Spark hex() matched MySQL HEX().
The integration and backward-compatibility suites on the current head, against staging, run in this PR's CI.

Screenshots or Recordings

N/A

Documentation

Documentation Link: docs: mysql binary column handling in destination olake-docs#528
N/A (bug fix, refactor, or test changes only)

Related PR's (If Any):

feat: carry MySQL binary and fixed-length binary columns as bytes into Iceberg #1183, MySQL binary column support, merged into this branch
docs: mysql binary column handling in destination olake-docs#528, the MySQL to Iceberg type table

…mpatibility-tests

…atatype

…ersion 8 ReformatBytes passes byte slices through and turns any other value into the bytes of the text a string column would hold, using reformatString, which the String case now shares. A nil value returns ErrNullValue. fallbackType and fallbackIcebergType are now functions, so they read the state version a sync loads. An undeclared or unregistered type falls back to Binary from state version 8 and to String below it.

…olumn 07a8978 made undeclared and unregistered types fall back to Binary from state version 8. Null and Unknown are unregistered, so Mongo's always-null field and its Regex, MinKey and MaxKey fields (typed Unknown) became Iceberg binary columns and failed the Mongo integration tests. fallbackType is a String var again, now right after the DataType const block, and TestToIceberg and TestToNewParquet pin Null and Unknown. The Mongo suite now checks id_nil's destination type. Parquet writes an always-null column as untyped bytes, which Spark reads as binary, so TestHandler takes a ParquetTypeMapping that replaces TypeMapping for Parquet, and MongoTypeMapping becomes a function so MongoParquetTypeMapping can map null to binary on a fresh copy.

A Kafka sweep went red because the v0.7.0 baseline's own stateless load hung, before the upgrade side ever ran the candidate. The report still listed it as "a cdc side never finished case """ under a state version 6 headline, which points at the state handover. When the case a side never finished is the stateless load, the reason now adds that both sides were still on the baseline, before the upgrade, so no state version is implicated.

…mpatibility-tests

…atatype

…mpatibility-tests

github-actions · 2026-10-19T08:44:40Z

+		if width, isFixed := types.FixedBinaryWidth(p.schema[column].Type()); isFixed && len(b) != width {
+			return fmt.Errorf("field[%s]: %s holds exactly %d bytes, got %d", column, p.schema[column].Type(), width, len(b))
+		}


A legitimate width change has no evolution path, so this turns into a permanent failure for the rest of the sync run. The type-merge machinery cannot see the drift: FlattenAndCleanData's diff check (elem.ValueType() == detectedType) and Fields.Process (currentType.ValueType() != detectedType) both treat fixed_binary(n) and Binary as the same family, so a column declared binary(4) at discovery that receives 8-byte values after a mid-sync ALTER TABLE ... BINARY(8) (post-ALTER binlog/backfill row images are 8 bytes) never sets diffFound/typeChange, EvolveSchema never runs, and every subsequent record fails here as a WriteFailure; in-process retries keep failing since p.schema never widens. Recovery requires restarting the sync so discovery re-reads the new width. Consider widening the field to Binary on a width mismatch (fold fixed_binary(n)+Binary → Binary) and surfacing it through the existing schema-change flow so EvolveSchema reopens partition files with a byte_array leaf, keeping this guard only for values seen before the schema catches up.

github-actions · 2026-10-19T08:44:40Z

+		if err := run("build the iceberg writer jar", "make", "-C", worktree, "iceberg.jar"); err != nil {
+			return err
+		}
+		return run("build the image", "docker", "build", "--build-arg", "DRIVER_NAME="+cfg.Driver, "-t", imageTag, worktree)


This commit-worktree build runs a plain docker build without --platform, while every other path in the harness honors the platform pin: the make-based build passes --platform from drivers/platforms.conf (Makefile docker.%.build), ensureImagePresent pulls with --platform, and DockerRunArgs runs with --platform cfg.ImagePlatform. For a platform-pinned driver this is reachable: db2 pins linux/amd64 (tests/db2/db2_test.go) and registers a compatibility suite whose baselines resolve to commit IDs (OLAKE_COMPATIBILITY_TEST_BASELINE accepts commit specs; ResolveToCommit rewrites them in RunBackwardCompatibility). On a non-amd64 host the commit-built image has the host architecture and the subsequent docker run --platform linux/amd64 fails with an image/platform mismatch, breaking the test-a-previous-commit flow for exactly the drivers that need the pin. Pass the platform here like the pull and run paths do.

Suggestion:

Suggested change

return run("build the image", "docker", "build", "--build-arg", "DRIVER_NAME="+cfg.Driver, "-t", imageTag, worktree)

buildArgs := []string{"build", "--build-arg", "DRIVER_NAME=" + cfg.Driver, "-t", imageTag}

if cfg.ImagePlatform != "" {

buildArgs = append(buildArgs, "--platform", cfg.ImagePlatform)

}

return run("build the image", "docker", append(buildArgs, worktree)...)

github-actions · 2026-10-19T08:44:40Z

+	require.Eventually(t, func() bool {
+		if ctx.Err() != nil {
+			return true
+		}


A cancelled/expired context is reported as "the sync made progress" without stats.json ever being observed, silently breaking this function's documented contract ("blocks until the running sync has reported its first records"). The only current caller (the kafka rebalance trigger in tests/kafka/kafka_util_test.go) compensates by re-checking rebalanceCtx.Err() right after, so no misbehavior today, but the helper is exported for driver suites to call per its doc comment -- a caller following the contract proceeds to its mid-sync mutation against a dead or never-started sync, producing flaky downstream verification failures that mask the real sync-startup problem. (Behavior carried over from the previous harness.) Fail loudly instead of returning true.

Suggestion:

Suggested change

require.Eventually(t, func() bool {

if ctx.Err() != nil {

return true

}

require.Eventually(t, func() bool {

if ctx.Err() != nil {

// A dead context must not read as "sync made progress": report it, so a caller

// does not run its mid-sync mutation against a sync that never started.

t.Errorf("context ended before %s reported progress: %v", statsPath, ctx.Err())

return false

}

github-actions · 2026-10-19T08:44:40Z

+	th.icebergDestination = testutils.Ternary(useArrowWriter, icebergArrowDestinationFile, icebergDestinationFile).(string)
+
+	return testFunc(ctx, t, testTable)


th.icebergDestination is set for the duration of testFunc but never restored afterwards, so it keeps pointing at the last-selected writer config long after the scoped call returns. Today this is benign (every iceberg sync resolves destinationFile inside an IcebergWriter scope), but any future sync that goes through runSyncAndVerify/destinationFile outside this scope will silently target whichever writer ran last instead of the committed base config. Restore the previous value so the selection stays scoped to the call, matching what the per-useArrowWriter parameter implies.

Suggestion:

Suggested change

th.icebergDestination = testutils.Ternary(useArrowWriter, icebergArrowDestinationFile, icebergDestinationFile).(string)

return testFunc(ctx, t, testTable)

previous := th.icebergDestination

th.icebergDestination = testutils.Ternary(useArrowWriter, icebergArrowDestinationFile, icebergDestinationFile).(string)

defer func() { th.icebergDestination = previous }()

return testFunc(ctx, t, testTable)

mihir-datazip had a problem deploying to integration_tests August 21, 2026 05:42 — with GitHub Actions Failure

mihir-datazip had a problem deploying to integration_tests August 21, 2026 05:43 — with GitHub Actions Failure

mihir-datazip force-pushed the backward-compatibility-tests branch from 9de32a7 to d4795ee Compare August 24, 2026 12:45

mihir-datazip temporarily deployed to integration_tests August 24, 2026 12:45 — with GitHub Actions Inactive

mihir-datazip had a problem deploying to integration_tests August 24, 2026 12:55 — with GitHub Actions Failure

mihir-datazip temporarily deployed to integration_tests August 24, 2026 13:10 — with GitHub Actions Inactive

mihir-datazip temporarily deployed to integration_tests August 24, 2026 13:27 — with GitHub Actions Inactive

mihir-datazip had a problem deploying to integration_tests August 24, 2026 13:33 — with GitHub Actions Error

mihir-datazip temporarily deployed to integration_tests August 24, 2026 13:57 — with GitHub Actions Inactive

chore: address comments

517ba83

github-actions Bot reviewed Oct 12, 2026

View reviewed changes

Comment thread tests/testutils/state_version.go

chore: address comments

39504d4

github-actions Bot reviewed Oct 12, 2026

View reviewed changes

Comment thread tests/testutils/integration/iceberg_index.go Outdated

Comment thread tests/testutils/integration/iceberg_index.go Outdated

chore: address comments

eedfdc8

github-actions Bot reviewed Oct 12, 2026

View reviewed changes

chore: address comments

3090e8e

github-actions Bot reviewed Oct 12, 2026

View reviewed changes

Comment thread drivers/kafka/docker-compose.yml

Comment thread tests/testutils/integration/2pc.go

mihir-datazip added 14 commits October 13, 2026 14:46

chore: remove parent node logic

e1876ea

chore: address comments

1ee0730

chore: fix new version partition for byte values

f358ff4

chore: extract decodeCollatedValue and add unit test

ef9f152

chore: consolidate the String fallback into fallbackType

c1becce

chore: merge remote-tracking branch 'origin/staging' into backward-co…

fb89e3a

…mpatibility-tests

chore: merge branch 'backward-compatibility-tests' into feat/binary-d…

dab72f7

…atatype

chore: raise the kafka compatibility floor to v0.7.1

71ba78c

chore: merge remote-tracking branch 'origin/staging' into backward-co…

fc46216

…mpatibility-tests

chore: merge branch 'backward-compatibility-tests' into feat/binary-d…

8671108

…atatype

chore: move state version 8 and the binary compatibility rule to v0.12.2

0f08dc4

github-actions Bot reviewed Oct 15, 2026

View reviewed changes

chore: address comments

666d258

github-actions Bot reviewed Oct 15, 2026

View reviewed changes

Comment thread ...e-iceberg-java-writer/src/main/java/io/olake/iceberg/tableoperator/IcebergTableOperator.java

Comment thread utils/state_utility.go

mihir-datazip added 2 commits October 16, 2026 12:36

chore: merge remote-tracking branch 'origin/staging' into backward-co…

0d403d9

…mpatibility-tests

chore: merge remote-tracking branch 'origin/staging' into backward-co…

1609ec8

…mpatibility-tests

github-actions Bot reviewed Oct 19, 2026

View reviewed changes

Comment thread types/fixed_binary.go

Comment thread utils/utils.go Outdated

fix(utils): bound type parameters to 9 digits to fit a 32-bit int

c657f8a

github-actions Bot reviewed Oct 19, 2026

View reviewed changes

-		return run("build the image", "docker", "build", "--build-arg", "DRIVER_NAME="+cfg.Driver, "-t", imageTag, worktree)
+		buildArgs := []string{"build", "--build-arg", "DRIVER_NAME=" + cfg.Driver, "-t", imageTag}
+		if cfg.ImagePlatform != "" {
+			buildArgs = append(buildArgs, "--platform", cfg.ImagePlatform)
+		}
+		return run("build the image", "docker", append(buildArgs, worktree)...)

		th.icebergDestination = testutils.Ternary(useArrowWriter, icebergArrowDestinationFile, icebergDestinationFile).(string)

		return testFunc(ctx, t, testTable)

Conversation

mihir-datazip commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

How Has This Been Tested?

Screenshots or Recordings

Documentation

Related PR's (If Any):

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mihir-datazip commented Aug 21, 2026 •

edited

Loading