feat: external-table feature series + MySQL SQLSTATEs [port to 4.2-dev of #27489, #27531, #27517, #27710, #27611] by fengttt · Pull Request #27769 · matrixorigin/matrixone

fengttt · 2026-08-27T22:51:45Z

What type of PR is this?

Which issue(s) this PR fixes:

issue #27488, issue #27518, issue #27517, issue #27610

What this PR does / why we need it:

Ports the external-table feature series from main to 4.2-dev, plus the
MySQL SQLSTATE / txn-fatal-error work that depends on it, as five separate
cherry-picks in their original order. This stacks on the datastream external
table already in 4.2-dev (#27479).

Ported commit	Feature
`861a0f5ea6` (#27489)	`esql_tvf` / `sql_tvf` table functions and `ENGINE = ESQL｜SQL` foreign external tables
`6742f958d4` (#27531)	`ENGINE = KAFKA` external table and `LAST_KAFKA_MESSAGE_ID()`
`3a3a80185b` (#27517, #27609)	error mode for external tables — `__mo_file_line` / `__mo_error_message` / `__mo_error_text`
`1d3483ac97` (#27710)	opt-in predicate pushdown for `ENGINE = SQL` external tables
`745d8364c5` (#27611)	MySQL SQLSTATEs, opt-in txn-fatal errors (`mo_rollback_txn_on_error`), typed `mo.sql` errors

Each commit message records its own adaptations; they are summarised here
because they are the whole review surface — no behaviour was redesigned for
4.2-dev.

Adaptations 4.2-dev required

build_show_util.go — no sqlMode threading. 4.2-dev's SHOW CREATE uses
formatStrInSingleQuotes / getRewriteSQLStmt, not the ...ForSQLMode
variants. The new Kafka and foreign-table option formatters, their call
sites, and their unit tests use the 4.2-dev convention. This is the same
adaptation the datastream port (feat: datastream external table (ENGINE = DATASTREAM) [port to 4.2-dev of #27469] #27479) made.
plan.Type has no Charset field. The __mo_query hidden column, the
synthetic Kafka columns, and the error-mode varchar columns are built like
their sibling hidden columns in 4.2-dev.
Function id renumbering. 4.2-dev has fewer builtins, so
ESQL_TVF_CONNECT / ESQL_TVF_DISCONNECT / SQL_TVF_CONNECT /
SQL_TVF_DISCONNECT become 563–566 and LAST_KAFKA_MESSAGE_ID becomes 567,
with FUNCTION_END_NUMBER 568. The main-only uuid_v* name-map entries are
not part of this port.
Main-only context dropped from conflict regions.
constructMongoScanScope (compile.go), remapRegularIndexPreInsert
(query_builder.go) and the temp-table commitTempTableStatement branch
(txn.go) do not exist in 4.2-dev and were swept into conflict hunks; only the
intended additions were kept. mysql_cmd_executor.go likewise keeps
4.2-dev's executeStmtWithTxn call and adds only the Kafka
FinalizeKafkaProgress(false) failure belt.
feat: MySQL SQLSTATEs, opt-in txn-fatal errors (mo_rollback_txn_on_error), and typed mo.sql errors #27611 rides along because it depends on this series. It updates
test/distributed/cases/function/foreign_exttab.result (Error 1146 (HY000) -> Error 1146 (42S02)), a file that only exists once feat: esql_tvf / sql_tvf table functions and ENGINE = ESQL|SQL foreign external tables #27489 is
ported, so it cannot go in a standalone PR against today's 4.2-dev. Its own
adaptations: main-only countUpdateChangedRows dropped from the
mysql_cmd_executor.go conflict hunk (only
rollbackWholeTxnOnPreExecutionError is added); ErrDuplicateEntry /
ErrWrongValueCountOnRow / ErrBadFieldError take the real SQLSTATEs
23000 / 21S01 / 42S22, which 4.2-dev still had on MySQLDefaultSqlState;
and the CODEOWNERS hunk is dropped, since it fixes a typo in a
/pkg/fulltext2 owner line 4.2-dev does not have.
Generated files are regenerated, not merged. mysql_sql.go,
plan.pb.go and pipeline.pb.go are rebuilt from the merged .y /
.proto sources. The grammar regenerates with zero shift/reduce conflicts.

Known gap: #27593 is NOT included

Main commit 0f10095703 — fix(kafka): use explicit transaction checkpoints
(#27593) — lands between commits 2 and 3 of this series and is not ported
here. It is a follow-up fix to the Kafka external table touching production
code (session_kafka.go, txn.go, external.go, kafka/config.go,
process/types.go), so 4.2-dev's Kafka engine keeps the original
implicit-commit progress path.

Because main's error-mode Kafka E2E cases were written on top of that fix,
they are adapted back to the pre-checkpoint harness (pendLastSet rather than
pendCheckpointSet) and the kafka_checkpoint last_id case is dropped. All
other error-mode cases are kept, including the negative control.

I confirmed by diffing the ported files against 1d3483ac97 that #27593 is the
only functional drift from main; the one remaining difference
(foreigntvf/esql.go) is a comment-only change from the unrelated fileservice
commit 1909f795c8. Happy to port #27593 as a follow-up if you want 4.2-dev
to match main.

Testing

go build ./... and go vet ./pkg/... ./test/... clean; gofmt clean across
all 127 changed files.

Unit suites passing on this branch:

sql/foreignext, sql/foreigntvf, sql/kafka, sql/datastream, sql/plan,
sql/plan/function, sql/colexec/external, sql/colexec/table_function,
sql/util/csvparser, all sql/parsers dialects, catalog, common/moerr,
the full frontend suite, vm/process, test/kafkaexttab (including
TestRunErrorModeAgainstSimulator and the TestRunErrorModeDetectsAWrongSplit
negative control) and test/esqltvf.

For #27611 specifically: its new tests pass (TestSqlStateMatchesMySQL,
TestDuplicateEntrySqlState, Test_isErrorRollbackWholeTxn,
TestNewErrorRollbackWholeTxnCoversEveryCode,
TestRollbackTxnOnErrorVarDefinition, TestPreExecutionRollbackIsGuarded and
the six typed mo.sql error tests). Because the SQLSTATE change ripples, I
also checked the blast radius: sql/plan and sql/plan/function, which assert
hardcoded SQLSTATEs, pass; and the BVT files containing HY000 carry it only
inside --ERROR comments that mo-tester ignores, for error numbers outside the
changed set.

BVT cases and resources are ported with the commits
(function/foreign_exttab, foreign_exttab_pushdown, foreign_conn_limit,
tvf_sql, kafka_exttab, table/external_table_error_mode{,_load},
pessimistic_transaction/rollback_txn_on_error,
procedure/starlark_sql_error); I have
not run the BVT suite locally — leaving that to CI.

…n external tables [port to 4.2-dev of matrixorigin#27489] Cherry-pick of 861a0f5 (merged to main as matrixorigin#27489) with the adaptations 4.2-dev requires: - `build_show_util.go`: 4.2-dev's SHOW CREATE does not thread `sqlMode`, so `formatForeignTableOptionsForShowCreate` and its call site use the no-`sqlMode` `formatStrInSingleQuotes` / `getRewriteSQLStmt` convention (same adaptation the datastream port made). - `function_id.go`: 4.2-dev has fewer builtins, so the four new ids are renumbered 563-566 (main: 572-575) and `FUNCTION_END_NUMBER` becomes 567. The main-only `uuid_v*` name-map entries are not part of this port. - `compile.go`: `constructMongoScanScope` is main-only context swept into the conflict region; only `compileForeignScan` is added here. - `query_builder.go`: 4.2-dev's `plan.Type` has no `Charset` field, so the `__mo_query` hidden column is built like its sibling hidden columns. - Generated files (`mysql_sql.go`, `plan.pb.go`, `pipeline.pb.go`) are regenerated from the merged `.y` / `.proto` sources rather than merged. Verified on 4.2-dev: `go build ./pkg/... ./test/...` and `go vet` clean; foreignext, foreigntvf, parsers/mysql, plan, plan/function, colexec/external, colexec/table_function packages pass, plus the compile and frontend foreign-scan/foreign-conn tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMQRVu998WixePcJCJvyA5

…o 4.2-dev of matrixorigin#27531] Cherry-pick of 6742f95 (merged to main as matrixorigin#27531) with the adaptations 4.2-dev requires: - `build_show_util.go`: 4.2-dev's SHOW CREATE does not thread `sqlMode`, so `formatKafkaTableOptionsForShowCreate` and its call site use the no-`sqlMode` `formatStrInSingleQuotes` / `getRewriteSQLStmt` convention. - `function_id.go`: `LAST_KAFKA_MESSAGE_ID` is renumbered 567 (main: 576) and `FUNCTION_END_NUMBER` becomes 568, continuing the 4.2-dev numbering. - `mysql_cmd_executor.go`: 4.2-dev calls `executeStmtWithTxn` rather than main's `executeStmtWithMaxExecutionTime`; only the Kafka `FinalizeKafkaProgress(false)` failure belt is added. - `txn.go`: main's temp-table `commitTempTableStatement` else-branch is not in 4.2-dev; only the `sessionFinalizeKafkaProgress(execCtx, true)` transaction terminal is added. - `compile.go`: `constructMongoScanScope` is main-only context swept into the conflict region; only `compileKafkaScan` is added here. - `query_builder.go`: 4.2-dev's `plan.Type` has no `Charset` field, so the synthetic Kafka varchar columns are built like the sibling hidden columns. - Generated files (`mysql_sql.go`, `plan.pb.go`, `pipeline.pb.go`) are regenerated from the merged `.y` / `.proto` sources rather than merged. Verified on 4.2-dev: `go build ./pkg/... ./test/...` and `go vet` clean; sql/kafka, plan, plan/function, colexec/external, util/csvparser, parsers/mysql, catalog and the full frontend suite pass, plus the compile Kafka/foreign/datastream scan tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMQRVu998WixePcJCJvyA5

…ssage / __mo_error_text [port to 4.2-dev of matrixorigin#27517] Cherry-pick of 3a3a801 (merged to main as matrixorigin#27517 / matrixorigin#27609) with the adaptations 4.2-dev requires: - `query_builder.go`: main-only `remapRegularIndexPreInsert` was swept into the conflict region; only `externalScanTolerates` is added here. 4.2-dev's `plan.Type` also has no `Charset` field, so the error-mode varchar columns are built like the sibling hidden columns. - `test/kafkaexttab/kafka_e2e_local_test.go`: main's error-mode e2e cases were written on top of matrixorigin#27593 ("fix(kafka): use explicit transaction checkpoints"), which is NOT part of this port. The cases are adapted back to the pre-checkpoint harness (`pendLastSet` rather than `pendCheckpointSet`) and the `kafka_checkpoint` last_id case is dropped. All other error-mode cases are kept. NOTE: main commit 0f10095 (matrixorigin#27593) is a follow-up fix to the Kafka external table that is not included in this port series; 4.2-dev's Kafka engine therefore still uses the original implicit-commit progress path. Verified on 4.2-dev: `go build ./pkg/... ./test/...` and `go vet` clean; colexec/external, plan, util/csvparser, catalog and test/kafkaexttab pass — the latter including TestRunErrorModeAgainstSimulator and the TestRunErrorModeDetectsAWrongSplit negative control. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMQRVu998WixePcJCJvyA5

…t to 4.2-dev of matrixorigin#27710] Cherry-pick of 1d3483a (merged to main as matrixorigin#27710) with the adaptations 4.2-dev requires: - `esql_sql_tvf_test.go`: the new `pushdown` SHOW CREATE case is adapted to the no-`sqlMode` `formatForeignTableOptionsForShowCreate` signature this port series uses on 4.2-dev. - `plan.pb.go` is regenerated from the merged `proto/plan.proto` rather than merged. Everything else (`reader_foreignscan.go`, `datastream/filter.go`, `foreignext/pushdown.go`, `foreigntvf/sql.go`, `compile.go`) applied cleanly. Verified on 4.2-dev: `go build ./pkg/... ./test/...` and `go vet` clean; foreignext, foreigntvf, datastream, plan, colexec/external pass, plus the compile foreign/kafka/datastream scan tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMQRVu998WixePcJCJvyA5

qodo-code-review · 2026-08-27T22:51:50Z

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

…ror), and typed mo.sql errors [port to 4.2-dev of matrixorigin#27611] Cherry-pick of 745d836 (merged to main as matrixorigin#27611) with the adaptations 4.2-dev requires: - `mysql_cmd_executor.go`: main-only `countUpdateChangedRows` was swept into the conflict region; only `rollbackWholeTxnOnPreExecutionError` is added here. Its call site in the COM_QUERY error defer applied cleanly. - `moerr/error.go`: the three entries that conflicted (`ErrDuplicateEntry`, `ErrWrongValueCountOnRow`, `ErrBadFieldError`) take the real MySQL SQLSTATEs 23000 / 21S01 / 42S22 — that change is the point of the commit; 4.2-dev still had them on `MySQLDefaultSqlState`. - `CODEOWNERS`: the upstream hunk fixes a typo (`@epegeric` -> `@cpegeric`) in a `/pkg/fulltext2` owner line that 4.2-dev does not have, so the hunk is dropped rather than added as a new ownership rule. This stacks on the external-table port series: the commit updates `test/distributed/cases/function/foreign_exttab.result` (`Error 1146 (HY000)` -> `Error 1146 (42S02)`), a file that only exists once matrixorigin#27489 is ported. Verified on 4.2-dev: `go build ./...` and `go vet ./pkg/... ./test/...` clean; gofmt clean. `common/moerr` and the full `frontend` suite pass, including the commit's new tests (TestSqlStateMatchesMySQL, TestDuplicateEntrySqlState, Test_isErrorRollbackWholeTxn, TestNewErrorRollbackWholeTxnCoversEveryCode, TestRollbackTxnOnErrorVarDefinition, TestPreExecutionRollbackIsGuarded and the six mo.sql typed-error tests). `sql/plan` and `sql/plan/function`, which assert hardcoded SQLSTATEs, also pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMQRVu998WixePcJCJvyA5

# Conflicts: # pkg/pb/pipeline/pipeline.pb.go

@regex

… 4.2-dev CI on this port series failed every new external-table BVT case. Three independent causes, all fixed here. 1. `pkg/vm/process/process2.go` — THE REAL BUG. `NewNoContextChildProc` and `NewNoContextChildProcWithChannel` did not copy `proc.Session` into the child, so every operator and builtin that reaches an optional session capability through `proc.GetSession()` saw a nil session and failed with "requires an interactive session" / "can only be read in an interactive session". main propagates the session here; 4.2-dev never needed it because its only `proc.Session` consumers were at compiler-context level, and the ported esql_tvf / sql_tvf / Kafka features are the first operator-level ones. Reproduced locally and confirmed NOT multi-CN specific: it failed with a single CN too, so the LAUNCH BVT group would have failed the same way once it ran those cases. A remote proc is built from scratch on the receiving CN and correctly keeps a nil session. 2. `external_table_error_mode_load.sql` — the case used multi-table `INSERT FIRST` / `INSERT ALL` (main matrixorigin#27470 / matrixorigin#27560), which 4.2-dev's parser does not have. The same routing is expressed as one INSERT per destination over the same scan, keeping the error-mode coverage intact (good rows, rejected rows with line/message/text, the every-row audit, and the "every source line accounted for exactly once" assertion). 3. The `.result` files were regenerated against 4.2-dev for three genuine differences from main: SHOW CREATE emits no per-column COLLATE (and different column metadata widths), an unset user variable reports "the user variable ... does not exist", and a missing table inside a loopback sql_tvf query surfaces as Error 1064 (42000) rather than 1146 (42S02). Note on regeneration: `genrs` relocated two `-- @regex(...)` directives and baked the environment-specific text they exist to tolerate (a Kafka dial error and a non-deterministic connection handle hash) into the expected output. Both hunks were restored by hand. Verified against a local CI-parity cluster (etc/launch-tae-compose, --profile launch-multi-cn, two CNs each with frontend port 6001 so the loopback DSN resolves as it does in CI): all seven previously failing cases pass (296 statements, 0 failures), plus regressions for the session change — cases/temporary 161/161, cases/datastream 43/43, cases/table 221/221. `go build ./...`, `go vet`, gofmt, molint, license-eye and golangci-lint clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMQRVu998WixePcJCJvyA5

Iceberg E2E Local was the last failing check. It is the only job that runs `go mod tidy -diff`, and it failed in `pkg/iceberg/adapter/iceberggo`, whose graph inherits the main module's requirements: the franz-go dependency added by the Kafka port raises transitive indirect versions (klauspost/compress 1.18.4 -> 1.18.7, pierrec/lz4/v4 4.1.25 -> 4.1.26, golang.org/x/crypto and friends), so the committed go.mod no longer matched a tidy resolution. Both modules are now tidy: - main module: the four franz-go requirements move into the direct require block and two stale indirects (dgryski/go-metro, bufbuild/protocompile) and their go.sum entries drop out. - pkg/iceberg/adapter/iceberggo: the inherited indirect versions are bumped to match. `go mod tidy -diff` is clean in both. The rest of the Iceberg job passes locally too: its go_test_core set (iceberg api/catalog/dml/io/maintenance/ metadata/model/ref/write, sql/iceberg, sql/compile, colexec/icebergdelete, colexec/external, pb/pipeline, sql/plan, sql/plan/explain), the iceberggo adapter tests, and the golden-vector provenance check. `go build ./...` and `go vet` clean; sql/kafka, colexec/external, frontend, compile, test/kafkaexttab and test/esqltvf all pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMQRVu998WixePcJCJvyA5

fengttt and others added 4 commits August 27, 2026 14:58

fengttt requested review from XuPeng-SH, aunjgr and iamlinjunhong as code owners August 27, 2026 22:51

fengttt temporarily deployed to ci August 27, 2026 22:51 — with GitHub Actions Inactive

fengttt had a problem deploying to ci August 27, 2026 22:51 — with GitHub Actions Error

fengttt temporarily deployed to ci August 27, 2026 22:52 — with GitHub Actions Inactive

matrix-meow added the size/XXL Denotes a PR that changes 2000+ lines label Aug 27, 2026

mergify Bot added the kind/feature label Aug 27, 2026

fengttt had a problem deploying to ci August 27, 2026 22:58 — with GitHub Actions Error

fengttt changed the title ~~feat: external-table feature series [port to 4.2-dev of #27489, #27531, #27517, #27710]~~ feat: external-table feature series + MySQL SQLSTATEs [port to 4.2-dev of #27489, #27531, #27517, #27710, #27611] Aug 27, 2026

fengttt temporarily deployed to ci August 27, 2026 23:28 — with GitHub Actions Inactive

fengttt had a problem deploying to ci August 27, 2026 23:28 — with GitHub Actions Failure

fengttt temporarily deployed to ci August 27, 2026 23:28 — with GitHub Actions Inactive

matrix-meow added size/XL Denotes a PR that changes [1000, 1999] lines and removed size/XXL Denotes a PR that changes 2000+ lines labels Aug 27, 2026

fengttt had a problem deploying to ci August 27, 2026 23:35 — with GitHub Actions Failure

fengttt had a problem deploying to ci August 28, 2026 00:14 — with GitHub Actions Failure

fengttt and others added 2 commits August 27, 2026 21:49

Merge remote-tracking branch 'upstream/4.2-dev' into port-exttab-4.2-dev

5ccc4b3

# Conflicts: # pkg/pb/pipeline/pipeline.pb.go

fengttt temporarily deployed to ci August 28, 2026 05:56 — with GitHub Actions Inactive

fengttt temporarily deployed to ci August 28, 2026 06:02 — with GitHub Actions Inactive

fengttt temporarily deployed to ci August 28, 2026 06:41 — with GitHub Actions Inactive

fengttt and others added 2 commits August 28, 2026 08:42

Merge remote-tracking branch 'upstream/4.2-dev' into port-exttab-4.2-dev

0b04deb

fengttt temporarily deployed to ci August 28, 2026 15:46 — with GitHub Actions Inactive

matrix-meow added size/XXL Denotes a PR that changes 2000+ lines and removed size/XL Denotes a PR that changes [1000, 1999] lines labels Aug 28, 2026

fengttt temporarily deployed to ci August 28, 2026 15:53 — with GitHub Actions Inactive

fengttt temporarily deployed to ci August 28, 2026 16:32 — with GitHub Actions Inactive

fengttt merged commit fca65de into matrixorigin:4.2-dev Aug 28, 2026
32 checks passed

fengttt temporarily deployed to ci August 28, 2026 17:18 — with GitHub Actions Inactive

fengttt temporarily deployed to ci August 28, 2026 17:29 — with GitHub Actions Inactive

fengttt had a problem deploying to ci August 28, 2026 17:40 — with GitHub Actions Failure

fengttt temporarily deployed to ci August 28, 2026 18:57 — with GitHub Actions Inactive

fengttt temporarily deployed to ci August 28, 2026 19:21 — with GitHub Actions Inactive

Navigation Menu

Uh oh!

feat: external-table feature series + MySQL SQLSTATEs [port to 4.2-dev of #27489, #27531, #27517, #27710, #27611] - #27769

feat: external-table feature series + MySQL SQLSTATEs [port to 4.2-dev of #27489, #27531, #27517, #27710, #27611]#27769
fengttt merged 9 commits into
matrixorigin:4.2-devfrom
fengttt:port-exttab-4.2-dev

fengttt commented Aug 27, 2026 •

edited

Loading

Uh oh!

qodo-code-review Bot commented Aug 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

fengttt commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

Which issue(s) this PR fixes:

What this PR does / why we need it:

Adaptations 4.2-dev required

Known gap: #27593 is NOT included

Testing

Uh oh!

qodo-code-review Bot commented Aug 27, 2026

Qodo reviews are paused for this user.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fengttt commented Aug 27, 2026 •

edited

Loading