Add configurable string based migration splitting

[sidju]

The change

Adds a MigrationSplitter field to migrate.Migrate. When set, each migration is split on the delimiter before being passed to the database driver, with each migration part executed as a separate Run call. This allows migrations containing SQL bodies with embedded semicolons (e.g. DO $$ ... $$;) to be split reliably and flexibly, and probably suits other needs as well.

The CLI flag wraps the user-supplied string with newlines so the delimiter must appear as a complete line:

migrate -migration-splitter '---' up

Existing behaviour is unchanged when the flag is not set (nil takes the original single-Run path).

( Inspired by goose and #590 (comment) )

The goal

This is intended to act as a more flexible/configurable multi statement solution. Since the migration splitters are placed by the user exactly where needed they won't accidentally split any strings and still doesn't require any syntax awareness.

As an added bonus, this creates an alternate (and non bugged, see #581 ) way to split migrations into multiple transactions.

Issues resolved by this PR

Exactly as requested:

• Allow something like "-- +goose StatementBegin" in addition to ";" for multi-statement splitter #1237

A good workaround:

• Postgres / SQL multistatement support #1078
• Semicolon in string is not allowed with x-multi-statement option #1323
• pq: unterminated dollar-quoted string at or near "$BODY$ #590

[Copilot]

Pull request overview

Adds an opt-in migration splitting mechanism to migrate.Migrate so a single migration body can be executed as multiple sequential database.Driver.Run calls, driven by a configurable delimiter (including a new CLI flag and Postgres documentation).

Changes:

• Add MigrationSplitter []byte to migrate.Migrate and split migration bodies into steps when configured.
• Add -migration-splitter CLI flag (wrapped with newlines to enforce whole-line delimiters).
• Add unit tests for splitter behavior and document usage in the Postgres driver README.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
migrate.go	Adds MigrationSplitter and stepwise execution logic in runMigrations.
migrate_delimiter_test.go	Introduces tests for splitter behavior using stub drivers.
internal/cli/main.go	Adds -migration-splitter flag and wires it into migrate.Migrate.
database/postgres/README.md	Documents using -migration-splitter as an alternative to semicolon splitting.

Comments suppressed due to low confidence (1)

migrate_delimiter_test.go:66

• The expected fragments currently include the delimiter line in the first step, which would mean the delimiter is being executed as SQL. If the delimiter is intended purely as a split boundary, the expectations should exclude it.

	want := []string{
		"CREATE TABLE foo (id INT);\n---\n",
		"DO $$ BEGIN RAISE NOTICE 'hi'; END $$;",
	}
	if !dbDrv.EqualSequence(want) {
		t.Errorf("MigrationSequence = %q, want %q", dbDrv.MigrationSequence, want)
	}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

migrate.go:770

• MigrationSplitter currently triggers splitting whenever it is non-nil, including when it is an empty slice ([]byte{}). In Go, bytes.Split(content, []byte{}) splits between UTF-8 sequences, which would execute the migration byte-by-byte as many Run calls. Also, the loop only skips zero-length fragments; delimiter-adjacent whitespace-only fragments (e.g. blank lines) would be executed and can lead to "empty query" errors in some drivers. Consider treating an empty splitter as disabled and skipping whitespace-only fragments.

				if m.MigrationSplitter != nil {
					content, err := io.ReadAll(migr.BufferedBody)
					if err != nil {
						return err
					}
					for _, step := range bytes.Split(content, m.MigrationSplitter) {
						if len(step) == 0 {
							continue
						}
						if err := m.databaseDrv.Run(bytes.NewReader(step)); err != nil {
							return err
						}
					}

database/postgres/README.md:54

• The docs say the splitter is matched "as a whole line" and instruct users to insert a line containing only ---. However, the CLI wraps the provided value with \n and the implementation matches a literal byte sequence, so the delimiter will not match with CRLF (\r\n) line endings and also won’t match if the delimiter is at the start/end of the file without surrounding \n. Please clarify these constraints (or document LF-only) so users don’t get surprising no-op splits.

Place a line containing only `---` between each step in your migration file. The splitter is matched
as a whole line, and each step is executed sequentially in a separate database call. This works with
any database driver — no URL modification required.

migrate_delimiter_test.go:120

• There’s no test covering the edge case where MigrationSplitter is an empty slice. Without an explicit guard, bytes.Split(content, []byte{}) would execute migrations as many tiny steps. Adding a regression test would help lock down the intended behavior (treat empty as disabled).

func TestMigrationSplitterTrailingDelimiterSkipsEmptyStep(t *testing.T) {
	const content = "stmt1;\n---\nstmt2;\n---\n"
	m, dbDrv := newMigrateWithContent(t, content)
	m.MigrationSplitter = []byte("\n---\n")

	if err := m.Up(); err != nil {
		t.Fatal(err)
	}

	want := []string{"stmt1;", "stmt2;"}
	if !dbDrv.EqualSequence(want) {
		t.Errorf("MigrationSequence = %q, want %q", dbDrv.MigrationSequence, want)
	}
}

[coveralls]

coverage: 54.517% (+0.1%) from 54.412% — sidju:master into golang-migrate:master

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

migrate.go:754

• runMigrations now always io.ReadAll's migr.BufferedBody (even when MigrationSplitter is nil/empty), which forces full buffering in memory and removes the previous streaming behavior of passing the reader directly to the driver. This is a behavior/perf regression for large migrations and contradicts the field doc that says migrations are passed to the driver "as-is" when the splitter is disabled.

			if migr.Body != nil {
				m.logVerbosePrintf("Read and execute %v\n", migr.LogString())
				content, err := io.ReadAll(migr.BufferedBody)
				if err != nil {
					return err

migrate.go:808

• splitMigrationSteps currently rebuilds each step by appending lines into a new buffer, and it preallocates large capacities (len(content) / len(content)-len(currentStep)) for each step. For migrations with many steps this can cause significant extra allocations and memory amplification. You can avoid the copies entirely by returning subslices of the original content while scanning line boundaries.

	steps := make([][]byte, 0, 1)
	currentStep := make([]byte, 0, len(content))

	for _, line := range bytes.SplitAfter(content, []byte("\n")) {
		if bytes.Equal(bytes.TrimRight(line, "\r\n"), splitter) {
			steps = append(steps, currentStep)
			currentStep = make([]byte, 0, len(content)-len(currentStep))

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

internal/cli/main.go:84

• The usage text for -migration-splitter doesn’t show that the flag expects an argument (unlike -prefetch N / -lock-timeout N). Adding a placeholder makes the CLI help clearer.

  -migration-splitter   Split migrations into sequential steps when a whole line equals this string

database/postgres/README.md:54

• The custom splitter docs don’t mention that delimiter lines at the start/end of a file (or consecutive delimiter lines) produce empty steps, which are still executed as separate Run calls. This is observable behavior (see tests) and can surprise users if their driver errors on empty statements.

Place a line containing only `---` between each step in your migration file. The splitter matches when
a whole line exactly equals the token (supports both LF and CRLF line endings), and each step is
executed sequentially in a separate database call.

[sidju]

@dhui
Sorry to ping you directly but I wished to inform you of the following:

• I have created this PR as part of my work, I am thus able and willing to quickly implement any requests you have if given in the next few months.
• I am more than willing to make changes in how this is configured, implemented, naming, etc. to suit your preferences; just say the word.
• If you aren't interested in the change, feel free to close it.