perf(source): build migrations index lazily to avoid quadratic startup - #1424
Open
YuheiTakagawa wants to merge 1 commit into
Open
perf(source): build migrations index lazily to avoid quadratic startup#1424YuheiTakagawa wants to merge 1 commit into
YuheiTakagawa wants to merge 1 commit into
Conversation
Migrations.Append rebuilt and re-sorted the whole index on every call, making source driver initialization O(n^2 log n) in the number of migration files. All in-memory source drivers (file, iofs, aws_s3, github, ...) call Append in a loop at Open/Init time, so startup cost grew quadratically: 81ms at 1,000 migrations, 2.2s at 5,000 and 11.3s at 10,000 on an i9-9900K. Append now only marks the index dirty; the index is rebuilt at most once, on the first read access (First/Prev/Next). No exported API changes. BenchmarkAppend5000 drops from ~2.2s/op to ~2ms/op.
|
coverage: 54.911% (+0.5%) from 54.412% — YuheiTakagawa:perf/source-lazy-index-build into golang-migrate:master |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
source.Migrations.AppendcallsbuildIndex()on every append, which rebuilds and re-sorts the entire version index each time. Since every in-memory source driver (file,iofs,aws_s3,github,bitbucket,go_bindata, ...) callsAppendin a loop when the driver is opened/initialized, loading n migration files costs O(n² log n).For long-lived projects this makes startup noticeably slow before any migration is even executed. Large migration counts are a real use case (see e.g. #1412, which fixes S3 listing for >1000 migrations).
Fix
Appendnow only marks the index as dirty. The index is rebuilt at most once, lazily, on the first read access (First, orPrev/NextviafindPos). This keeps the total cost at O(n log n) regardless of how many files are appended.Append's duplicate-rejection behavior is unchanged.Migrations: drivers build it once at init and read afterwards, same as before.Benchmarks
Added
BenchmarkAppend{100,1000,5000}(append n up/down pairs, then one read so the index build is included). Same benchmark code against both implementations (i9-9900K, Windows, go1.25; the n=10,000 row was measured with an equivalent out-of-tree benchmark):Before scales ~quadratically (4x files → ~16x time); after is linearithmic.
Verification
go test ./source/— passes (existing tests + new interleaved append/read regression test)go vet ./source/— cleangolangci-lint run ./source/— no new issues