Repository navigation
ci: shard the Trino, Presto and Spark e2e jobs - #18
Merged
Merged
Conversation
…each Their jobs took 45 to over 80 minutes: each of their queries pays a fixed cost (planning, scheduling, polling) that dwarfs the work of a test's few rows. SYNALOG_E2E_SHARD=i/n runs one n-th of the selected tests, split by fixture (its tests share its rows) with a hash the same on every machine; CI runs each of the three engines in three parallel jobs, each with its own server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trino, Presto and Spark e2e jobs took 45 to over 80 minutes: every query on those engines pays a fixed cost (planning, scheduling tasks, polling for results) that dwarfs the work of a test's few rows, and each engine runs about 15,000 tests.
SYNALOG_E2E_SHARD=i/n(tests/e2e/conftest.py) runs one n-th of the selected tests, split by fixture or program, so the tests that share a fixture's rows stay in one shard; the split hashes the name (CRC-32), the same on every machine. An invalid shard is refused.🤖 Generated with Claude Code