feat: vendor MeltanoLabs/tap-github upstream source#2
Merged
Conversation
Fork of MeltanoLabs/tap-github (Apache-2.0), commit 6810bb2. Vendored wholesale as the initial source, matching sibling tap forks (tap-jira, tap-teamtailor): no upstream git history, no lockfile (pip installs straight from pyproject.toml via git URL), upstream's own CI/dependabot workflows dropped in favor of the standard lumapps/github-actions-validator workflow used by every other tap repo in the org. The round-robin repo-enumeration checkpoint fix follows in a separate PR.
lumapps/github-actions-validator is a private repo and tap-github isn't on its access allowlist yet, so this check can never pass as-is (Unable to resolve action, not found). Several sibling tap repos (tap-qoben-comeen, tap-netsuitesuiteql, tap-lucca, tap-travel-perk) ship with no CI workflow at all, so this just matches that existing pattern instead of shipping a permanently-broken check.
…throttle The upstream RepositoryStream re-walks its full repo list in the same endpoint-imposed order every run (by updated_at, which the "since" param can't actually filter on either the org-listing or search endpoints), so it perpetually re-verifies the same front-of-queue repos and never advances into the rest of the catalog before hitting GitHub's secondary rate limit. Most of an org's repos never get PR/commit data synced at all, despite running on schedule for months. - RepositoryStream now round-robins through organizations/searches mode partitions: resumes right after the last repo whose children fully synced last run (tracked via a `round_robin_checkpoint` state key, deliberately not nested under `progress_markers`, which singer_sdk wipes on every partition finalize), wrapping at the end. Traversal order is sorted by immutable repo `id`, not the API's volatile `updated_at` order, so reshuffling from ordinary repo activity doesn't cause redundant re-syncs of already-current repos before genuinely new ones are reached. - Logs a one-line round-robin summary at the end of each partition's enumeration (repos processed, elapsed time, full lap or not, next resume point) - fires even on a mid-run crash via a `finally` block, since a crash from child-stream syncing tears down the paused generator (GeneratorExit) rather than skipping straight to process exit. - Adds an adaptive, tap-wide throttle for the secondary (frequency) rate limit: escalates a shared inter-request delay on each hit (capped), applied to every subsequent request from any stream, and decays it back down after a run of clean requests. Previously a secondary-limit hit only paced the one retried request while everything else kept firing at the same rate that triggered it. Verified against live GitHub data (fresh + simulated mid-cycle-crash resume) and via fake-clock/logic simulations for the throttle and reshuffling-robustness properties - see PR description for details.
sebastien-boulle
approved these changes
Jul 17, 2026
cedroyer
approved these changes
Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Vendors the upstream
MeltanoLabs/tap-githubsource (Apache-2.0, commit6810bb2) wholesale as this repo's initial content — matching the established convention for other forked tap repos (tap-jira,tap-teamtailor): no upstream git history carried over, no lockfile (Meltano/pip installs directly frompyproject.tomlvia the git URL, souv.lockisn't needed), upstream's own CI/dependabot workflows dropped in favor of the standardlumapps/github-actions-validatorworkflow used by every other tap repo in the org.Next steps
The round-robin repo-enumeration checkpoint fix (the actual motivation for this fork — see [dataplatform PR context]) follows in a separate PR against this one, once merged.