Commit 53c24f3
authored
Rate-limit circuit breaker for GitHub MCP backend tool calls (#3799)
Four tools simultaneously hit GitHub's 15k req/reset limit with no
gateway-level protection — rate-limited responses propagated directly to
agents, which retried immediately and worsened the storm.
## Changes
### Phase 1: Rate-limit detection + proxy backoff
- **Gateway mode** (`unified.go`): After each `executeBackendToolCall`,
inspects the tool result for GitHub MCP rate-limit error patterns
(`isError: true` + text matching "rate limit exceeded", "secondary rate
limit", "too many requests", etc.) using `isRateLimitToolResult()`.
Extracts reset time from `[rate reset in Ns]` in the error text.
- **Proxy mode** (`handler.go`): `injectRetryAfterIfRateLimited()`
checks `HTTP 429` or `X-Ratelimit-Remaining: 0` after each upstream
forward, injects a `Retry-After` header derived from
`X-Ratelimit-Reset`, and logs at ERROR level.
### Phase 2: Per-backend circuit breaker (`circuit_breaker.go`)
Classic three-state machine per backend server ID:
```
CLOSED ──(N consecutive rate-limit errors)──▶ OPEN
▲ │
│ (probe succeeds) (cooldown/reset elapsed)
│ ▼
└────────────────────────────────── HALF-OPEN
```
- **OPEN** rejects requests immediately with `ErrCircuitOpen` (includes
reset timestamp).
- **HALF-OPEN** allows one probe; re-opens on another rate-limit, closes
on success.
- Transport errors (connection failures) leave the counter unchanged —
only actual rate-limit tool results affect it.
### Configuration
Two new optional per-server fields in TOML/JSON:
```toml
[servers.github]
rate_limit_threshold = 3 # consecutive 429s before opening (default: 3)
rate_limit_cooldown = 60 # seconds OPEN before probing (default: 60)
```
> [!WARNING]
>
> <details>
> <summary>Firewall rules blocked me from connecting to one or more
addresses (expand for details)</summary>
>
> #### I tried to connect to the following addresses, but was blocked by
firewall rules:
>
> - `example.com`
> - Triggering command: `/tmp/go-build151504929/b510/launcher.test
/tmp/go-build151504929/b510/launcher.test
-test.testlogfile=/tmp/go-build151504929/b510/testlog.txt
-test.paniconexit0 -test.timeout=10m0s -test.v=true 64/src/net -trimpath
x_amd64/link -p 5607675/b085/ -lang=go1.25 x_amd64/link -I
/opt/hostedtoolc-errorsas 5607675/b151/ x_amd64/vet --gdwarf-5
rk/XFoNDgDe6LSlTdocker-cli-plugin-metadata` (dns block)
> - Triggering command: `/tmp/go-build1093949210/b510/launcher.test
/tmp/go-build1093949210/b510/launcher.test
-test.testlogfile=/tmp/go-build1093949210/b510/testlog.txt
-test.paniconexit0 -test.timeout=10m0s -test.v=true /version/version.go
/version/version_test.go x_amd64/vet 64/src/runtime/cbash ernal/oidc
ache/go/1.25.8/x--noprofile x_amd64/vet -ato�� UQYbtZDox -buildtags
x_amd64/vet -errorsas -ifaceassert -nilfunc x_amd64/vet` (dns block)
> - Triggering command: `/tmp/go-build1807685167/b514/launcher.test
/tmp/go-build1807685167/b514/launcher.test
-test.testlogfile=/tmp/go-build1807685167/b514/testlog.txt
-test.paniconexit0 -test.timeout=10m0s` (dns block)
> - `invalid-host-that-does-not-exist-12345.com`
> - Triggering command: `/tmp/go-build151504929/b492/config.test
/tmp/go-build151504929/b492/config.test
-test.testlogfile=/tmp/go-build151504929/b492/testlog.txt
-test.paniconexit0 -test.timeout=10m0s -test.v=true
ache/go/1.25.8/x-errorsas andler.go x_amd64/compile` (dns block)
> - Triggering command: `/tmp/go-build1093949210/b492/config.test
/tmp/go-build1093949210/b492/config.test
-test.testlogfile=/tmp/go-build1093949210/b492/testlog.txt
-test.paniconexit0 -test.timeout=10m0s -test.v=true t/config.go
t/driver.go x_amd64/compile -p runtime/pprof -lang=go1.25
x_amd64/compile -uns�� -unreachable=false
/tmp/go-build151504929/b092/vet.cfg x_amd64/vet -c=4 -nolocalimports
-importcfg x_amd64/vet` (dns block)
> - Triggering command: `/tmp/go-build1807685167/b496/config.test
/tmp/go-build1807685167/b496/config.test
-test.testlogfile=/tmp/go-build1807685167/b496/testlog.txt
-test.paniconexit0 -test.timeout=10m0s go1.25.8 -c=4 -nolocalimports
-importcfg /tmp/go-build1747915822/b123/importcfg -embedcfg
/tmp/go-build1747915822/b123/embedcfg -pack` (dns block)
> - `nonexistent.local`
> - Triggering command: `/tmp/go-build151504929/b510/launcher.test
/tmp/go-build151504929/b510/launcher.test
-test.testlogfile=/tmp/go-build151504929/b510/testlog.txt
-test.paniconexit0 -test.timeout=10m0s -test.v=true 64/src/net -trimpath
x_amd64/link -p 5607675/b085/ -lang=go1.25 x_amd64/link -I
/opt/hostedtoolc-errorsas 5607675/b151/ x_amd64/vet --gdwarf-5
rk/XFoNDgDe6LSlTdocker-cli-plugin-metadata` (dns block)
> - Triggering command: `/tmp/go-build1093949210/b510/launcher.test
/tmp/go-build1093949210/b510/launcher.test
-test.testlogfile=/tmp/go-build1093949210/b510/testlog.txt
-test.paniconexit0 -test.timeout=10m0s -test.v=true /version/version.go
/version/version_test.go x_amd64/vet 64/src/runtime/cbash ernal/oidc
ache/go/1.25.8/x--noprofile x_amd64/vet -ato�� UQYbtZDox -buildtags
x_amd64/vet -errorsas -ifaceassert -nilfunc x_amd64/vet` (dns block)
> - Triggering command: `/tmp/go-build1807685167/b514/launcher.test
/tmp/go-build1807685167/b514/launcher.test
-test.testlogfile=/tmp/go-build1807685167/b514/testlog.txt
-test.paniconexit0 -test.timeout=10m0s` (dns block)
> - `slow.example.com`
> - Triggering command: `/tmp/go-build151504929/b510/launcher.test
/tmp/go-build151504929/b510/launcher.test
-test.testlogfile=/tmp/go-build151504929/b510/testlog.txt
-test.paniconexit0 -test.timeout=10m0s -test.v=true 64/src/net -trimpath
x_amd64/link -p 5607675/b085/ -lang=go1.25 x_amd64/link -I
/opt/hostedtoolc-errorsas 5607675/b151/ x_amd64/vet --gdwarf-5
rk/XFoNDgDe6LSlTdocker-cli-plugin-metadata` (dns block)
> - Triggering command: `/tmp/go-build1093949210/b510/launcher.test
/tmp/go-build1093949210/b510/launcher.test
-test.testlogfile=/tmp/go-build1093949210/b510/testlog.txt
-test.paniconexit0 -test.timeout=10m0s -test.v=true /version/version.go
/version/version_test.go x_amd64/vet 64/src/runtime/cbash ernal/oidc
ache/go/1.25.8/x--noprofile x_amd64/vet -ato�� UQYbtZDox -buildtags
x_amd64/vet -errorsas -ifaceassert -nilfunc x_amd64/vet` (dns block)
> - Triggering command: `/tmp/go-build1807685167/b514/launcher.test
/tmp/go-build1807685167/b514/launcher.test
-test.testlogfile=/tmp/go-build1807685167/b514/testlog.txt
-test.paniconexit0 -test.timeout=10m0s` (dns block)
> - `this-host-does-not-exist-12345.com`
> - Triggering command: `/tmp/go-build151504929/b519/mcp.test
/tmp/go-build151504929/b519/mcp.test
-test.testlogfile=/tmp/go-build151504929/b519/testlog.txt
-test.paniconexit0 -test.timeout=10m0s -test.v=true
1.80.0/metadata/metadata.go 5607675/b151/ x_amd64/vet --gdwarf-5 --64 -o
x_amd64/vet 5607�� 1.10.2/active_help.go 1.10.2/args.go x_amd64/vet
--gdwarf-5 --64 -o x_amd64/vet` (dns block)
> - Triggering command: `/tmp/go-build1093949210/b519/mcp.test
/tmp/go-build1093949210/b519/mcp.test
-test.testlogfile=/tmp/go-build1093949210/b519/testlog.txt
-test.paniconexit0 -test.timeout=10m0s -test.v=true submodules | hea-p
-buildtags x_amd64/vet -errorsas -ifaceassert -nilfunc x_amd64/vet
n-me�� g_.a -buildtags 64/pkg/tool/linu-nolocalimports -errorsas
ernal/config -nilfunc
64/pkg/tool/linu/tmp/go-build1093949210/b226/_testmain.go` (dns block)
> - Triggering command: `/tmp/go-build1807685167/b523/mcp.test
/tmp/go-build1807685167/b523/mcp.test
-test.testlogfile=/tmp/go-build1807685167/b523/testlog.txt
-test.paniconexit0 -test.timeout=10m0s 9560�� g_.a /sys/fs/cgroup
ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet JwJi0zHsZ ernal/config
x_amd64/vet ache/go/1.25.8/x-buildtags 9560�� ZjxfQrmK0 x_amd64/vet
x_amd64/vet rg/x/net@v0.52.0bash cfg 64/pkg/tool/linu--version
x_amd64/vet` (dns block)
>
> If you need me to access, download, or install something from one of
these locations, you can either:
>
> - Configure [Actions setup
steps](https://gh.io/copilot/actions-setup-steps) to set up my
environment, which run before the firewall is enabled
> - Add the appropriate URLs or hosts to the custom allowlist in this
repository's [Copilot coding agent
settings](https://github.com/github/gh-aw-mcpg/settings/copilot/coding_agent)
(admins only)
>
> </details>7 files changed
Lines changed: 1081 additions & 1 deletion
File tree
- internal
- config
- proxy
- server
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
215 | 215 | | |
216 | 216 | | |
217 | 217 | | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
218 | 230 | | |
219 | 231 | | |
220 | 232 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
| 11 | + | |
10 | 12 | | |
11 | 13 | | |
12 | 14 | | |
| |||
350 | 352 | | |
351 | 353 | | |
352 | 354 | | |
| 355 | + | |
| 356 | + | |
353 | 357 | | |
354 | 358 | | |
| 359 | + | |
355 | 360 | | |
356 | 361 | | |
357 | 362 | | |
| |||
416 | 421 | | |
417 | 422 | | |
418 | 423 | | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
419 | 484 | | |
420 | 485 | | |
421 | 486 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
0 commit comments