Skip to content

fix(retry): avoid replaying unread body data - #4621

Merged
raykroeker merged 1 commit into
linkerd:mainfrom
cprayer:fix/replay-unpolled-body
Sep 18, 2026
Merged

raykroeker merged 1 commit into
linkerd:mainfrom
cprayer:fix/replay-unpolled-body

Conversation

@cprayer

@cprayer cprayer commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Fixes linkerd/linkerd2#15649

Background

When a retry starts with an empty replay buffer, replay_body remains set while the original body is polled. The first DATA chunk is forwarded and buffered, then replayed on the next poll within the same attempt.

Changes

  • Clear replay_body when the replay buffer is empty and uncapped, before polling the original body
  • Add a regression test verifying that an unpolled body is returned exactly once

Validation

  • Regression test reproduces duplicated bytes before the fix
  • All 24 retry crate tests pass on this branch
  • Applying the fix and regression test to upstream a66af811 passes all 25 retry crate tests
  • Docker reproducer: 10 runs, each with 29–30 duplicated bodies before the fix and zero afterward; consumed-body 503, early 503, and healthy-backend checks also pass

AI tools assisted with the investigation, reproducer, and PR description.

@cprayer
cprayer requested a review from a team as a code owner September 9, 2026 14:52
@cprayer
cprayer force-pushed the fix/replay-unpolled-body branch from efad888 to 858dfc4 Compare September 10, 2026 15:25
@anthoturc

Copy link
Copy Markdown
Contributor

hey @cprayer, could you rebase your branch? it LGTM

Retrying a body that was dropped before its first poll leaves an empty
replay buffer. The retry reads and records the first DATA frame, then
replays that same frame again on its next poll.

Consume the empty replay phase before reading the original body, while
preserving the existing capped-buffer error path.

The new regression test failed with duplicated bytes before the fix.
All 24 retry tests, all-target clippy, and formatting checks pass.

Refs linkerd/linkerd2#15649

Signed-off-by: Taemin Shin <cprayer13@gmail.com>
@cprayer
cprayer force-pushed the fix/replay-unpolled-body branch from 858dfc4 to dfe642f Compare September 18, 2026 16:31
@cprayer

cprayer commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

@anthoturc Thanks! Rebased onto the latest main

@anthoturc

Copy link
Copy Markdown
Contributor

also for other reviewers, i was able to throw a slightly simpler version of the repro that @cprayer provided and confirm that the expected retry behavior is working properly.

kubectl -n replay-repro logs -f deploy/server -c server
2026/09/18 17:57:19 listening on :8080 (h2c)
2026/09/18 17:57:19 req 0: attempt 1, replying 503 without reading the body
2026/09/18 17:57:20 req 0: attempt 2, read 11 bytes "hello world" -- OK
2026/09/18 17:57:25 req 1: attempt 1, replying 503 without reading the body
2026/09/18 17:57:26 req 1: attempt 2, read 11 bytes "hello world" -- OK
2026/09/18 17:57:31 req 2: attempt 1, replying 503 without reading the body
2026/09/18 17:57:32 req 2: attempt 2, read 11 bytes "hello world" -- OK
2026/09/18 17:57:37 req 3: attempt 1, replying 503 without reading the body
2026/09/18 17:57:38 req 3: attempt 2, read 11 bytes "hello world" -- OK
2026/09/18 17:57:43 req 4: attempt 1, replying 503 without reading the body
2026/09/18 17:57:44 req 4: attempt 2, read 11 bytes "hello world" -- OK
2026/09/18 17:57:49 req 5: attempt 1, replying 503 without reading the body
2026/09/18 17:57:50 req 5: attempt 2, read 11 bytes "hello world" -- OK

With the fix applied, we can see that the server is only getting the original body as opposed to the duplicate (which would have lead to 22 bytes being read)

@raykroeker raykroeker left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @cprayer and @anthoturc !

@raykroeker
raykroeker merged commit 0cdcc07 into linkerd:main Sep 18, 2026
18 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Request body duplication when retrying before reading the body

3 participants