fix(postgres): roll back a transaction cancelled during BEGIN - #4394
Open
rubenfiszel wants to merge 1 commit into
Open
fix(postgres): roll back a transaction cancelled during BEGIN#4394rubenfiszel wants to merge 1 commit into
rubenfiszel wants to merge 1 commit into
Conversation
`PgTransactionManager::begin` raises `transaction_depth` only after the BEGIN round trip returns, but `start_rollback` -- which both its own `Rollback` drop guard and `Transaction`'s drop guard call -- is a no-op while that depth is zero. A future cancelled during the await therefore queues nothing, and the session is left inside a transaction. `Floating::return_to_pool` then validates the connection with `ping()`, which for Postgres is a bare `wait_until_ready`: it drains the `ReadyForQuery` but never inspects its transaction-status byte, so the connection is judged healthy and handed to the next borrower. Their statements run inside the stale transaction and hold its locks, and the first error turns the session into `idle in transaction (aborted)`, after which every unrelated query on that connection fails with 25P02 until `max_lifetime` recycles it. Claim the depth before the round trip and unwind it if the BEGIN did not take, so the drop guards have something to act on. The queued ROLLBACK is written to the same buffer as the BEGIN and so is always flushed after it. Closes transact-rs#4393. Refs transact-rs#2054, transact-rs#2819, transact-rs#3980.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4393.
The problem
PgTransactionManager::beginraisestransaction_depthonly after theBEGINround trip:start_rollbackis a no-op while that depth is zero, and it is what both guards call — theRollbackguard added in #2819 and theTransaction-level one added in #3980. So a future cancelled in that window queues nothing and the session stays inside a transaction.Floating::return_to_poolthen validates the connection withraw.ping(), which for Postgres is a barewait_until_ready: it drains theReadyForQuerybut never inspects its transaction-status byte. The connection is judged healthy and handed to the next borrower, whose statements run inside the stale transaction and hold its locks. The first error turns the session intoidle in transaction (aborted), after which every unrelated query on that connection fails with25P02— and it never self-heals, because the depth is still zero, so no later drop queues a rollback either. Onlymax_lifetimeclears it.In production this turned a transient database slowdown into a ~20 minute outage: one poisoned connection, reused hundreds of times a second, surfacing the same
25P02at a dozen unrelated call sites at once.#2054 covers this and was closed by #2057, which fixed SQLite. #3980 restructures
Transaction::beginso a cancelled caller triggers the drop guard, but for Postgres that still funnels into the same depth-gatedstart_rollback, so it does not reach this case. Reproduced on both 0.8.6 and 0.9.0.The change
Claim the depth before the round trip and unwind it if the
BEGINdid not take, so the guards have something to act on.The queued
ROLLBACKgoes into the same write buffer as theBEGIN, and is flushed after it — so the statement it rolls back has always been sent first. That also holds for the savepoint case: a cancelled nestedbeginnow queuesROLLBACK TO SAVEPOINT _sqlx_savepoint_{depth}for the savepoint it just queued, rather than the enclosing one.Test
it_rolls_back_a_transaction_cancelled_during_beginintests/postgres/postgres.rs. It cancels abegin_with("BEGIN; SELECT pg_sleep(2);")— a plainBEGINanswers too quickly to cancel reliably, and the sleep widens the same round trip — then asserts the session is back toidlerather thanidle in transaction.Verified against a real server: the test fails on
mainwithleft: Some("idle in transaction"), and passes with the change. I also ran the whole--test postgressuite before and after: the failure set is identical apart from this test (4 pre-existing failures in my environment, from fixtures my database lacks), so nothing else changes behaviour — includingit_can_work_with_failed_transactions,it_can_work_with_nested_transactionsandit_can_fail_and_recover, which all pass.Known trade-off
A
BEGINthat is rejected by the server now queues aROLLBACKon a session that never entered a transaction, which Postgres answers withWARNING: there is no transaction in progress. That seemed clearly preferable to leaking the transaction, but if you would rather not change that path, the alternative is to track "BEGIN sent, outcome unknown" in a separate flag instead of reusing the depth.A cheaper fix you may prefer
Postgres reports transaction status in every
ReadyForQuery, and it is already decoded intoPgConnection::transaction_status.in_transaction()ispub(crate), and the publicConnection::is_in_transaction()returns the client-side depth — precisely the value that is wrong here. Havingreturn_to_poolconsult the server-reported status would catch this and any other cancellation that leaves a connection dirty, at no extra round trip. Happy to take the PR in that direction instead if you prefer it.