fix(controller): normalize the cache key when reading from the cache - #463
Open
VenishPaneliya wants to merge 1 commit into
Open
fix(controller): normalize the cache key when reading from the cache#463VenishPaneliya wants to merge 1 commit into
VenishPaneliya wants to merge 1 commit into
Conversation
Writes go through self.cache_url(), which normalizes the URL and drops the fragment, but _load_from_cache() looked the entry up under the raw request.url. A request whose URL carries a fragment therefore stored an entry it could never read back again: the response was re-fetched on every call while the cache entry sat there unused. cached_request(), cache_response() and update_cached_response() all key on self.cache_url(request.url) already, including both cache.delete() calls in cached_request(), so the read was the only path left unnormalized. It normalized too until 11fbcfe unified the low-level cache loading code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_load_from_cache()looks a cache entry up under the rawrequest.url, but every write stores it underself.cache_url(request.url), which normalizes the URL and drops the fragment. When those two differ the entry is written to one key and read from another, so the lookup can never succeed.A fragment is the common way to hit this, because
requestskeeps it onPreparedRequest.url:so
GET http://example.com/data#sectionstores underhttp://example.com/dataand then looks forhttp://example.com/data#section.Effect
Three identical GETs of a plainly cacheable response (
Cache-Control: max-age=3600), counting requests that actually reach the server:http://127.0.0.1:PORT/datahttp://127.0.0.1:PORT/datahttp://127.0.0.1:PORT/data#sectionhttp://127.0.0.1:PORT/dataThe entry is sitting in the cache the whole time — it just can't be found. The response is re-fetched on every call, and the entry is rewritten each time rather than reused. This is default behaviour with a plain
CacheControl(requests.Session()); no options are involved.Since a fragment is never sent to the server, both rows are requests for the same resource and should behave identically. RFC 9111 §4 keys a stored response on the request target, and the fragment is not part of it.
Cause
Every other cache access in
CacheControlleralready normalizes:cached_request()—cache_url = self.cache_url(request.url), used for bothcache.delete()callscache_response()—cache_url = self.cache_url(request.url)update_cached_response()—cache_url = self.cache_url(request.url)The read was the only one left out. It normalized too until 11fbcfe ("Unify low-level cache loading code path, and ensure body is always loaded"), which moved the lookups from
cached_request()/conditional_headers()into the new_load_from_cache()helper and did not carry theself.cache_url(...)call across. Before that commit the read path readcache_url = self.cache_url(request.url)/self.cache.get(cache_url).Change
Two lines in
_load_from_cache(), so the read is keyed the same way as the writes. Theassertis kept, moved ontorequest.urlsince that is now what is checked before use.Serializer.loads()only validatesVaryheaders, not the URL, so nothing downstream depended on the unnormalized key.Tests
Added
tests/test_regressions.py::TestCacheKeyNormalization::test_fragment_in_url_still_hits_the_cache, using the existingurlfixture andcache_60endpoint.master: fails —assert r.from_cacheisFalseruff checkandruff format --diffclean on both changed files (pinned 0.14.3)mypy cachecontrolreports the same 10 pre-existing errors before and after (identical set); none incontroller.pyare new