Skip to content

fix(controller): normalize the cache key when reading from the cache - #463

Open
VenishPaneliya wants to merge 1 commit into
psf:masterfrom
VenishPaneliya:fix/cache-key-normalization-on-read
Open

fix(controller): normalize the cache key when reading from the cache#463
VenishPaneliya wants to merge 1 commit into
psf:masterfrom
VenishPaneliya:fix/cache-key-normalization-on-read

Conversation

@VenishPaneliya

Copy link
Copy Markdown

_load_from_cache() looks a cache entry up under the raw request.url, but every write stores it under self.cache_url(request.url), which normalizes the URL and drops the fragment. When those two differ the entry is written to one key and read from another, so the lookup can never succeed.

A fragment is the common way to hit this, because requests keeps it on PreparedRequest.url:

>>> requests.Request("GET", "http://example.com/data#section").prepare().url
'http://example.com/data#section'

so GET http://example.com/data#section stores under http://example.com/data and then looks for http://example.com/data#section.

Effect

Three identical GETs of a plainly cacheable response (Cache-Control: max-age=3600), counting requests that actually reach the server:

URL requested server hits cache key stored
http://127.0.0.1:PORT/data 1 http://127.0.0.1:PORT/data
http://127.0.0.1:PORT/data#section 3 http://127.0.0.1:PORT/data

The entry is sitting in the cache the whole time — it just can't be found. The response is re-fetched on every call, and the entry is rewritten each time rather than reused. This is default behaviour with a plain CacheControl(requests.Session()); no options are involved.

Since a fragment is never sent to the server, both rows are requests for the same resource and should behave identically. RFC 9111 §4 keys a stored response on the request target, and the fragment is not part of it.

Cause

Every other cache access in CacheController already normalizes:

  • cached_request()cache_url = self.cache_url(request.url), used for both cache.delete() calls
  • cache_response()cache_url = self.cache_url(request.url)
  • update_cached_response()cache_url = self.cache_url(request.url)

The read was the only one left out. It normalized too until 11fbcfe ("Unify low-level cache loading code path, and ensure body is always loaded"), which moved the lookups from cached_request()/conditional_headers() into the new _load_from_cache() helper and did not carry the self.cache_url(...) call across. Before that commit the read path read cache_url = self.cache_url(request.url) / self.cache.get(cache_url).

Change

Two lines in _load_from_cache(), so the read is keyed the same way as the writes. The assert is kept, moved onto request.url since that is now what is checked before use.

Serializer.loads() only validates Vary headers, not the URL, so nothing downstream depended on the unnormalized key.

Tests

Added tests/test_regressions.py::TestCacheKeyNormalization::test_fragment_in_url_still_hits_the_cache, using the existing url fixture and cache_60 endpoint.

  • Against unpatched master: failsassert r.from_cache is False
  • With the change: passes
  • Full suite: 103 passed before, 104 passed after — no other test changes behaviour
  • ruff check and ruff format --diff clean on both changed files (pinned 0.14.3)
  • mypy cachecontrol reports the same 10 pre-existing errors before and after (identical set); none in controller.py are new

Writes go through self.cache_url(), which normalizes the URL and drops the
fragment, but _load_from_cache() looked the entry up under the raw
request.url. A request whose URL carries a fragment therefore stored an
entry it could never read back again: the response was re-fetched on every
call while the cache entry sat there unused.

cached_request(), cache_response() and update_cached_response() all key on
self.cache_url(request.url) already, including both cache.delete() calls in
cached_request(), so the read was the only path left unnormalized. It
normalized too until 11fbcfe unified the low-level cache loading code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant