Repository navigation
Stay under Trakt's rate limit for large libraries - #219
Merged
Merged
Conversation
…s read A show past the 60-show progress budget took its last aired episode from the last episode the user watched. For a show with episodes still to watch that frontier is behind the real one, so the recent calendar counted episodes Trakt's aired_episodes already counted, flagged the show as having new episodes, and the Up Next snapshot read its season tree and specials progress to place it. On a large library that was two reads for every backlog show with an episode in the last 33 days, on every launch. The last watched episode is the last aired one only when nothing is left to watch. For any other show without a progress read the frontier is now null, which the calendar reconcile already treats as unknown and leaves alone. Caught-up shows keep their frontier, so a new season of a show the user finished is still found and placed.
Each calendar window read the hidden and dropped lists itself, so a first sync read them three times: once for the library and once for each of the two calendar windows. Stopping a show in the app also re-read both calendar windows, with their hidden lists, only to filter one show out. The library already marks every hidden or dropped show, and every show on the user's calendar is in the library. The calendar read now returns only its episodes. The Calendar tab, Up Next's coming-up list, episode reminders and the recently-aired reconcile filter by the library's hidden flags through a new useHiddenShowIds hook, and stopping a show no longer invalidates the calendars. A show stopped or dropped on another device now leaves the calendar as soon as the library reloads, instead of when the calendar's hour-long cache next refreshes.
Search's add-to-watchlist button kept its own query over the shows watchlist, so a first sync read every watchlist page twice: once for the library and once for that query. The button already treated every library show as listed, and the library holds every watchlisted show, so the second read added nothing. Show membership now comes from the library alone. The movies watchlist query stays, because the movie library is not loaded until the Library tab asks for it.
The Trakt client joined identical GETs only once a request had been sent. Every pooled read takes one of six slots before it reaches the client, so when two screens asked for the browse rails while Trakt was slow, the second screen's reads that were still queued for a slot found the first ones finished and sent their own: six requests for four rails. Identical pooled reads now share one promise from the moment they are asked for, so a caller waiting for a slot joins the read already queued or in flight. This replaces the client's own in-flight map, which only covered reads that had already left.
When the recent calendar shows a new episode of a show the user had caught up on, Up Next reads that show's season tree to find the next episode. The trees were not persisted, so every launch read them all again, two requests each, even when the persisted library and calendar were fresh and nothing had changed on Trakt. A season tree is now persisted for a show in the library, by the same rule as the show art a library card paints. It keeps its one-hour content stale time, so a launch within the hour reuses it and a later one refreshes it.
The scenario replay now fails if Trakt's rate limit refuses any read, if a scenario that writes nothing fetches the same URL twice, or if a warm launch on a fresh cache reads anything beyond the activities poll and the Search tab's four rails. The calendar hidden-flag change is no longer listed as pending in the known-over table. The calendar data no longer carries hidden show ids, so the persisted cache buster is bumped and older caches are discarded.
Show watchlist membership now comes from the library, so the watchlist query, its runtime loader and Search's add and remove paths only ever handle movies. They are narrowed to movies, and the unused shows paths are removed.
The calendar filters stopped and dropped shows by the library's flags, but with "every watchlist episode" on it did not wait for the library, so a hidden show could show until the library arrived. useCalendarFocus now owns both sets the calendar needs from the library, played and hidden, and holds the Calendar on its loading plates until the library answers, whatever that setting is. If the library read fails, the calendar shows every row unfiltered, as it already did for played shows. Up Next's coming-up list takes the hidden set from the same hook, and reminders take it from the library snapshot they already read, so the separate useHiddenShowIds hook is gone.
Pooled reads are shared only between callers asking for the same thing, a failure reaches every caller of a shared read, and the next call after it reads afresh. These replace the client-level coalescing test removed when the sharing moved into the pool.
A show's season tree, progress and episode details each merge in the user's per-episode watched state, but an activity change only refetched the library and history. With season trees now kept across launches, a tree could show an episode as unwatched after it was watched on another device. Changes to watched history, progress or hidden shows now invalidate every show's progress, season tree and episode queries, so the ones on screen are read again. Season trees are now persisted only while a screen uses them: Up Next placing a card, or a detail screen that is open. A detail visit no longer adds its tree to the persisted cache for good. The replay gains two scenarios. One opens a show, writes a play for it straight to the fake as another device would, relaunches warm, and checks that the tree is read again and shows the episode watched. The other checks that a backlog show past the progress budget stays out of Up Next even when an episode aired lately, and that Library counts its episodes left. The cache witness is reseeded; no persisted shape changed.
The two tests that hold the library read and check the Calendar waits for it now share one helper instead of repeating the same steps.
Pull request footprint
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Trakt allows 500 GETs per user per 5 minutes (https://developer.trakt.tv/docs/rate-limiting), and the Trakt fake in
packages/core/test/trakt-fake/enforces it. The heavy-library replay inpackages/core/test/harness/scenario-budget.test.tsxmeasured:Each 429 pauses every read for Trakt's Retry-After, which is how large libraries ended up with multi-minute stalls. On top of that, the first sync read every watchlist-shows page twice, hidden/dropped three times and
last_activitiestwice, and two screens asking for the browse rails while Trakt was slow cost 6 GETs instead of 4.Root causes
lastAiredKeyinpackages/core/src/data/trakt/library.tstook a show's last aired episode from the last episode the user watched whenever the show was past the 60-show progress budget. For a show with episodes still to watch, that frontier is behind the real one.reconcileRecentlyAired(domain/recently-aired.ts) then counted calendar episodes Trakt had already counted,needsNextEpisodeflagged the show, anduseLibrarySnapshotread its season tree throughloadShowSeasons: two GETs per show (/shows/:id/seasonsand/shows/:id/progress/watchedwith specials). In the heavy library that was 282 shows, so 564 GETs on every launch.createQueryCachePolicyinruntime/query-cache.tspersisted only a library show's art, never its season tree. A launch with a fresh persisted cache and unchanged activities still read all the trees again.loadCalendarinapp/create-runtime.tsread both lists for each of the two calendar windows, on top of the library's own read inread-budget.ts. Stopping a show in the app also invalidated both calendars, which read them again.useWatchlistAddmounted a watchlist query, key["watchlist", "shows"], over the same/sync/watchlist/showsURL the library reads under["library"]. The keys differ, so TanStack Query could not dedupe them.last_activitiesread twice. The second read was the 60-second interval poll firing while every read waited out a 429 pause. It goes away with the 429s.TraktClient.getjoined identical GETs only once a request had been sent. Pooled reads (pooled-endpoints.ts) first wait for one of six slots inwithReadRateRetry. The second screen's reads still queued for a slot found the first ones finished and sent their own.Fixes
last_activitiesnow invalidates every show's progress, season tree and episode queries (data/query-invalidation.ts). Each of these merges the user's per-episode watched state, so a tree kept across launches can no longer show an episode watched on another device as unwatched. Only the trees on screen are read again.useCalendarFocusowns the played and hidden sets. The Calendar stays on its loading plates until the library answers, whatever the "every watchlist episode" setting is. If the library read fails, the Calendar shows every row unfiltered, as it already did for played shows.loadMovieWatchlistIds) and Search's add and remove paths now handle movies only, because the movie library loads only with the Library tab's Movies segment.Behaviour changes to know about
After
Warm launch, fresh cache and unchanged activities, per path:
/sync/last_activities/shows/:id/progress/watched/shows/:id/seasons/shows/trending,/shows/popular,/movies/trending,/movies/popularThe four rails are the Search tab's own read when it mounts, and do not depend on the activities diff. They go away with the planned change that loads the Search tab's data on first focus, which the known-over table already lists.
What the first sync still reads: 60 progress reads for the most recently watched backlog shows (the policy budget), 40 season trees (80 GETs) for caught-up shows that have a new episode, 100 show reads for queue row art, and 36 list pages. The known-over table names the planned changes that cover the art and the trees.
Tests
scenario-budget.test.tsx:loadBrowsecalls made while the rails are held cost 4 requests, and a later call costs 4 more.pooled-endpoints.test.ts:trakt-library.test.ts: a backlog show without a progress read has no aired frontier.query-cache.test.ts: a library show's season tree is persisted only while a screen observes it, and an unseen show's tree never is.query-invalidation.test.ts: the watched and hidden activity changes route to the show trees.library-snapshot.test.tsx: a show hidden in the library is left out of the recently-aired reconcile.calendar-read.test.ts: a dropped show is filtered by the library's flag.watchlist-add.test.tsx: a failed remove restores the library, the movie library and the movie watchlist.calendar.test.tsx:up-next.test.tsx: a hidden show is left off Up Next's coming-up list.hide-show.test.tsx: stopping a show does not invalidate the calendars.Every new or changed assertion was checked by reverting the change it covers and confirming that the test fails.
Size
Coverage-Rationale:
useCalendarFocus(lines 20-35) is exercised through the native Jest suite, which the core coverage report does not include:calendar.test.tsxcovers played versus watchlist-only shows, stopped and dropped shows left off, the hold until the library answers (with "Every watchlist episode" off and on), and unfiltered rows when the library cannot be read;settings.test.tsxcovers the switch revealing an episode.