Skip to content

Stop integration tests calling live upstream providers #284

Description

@ALeonard9

Story

As Adam-as-operator, I want the integration tests to stop calling live upstream providers, so that a build fails only when the code is wrong and not when TMDB has a bad minute.

Acceptance Criteria

  • tests/integration/router_visibility_viewer_test.py mocks the upstream providers instead of creating catalog entries from real identifiers, following the pattern already used in router_movies_test.py
  • Every other integration test that reaches a provider is found and given the same treatment — the audit is part of the work, not just this one file
  • A guard in tests/conftest.py fails any test that attempts outbound HTTP, so the next unmocked call is caught at authoring time rather than months later during a profiling pass
  • The guard names the offending host in its failure message
  • The full suite passes with the machine's network unavailable
  • Coverage is unchanged — no test is deleted or weakened to make it hermetic

Context

Found while profiling the suite after Phases 0–1 merged (#283). This is a correctness and reliability problem; the speed is a symptom.

tests/integration/router_visibility_viewer_test.py (added by #277) mocks nothing — grep -c "patch\|mock" returns 0, where router_movies_test.py has 11 guards. Its profile fixture calls _stock_every_shelf, which creates eight catalog entries per test using real identifiers (tt0113277, 9780441172719, igdb: 1111), so each of its 21 tests reaches TMDB, TVMaze, Open Library and IGDB for real.

Measured:

28.57s setup  router_visibility_viewer_test.py::test_anonymous_sees_only_public_shelves

That is one test. The file accounts for roughly 350 of the suite's 405 seconds.

Two consequences, in order of importance:

  1. CI is flaky by construction. A provider outage, a rate limit, or a changed upstream response fails a build for reasons unrelated to the change under review. Worse, it can pass misleadingly — a test asserting on enriched metadata is really asserting on whatever the provider returned today.
  2. It is slow.

The tests themselves are good — the viewer matrix in that file is the strongest coverage in the sharing epic. Nothing about it needs weakening; it just needs the providers stubbed.

The conftest guard is the part that stops this recurring. This went unnoticed through a full implementation and review cycle because a green suite says nothing about how it was green.

Estimate

  • Recommended model: Sonnet 5 — a mechanical change with an established in-repo pattern to copy, plus a contained conftest addition.
  • Human effort: S — one review pass confirming coverage did not shift.

Notes for Implementation

Prefer a single autouse fixture that patches at the provider-client boundary over per-test decorators; 21 tests each carrying their own @patch stack is how the next file ends up unmocked too.

Check router_visibility_test.py and router_follows_test.py as well — neither contains the string patch, and both create catalog rows.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:infraHosting, deploy, environments, CI/CDchoreMaintenance work with no user-visible behavior changepriority:p2Do soon

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions