Skip to content

Define the test-hermeticity strategy for external-provider clients #14

Description

@vbergeron

Why

CLAUDE.md's tiering (test/manifold pure, test/integration needs a real engine, test/smoke one-shot assumptions about the outside world, excluded from mix test) and config/test.exs's hermeticity depend, for the local model, on a trick that only works for local hosting: pointing model_path at a file that doesn't exist. That cleanly parks Llama.Server in :no_model with no network call at all, which is what makes test/integration/no_model_test.exs and the whole no-model fallback path testable without touching the network.

An external-provider client has no equivalent filesystem trick — the "model missing" analog would otherwise be "make a real HTTP call to a real API," which mix test must never do by default.

What

Decide, and document in CLAUDE.md alongside the existing tiering rules:

  • Whether the external client is ever exercised by test/manifold or test/integration at all, or whether correctness tests only ever run against the local/no-model paths already built (the Model behaviour from Extract a Manifold.Model behaviour so local and external providers are pluggable #9 makes this easy: a test double implementing the behaviour, not a real HTTP client, stands in for "external model" in unit tests).
  • Whether one narrow test/smoke test is warranted to pin the one assumption that matters per provider — e.g. "provider X's streaming SSE format is Y" — worded as an ASSUMPTION: per the existing smoke-test convention, run rarely, never as part of mix test.
  • How credentials for that smoke test (if any) are supplied without landing a real API key in CI config.

Depends on

#9 (a behaviour is what makes a test double possible in the first place) and the external-client issue (need to know what there is to test).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions