You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
CLAUDE.md's tiering (test/manifold pure, test/integration needs a real engine, test/smoke one-shot assumptions about the outside world, excluded from mix test) and config/test.exs's hermeticity depend, for the local model, on a trick that only works for local hosting: pointing model_path at a file that doesn't exist. That cleanly parks Llama.Server in :no_model with no network call at all, which is what makes test/integration/no_model_test.exs and the whole no-model fallback path testable without touching the network.
An external-provider client has no equivalent filesystem trick — the "model missing" analog would otherwise be "make a real HTTP call to a real API," which mix test must never do by default.
What
Decide, and document in CLAUDE.md alongside the existing tiering rules:
Whether the external client is ever exercised by test/manifold or test/integration at all, or whether correctness tests only ever run against the local/no-model paths already built (the Model behaviour from Extract a Manifold.Model behaviour so local and external providers are pluggable #9 makes this easy: a test double implementing the behaviour, not a real HTTP client, stands in for "external model" in unit tests).
Whether one narrow test/smoke test is warranted to pin the one assumption that matters per provider — e.g. "provider X's streaming SSE format is Y" — worded as an ASSUMPTION: per the existing smoke-test convention, run rarely, never as part of mix test.
How credentials for that smoke test (if any) are supplied without landing a real API key in CI config.
Depends on
#9 (a behaviour is what makes a test double possible in the first place) and the external-client issue (need to know what there is to test).
Why
CLAUDE.md's tiering (
test/manifoldpure,test/integrationneeds a real engine,test/smokeone-shot assumptions about the outside world, excluded frommix test) andconfig/test.exs's hermeticity depend, for the local model, on a trick that only works for local hosting: pointingmodel_pathat a file that doesn't exist. That cleanly parksLlama.Serverin:no_modelwith no network call at all, which is what makestest/integration/no_model_test.exsand the whole no-model fallback path testable without touching the network.An external-provider client has no equivalent filesystem trick — the "model missing" analog would otherwise be "make a real HTTP call to a real API," which
mix testmust never do by default.What
Decide, and document in CLAUDE.md alongside the existing tiering rules:
test/manifoldortest/integrationat all, or whether correctness tests only ever run against the local/no-model paths already built (theModelbehaviour from Extract aManifold.Modelbehaviour so local and external providers are pluggable #9 makes this easy: a test double implementing the behaviour, not a real HTTP client, stands in for "external model" in unit tests).test/smoketest is warranted to pin the one assumption that matters per provider — e.g. "provider X's streaming SSE format is Y" — worded as anASSUMPTION:per the existing smoke-test convention, run rarely, never as part ofmix test.Depends on
#9 (a behaviour is what makes a test double possible in the first place) and the external-client issue (need to know what there is to test).