fix(models): retire dead model ids, reconcile picker with gateway, fix franklin models pricing#102
Merged
Merged
Conversation
added 2 commits
July 14, 2026 20:43
…he gateway
The in-session /model picker rendered a hand-maintained array and never
talked to the gateway, so it drifted on every upstream change. (`franklin
models` was already live-fetching — two surfaces, one stale.)
Dead ids, verified by probing the chat endpoint (402 = exists, 400 =
unknown; the /api/v1/models listing is NOT authoritative — it hides
working models like grok-3 and gpt-5-nano):
- anthropic/claude-haiku-4.5-20251001 -> 400. Used in 9 places, incl.
compact.ts, which selected it for every Sonnet/GPT-5.4/5.5/Gemini-Pro
session — so long sessions compacted against a model that 400s. The
gateway wants the undated id, which the repo already used elsewhere.
- openai/gpt-5 -> 400 in the tool-use fallback chain; now gpt-5.4.
Both ids stay in pricing.ts/tokens.ts: those are keyed by whatever id a
past session recorded, and the file already keeps retired ids for that.
nvidia/llama-4-maverick retired everywhere. It does not 410 — it answers
200 while the free pool silently serves something else (probed live: it
and qwen3-next both came back as nemotron-3-super-120b-a12b-free). That
made the free "diverse-family" fallback chains fake: primary and fallback
resolved to the same backend, so the fallback could never rescue a turn
the primary failed. Replaced with mistral-nemotron, which verifiably
serves itself. Also dropped it from the vision allowlist, which
contradicted routeRequest()'s own "maverick is text-only" comment —
free vision turns were routed to a text-only model.
Picker now reconciles against the live catalog via reconcilePicker():
rows the gateway stopped listing drop themselves, prices refresh, curated
labels/order stay (gateway names overflow the tuned columns), and it
falls back to the static list when offline. Adds Grok 4.5 — the gateway
calls it xAI's flagship, so `grok` follows it per the bare-alias-tracks-
flagship convention; 4.3 stays pinned (cheaper, 1M ctx vs 4.5's 500K).
Verified: 559/559 tests (5 new reconciliation tests), picker driven
end-to-end, offline fallback exercised, and all 20 models the picker now
offers probed live — every one resolves.
The command modelled every model as { input, output } and printed
`$0.00/M` for anything not token-metered — so the entire image / video /
music / speech catalog (22 paid models) rendered as free, under a "Paid
Models" heading. On a price list, that is the worst direction to be wrong
in. The gateway actually bills seven different ways.
Rewritten on top of gateway-models.ts instead of its own fetch + wrong
local interface — it already types every billing mode, caches 5 minutes,
and serves stale-on-error. Each mode now renders in its own unit, and
unknown modes are still listed with raw pricing rather than a wrong unit:
silently dropping a model is how $0.00/M stayed invisible.
Same root cause, latent: per_character and per_generation are real
gateway billing modes but were missing from BillingMode, so
estimateCostUsd fell through the switch and returned 0 — a paid
ElevenLabs speech/sfx call estimated as free. No speech tool calls it
today, so nothing leaked; fixed before one does.
Also: the Context column had a header but was never populated (the
gateway returns context_window), and the sort keyed off pricing.input,
floating every media model to the top.
Adds the first tests for this file — the precision assertions target the
bug's exact shape ($0.435 must not round to $0.44; a real per-image price
must never render as $0.00). 562/562 pass; output verified live.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reported as "Franklin models still list old models." It was structural, and it turned out to be two bugs.
1. The picker never talked to the gateway
The in-session
/modelpicker rendered a hand-maintained array, so it drifted on every upstream change. (franklin modelswas live-fetching — two surfaces, one stale.)gateway-models.tsalready existed with a live catalog client whose docblock claims it "replaces the hardcoded pricing/model tables" — it didn't; only the media tools imported it.It now reconciles via
reconcilePicker(): rows the gateway stopped listing drop themselves, prices refresh live, curated labels/order stay (gateway names overflow the tuned columns), and it falls back to the static list when offline.Dead ids
Verified by probing the chat endpoint — 402 = exists, 400 = unknown. The
/api/v1/modelslisting is not authoritative: it hides working models (grok-3,gpt-5-nano,opus-4.6,kimi-k2.6are all absent yet alive). A naive diff against it produced five false "dead model" findings that were caught and discarded.anthropic/claude-haiku-4.5-20251001→ 400compact.ts— selected for every Sonnet/GPT-5.4/5.5/Gemini-Pro session, so long sessions compacted against a model that 400sopenai/gpt-5→ 400Both stay in
pricing.ts/tokens.ts— those are keyed by whatever id a past session recorded, and the file already retains retired ids for that.Maverick doesn't 410 — it lies
nvidia/llama-4-maverickleft the catalog but still answers 200, silently served by another model (probed live: it and the free defaultqwen3-nextboth came back asnemotron-3-super-120b-a12b-free). That made the free "diverse-family" fallback chains fake — primary and fallback resolved to the same backend, so the fallback could never rescue a turn the primary failed. Replaced withmistral-nemotron, verified to serve itself. Also removed from the vision allowlist, which contradictedrouteRequest()'s own "maverick is text-only" comment — free vision turns were routed to a text-only model.Adds Grok 4.5 (the gap that prompted the report): the gateway calls it xAI's flagship, so
grokfollows it per the bare-alias-tracks-flagship convention. 4.3 stays pinned — cheaper, and 1M ctx vs 4.5's 500K.2.
franklin modelspriced the whole media catalog at $0.00It modelled every model as
{ input, output }, so the entire image/video/music/speech catalog (22 paid models) printed as$0.00/Munder a "Paid Models" heading. On a price list, that's the worst direction to be wrong in. The gateway bills seven different ways; each now renders in its own unit, and unknown modes are still listed with raw pricing rather than a wrong one.Same root cause, latent:
per_characterandper_generationare real billing modes but were missing fromBillingMode, soestimateCostUsdfell through the switch and returned 0 — a paid ElevenLabs call estimated as free. No speech tool calls it today, so nothing leaked; fixed before one does.Verification
franklin modelsoutput verified live against all 78 models