Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions cmd/router/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -586,13 +586,20 @@ func main() {
// registered, so the Service's nil-check disables Tier-3 correctly (a
// typed-nil concrete pointer would defeat it).
var compactionSz proxy.CompactionSummarizer
if client, ok := providerMap[handoverProviderName]; ok {
ps := proxy.NewProviderSummarizer(client, handoverModel, handoverTimeout)
client, handoverProviderRegistered := providerMap[handoverProviderName]
// The summarizer emits Anthropic Messages bodies directly, with no
// translator in front of it.
handoverFamilyOK := providers.FamilyFor(handoverProviderName) == providers.FamilyAnthropic
switch {
case !handoverProviderRegistered:
logger.Info("Handover summarizer disabled (provider not registered); switch turns will preserve full history instead", "requested_provider", handoverProviderName)
case !handoverFamilyOK:
logger.Warn("Handover summarizer disabled (provider does not speak the Anthropic Messages format); switch turns will preserve full history instead", "requested_provider", handoverProviderName)
default:
ps := proxy.NewProviderSummarizer(client, handoverProviderName, handoverModel, handoverTimeout)
summarizer = ps
compactionSz = ps
logger.Info("Handover summarizer wired", "provider", handoverProviderName, "model", handoverModel, "timeout_ms", handoverTimeout.Milliseconds())
} else {
logger.Info("Handover summarizer disabled (provider not registered); switch turns will preserve full history instead", "requested_provider", handoverProviderName)
}
compactionPct := parseEnvFloat("ROUTER_COMPACTION_PCT", proxy.DefaultCompactionTriggerPct)

Expand Down
10 changes: 10 additions & 0 deletions docs/CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,9 @@ Set `DATABASE_URL` directly, or compose it from the individual vars:
| `ROUTER_SUBAGENT_MODEL` | *(none)* | Route Claude Code Task-tool sub-agent turns to a distinct model, independent of `ROUTER_HARD_PIN_MODEL` — e.g. a local/self-hosted OpenAI-compatible model (point `OPENROUTER_BASE_URL` at your local server) while the main loop keeps using Anthropic/whatever the scorer picks. Requires `ROUTER_SUBAGENT_PROVIDER`; either alone is ignored. Takes effect regardless of `ROUTER_HARD_PIN_EXPLORE`, but the HMM strategy keeps its own sub-agent handling and isn't affected. |
| `ROUTER_SUBAGENT_PROVIDER` | *(none)* | Pair with `ROUTER_SUBAGENT_MODEL`. |
| `ROUTER_TRANSLATION_COMPATIBILITY_MODE` | `shadow` | Translation representability rollout: `off` disables broad filtering, `shadow` records candidate exclusions without changing routes, and `enforce` makes declared semantic requirements hard routing constraints. Native-only safety paths (such as unsupported Responses tool unions and native Gemini ingress) remain protected unless mode is `off`. |
| `ROUTER_HANDOVER_PROVIDER` | `anthropic` | Provider that summarizes prior history when the planner switches models, and for the compaction cascade's structured summary. Must be registered and speak the Anthropic Messages format, or summarization stays off and switch turns forward full history. |
| `ROUTER_HANDOVER_MODEL` | `claude-haiku-4-5` | Model used for that summary. |
| `ROUTER_HANDOVER_TIMEOUT_MS` | `8000` | Deadline for the summary call. On timeout the full prior history is forwarded unchanged. |
| `ROUTER_COMPACTION_PCT` | `0.85` | Fraction of the largest eligible model's context window at which the proactive compaction cascade engages (clear old tool results → structured summary → trim). Range `(0,1]`; `0` disables compaction (over-window requests then 413). Mirrors Claude Code's ~0.85 auto-compact trigger. |
| `ROUTER_ONNX_ASSETS_DIR` | `/opt/router/assets` | Directory containing `model.onnx` + `tokenizer.json`. |
| `ROUTER_ONNX_LIBRARY_DIR` | *(system default)* | Path to `libonnxruntime` (e.g. `/opt/homebrew/lib` on Apple Silicon). |
Expand Down Expand Up @@ -152,6 +155,13 @@ permitted binding left is forced normally and served through that binding. A
live session whose forced pin is later excluded fails the same way rather than
quietly reverting to automatic routing — clear it with `/unforce-model`.

The router's own synthetic calls obey the same lists. The handover and
compaction summarizers ship the entire prior conversation upstream, so an
excluded `ROUTER_HANDOVER_PROVIDER` (or `ROUTER_HANDOVER_MODEL`) means no
summary is requested at all: a switch turn forwards full history and
compaction falls through to trimming, exactly as on a summarizer error.
Compaction's window-aware model choice skips excluded models too.

Excluding every provider that serves the models you route to leaves requests
with nowhere to go (HTTP 503 from the scorer), so exclude deliberately.

Expand Down
2 changes: 2 additions & 0 deletions internal/proxy/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ The per-action flow is more than "scorer → dispatch". Pinned session, planner

The provider-backed `Summarizer` implementation for handover lives in [`handover.go`](handover.go); the inner-ring `handover` package only defines the contract. On summarizer timeout or error, proxy keeps the full prior history unchanged (it does **not** trim) — a pricier switch action beats silently dropping the conversation the switched-to model needs.

A summary carries the whole prior conversation, so it goes through [`gateSummarizerCall`](summarizer_policy.go) first: excluded providers/models and the tenant credential boundary refuse it outright, and a refusal takes the same full-history / trim fallback as a summarizer error. `Summarizer` therefore exposes `Provider()`/`Model()` — the identity credential resolution and exclusions key off.

## Proactive context-window compaction

`ProxyMessages` / `ProxyOpenAIChatCompletion` call [`maybeCompact`](compaction.go) **before** routing so an over-long session is compacted rather than dead-ending in the scorer with no eligible provider. It engages when the estimate reaches `ROUTER_COMPACTION_PCT` (default 0.85) of the largest eligible model's window and runs Claude Code's tiered cascade: (1) `ClearOldToolResults` — local, clears stale tool results; (2) structured 9-section summary via a **window-aware** Anthropic-family summarizer (`SummarizeForCompaction`; haiku when the history fits, `claude-fable-5` for larger) rewritten with `RewriteForCompaction(summary, recentTurns)`; (3) progressive `TrimLastNMessages` rescue. If even the trimmed floor overflows, it returns `ErrContextWindowExceeded` → HTTP 413 (distinct from the "no provider keys" `ErrNoEligibleProvider`). The summary call is billed as a `_precompaction_summary` ledger row. Trigger below the window (not at overflow) is load-bearing: a summarizer can only ingest a history that still fits *some* model.
Expand Down
2 changes: 2 additions & 0 deletions internal/proxy/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ The per-action flow is more than "scorer → dispatch". Pinned session, planner

The provider-backed `Summarizer` implementation for handover lives in [`handover.go`](handover.go); the inner-ring `handover` package only defines the contract. On summarizer timeout or error, proxy keeps the full prior history unchanged (it does **not** trim) — a pricier switch action beats silently dropping the conversation the switched-to model needs.

A summary carries the whole prior conversation, so it goes through [`gateSummarizerCall`](summarizer_policy.go) first: excluded providers/models and the tenant credential boundary refuse it outright, and a refusal takes the same full-history / trim fallback as a summarizer error. `Summarizer` therefore exposes `Provider()`/`Model()` — the identity credential resolution and exclusions key off.

## Proactive context-window compaction

`ProxyMessages` / `ProxyOpenAIChatCompletion` call [`maybeCompact`](compaction.go) **before** routing so an over-long session is compacted rather than dead-ending in the scorer with no eligible provider. It engages when the estimate reaches `ROUTER_COMPACTION_PCT` (default 0.85) of the largest eligible model's window and runs Claude Code's tiered cascade: (1) `ClearOldToolResults` — local, clears stale tool results; (2) structured 9-section summary via a **window-aware** Anthropic-family summarizer (`SummarizeForCompaction`; haiku when the history fits, `claude-fable-5` for larger) rewritten with `RewriteForCompaction(summary, recentTurns)`; (3) progressive `TrimLastNMessages` rescue. If even the trimmed floor overflows, it returns `ErrContextWindowExceeded` → HTTP 413 (distinct from the "no provider keys" `ErrNoEligibleProvider`). The summary call is billed as a `_precompaction_summary` ledger row. Trigger below the window (not at overflow) is load-bearing: a summarizer can only ingest a history that still fits *some* model.
Expand Down
4 changes: 4 additions & 0 deletions internal/proxy/authoritative_turnloop_internal_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,10 @@ func (s *authoritativeHandoverSummarizer) Provider() string {
return providers.ProviderAnthropic
}

func (s *authoritativeHandoverSummarizer) Model() string {
return DefaultHandoverModel
}

func TestAuthoritativePolicySelectsEveryEligibleTurn(t *testing.T) {
strategy := router.Strategy("authoritative-test")
tests := []struct {
Expand Down
37 changes: 18 additions & 19 deletions internal/proxy/compaction.go
Original file line number Diff line number Diff line change
Expand Up @@ -92,12 +92,17 @@ func (s *Service) maxEligibleContextWindow(policyExcluded map[string]struct{}, s
return maxWindow
}

// selectCompactionSummarizer returns the cheapest configured summarizer model
// selectCompactionSummarizer returns the cheapest permitted summarizer model
// whose context window can ingest historyTokens plus summary headroom, or ""
// when none can (caller falls back to trimming).
func (s *Service) selectCompactionSummarizer(historyTokens int) string {
// when none can (caller falls back to trimming). Excluded models are skipped:
// the summary call would ship the whole history to a banned model.
func (s *Service) selectCompactionSummarizer(ctx context.Context, historyTokens int) string {
need := historyTokens + compactionSummaryOutputReserve
excluded := s.excludedModelsForRequest(ctx)
for _, m := range compactionSummarizerModels {
if _, banned := excluded[m]; banned {
continue
}
if catalog.ContextWindowFor(m) >= need {
return m
}
Expand Down Expand Up @@ -204,33 +209,27 @@ func (s *Service) billCompactionSummary(ctx context.Context, requestID, external
}

// runCompactionSummary picks a window-aware summarizer model and dispatches the
// structured summary call, honoring the tenant-boundary credential rules used
// by the switch-handover path. Returns ok=false (and logs) when no summarizer
// fits the history, the tenant boundary forbids the call, or the call fails —
// in every such case the caller falls through to trimming.
// structured summary call, honoring the exclusion and tenant-boundary rules
// used by the switch-handover path. Returns ok=false (and logs) when no
// permitted summarizer fits the history, policy forbids the call, or the call
// fails — in every such case the caller falls through to trimming.
func (s *Service) runCompactionSummary(ctx context.Context, env *translate.RequestEnvelope, reqHeaders http.Header) (string, handover.Usage, string, bool) {
log := observability.FromContext(ctx)

model := s.selectCompactionSummarizer(env.ContextOverflowTokenEstimate())
model := s.selectCompactionSummarizer(ctx, env.ContextOverflowTokenEstimate())
if model == "" {
log.Info("Compaction Tier-3 skipped: history exceeds every summarizer window", "history", env.ContextOverflowTokenEstimate())
log.Info("Compaction Tier-3 skipped: no permitted summarizer window fits the history", "history", env.ContextOverflowTokenEstimate())
return "", handover.Usage{}, "", false
}

sumProvider := s.compactionSummarizer.Provider()
sumCreds := resolveSummarizerCreds(ctx, sumProvider, reqHeaders)
if sumCreds == nil && s.requestUsesNonDeploymentCreds(ctx, reqHeaders) {
log.Info("Compaction Tier-3 skipped: would cross tenant boundary", "sum_provider", sumProvider)
gate := s.gateSummarizerCall(ctx, sumProvider, model, reqHeaders)
if !gate.Allowed {
log.Info("Compaction Tier-3 skipped by policy", "skip_reason", gate.SkipReason, "sum_provider", sumProvider, "summary_model", model)
return "", handover.Usage{}, "", false
}
summCtx := ctx
if sumCreds != nil {
summCtx = context.WithValue(ctx, CredentialsContextKey{}, sumCreds)
} else {
summCtx = clearCredentials(ctx)
}

summary, usage, err := s.compactionSummarizer.SummarizeForCompaction(summCtx, env, model, DefaultCompactionMaxTokens)
summary, usage, err := s.compactionSummarizer.SummarizeForCompaction(gate.summarizerContext(ctx), env, model, DefaultCompactionMaxTokens)
if err != nil {
log.Warn("Compaction summarizer failed; falling back to trim", "err", err, "model", model)
return "", handover.Usage{}, "", false
Expand Down
7 changes: 4 additions & 3 deletions internal/proxy/compaction_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -190,9 +190,10 @@ func TestWithCompaction_ZeroPctDisables(t *testing.T) {

func TestSelectCompactionSummarizer_WindowAware(t *testing.T) {
s := &Service{}
assert.Equal(t, DefaultHandoverModel, s.selectCompactionSummarizer(1_000), "small history → cheap model")
assert.Equal(t, largeWindowSummarizerModel, s.selectCompactionSummarizer(300_000), "history over the cheap model's window → large-window model")
assert.Equal(t, "", s.selectCompactionSummarizer(5_000_000), "history over every window → none")
ctx := context.Background()
assert.Equal(t, DefaultHandoverModel, s.selectCompactionSummarizer(ctx, 1_000), "small history → cheap model")
assert.Equal(t, largeWindowSummarizerModel, s.selectCompactionSummarizer(ctx, 300_000), "history over the cheap model's window → large-window model")
assert.Equal(t, "", s.selectCompactionSummarizer(ctx, 5_000_000), "history over every window → none")
}

func TestMaxEligibleContextWindow(t *testing.T) {
Expand Down
48 changes: 25 additions & 23 deletions internal/proxy/handover.go
Original file line number Diff line number Diff line change
Expand Up @@ -65,16 +65,23 @@ const compactionInstruction = "The conversation is being compacted to fit the mo

// ProviderSummarizer adapts a providers.Client to handover.Summarizer by
// building a small Anthropic Messages request from the prior conversation.
// The client must speak the Anthropic Messages format natively; provider names
// the upstream it dispatches to so credential resolution, exclusion policy and
// billing attribution all key off the same identity.
type ProviderSummarizer struct {
client providers.Client
provider string
model string
timeout time.Duration
maxTokens int
}

// NewProviderSummarizer constructs a summarizer adapter. Empty/zero args
// fall back to defaults.
func NewProviderSummarizer(client providers.Client, model string, timeout time.Duration) *ProviderSummarizer {
// NewProviderSummarizer constructs a summarizer adapter dispatching to the
// named provider. Empty/zero args fall back to defaults.
func NewProviderSummarizer(client providers.Client, provider, model string, timeout time.Duration) *ProviderSummarizer {
if provider == "" {
provider = providers.ProviderAnthropic
}
if model == "" {
model = DefaultHandoverModel
}
Expand All @@ -83,6 +90,7 @@ func NewProviderSummarizer(client providers.Client, model string, timeout time.D
}
return &ProviderSummarizer{
client: client,
provider: provider,
model: model,
timeout: timeout,
maxTokens: DefaultHandoverMaxTokens,
Expand All @@ -100,7 +108,12 @@ func (s *ProviderSummarizer) WithMaxTokens(n int) *ProviderSummarizer {

// Provider returns the upstream provider this summarizer dispatches to.
func (s *ProviderSummarizer) Provider() string {
return providers.ProviderAnthropic
return s.provider
}

// Model returns the model the switch-handover summary is dispatched to.
func (s *ProviderSummarizer) Model() string {
return s.model
}

// ErrEmptySummary is returned when the upstream call succeeded but no
Expand Down Expand Up @@ -162,7 +175,7 @@ func (s *ProviderSummarizer) summarize(ctx context.Context, env *translate.Reque
prep.Headers.Set("anthropic-version", "2023-06-01")

decision := router.Decision{
Provider: providers.ProviderAnthropic,
Provider: s.provider,
Model: model,
Reason: kind + "_summary",
}
Expand Down Expand Up @@ -193,7 +206,7 @@ func (s *ProviderSummarizer) summarize(ctx context.Context, env *translate.Reque
}
usage := extractAnthropicUsage(respBody)
usage.Model = model
usage.Provider = providers.ProviderAnthropic
usage.Provider = s.provider
return text, usage, nil
}

Expand Down Expand Up @@ -309,34 +322,23 @@ func (s *Service) runCompactionHandover(ctx context.Context, env *translate.Requ
out.Invoked = true

var (
sumProvider string
sumCreds *Credentials
canCallSummarizer bool
sumProvider string
gate summarizerGate
)
if s.summarizer != nil {
sumProvider = s.summarizer.Provider()
sumCreds = resolveSummarizerCreds(ctx, sumProvider, reqHeaders)
nonDepCreds := s.requestUsesNonDeploymentCreds(ctx, reqHeaders)
canCallSummarizer = sumCreds != nil || !nonDepCreds
gate = s.gateSummarizerCall(ctx, sumProvider, s.summarizer.Model(), reqHeaders)
}

switch {
case s.summarizer == nil:
out.FallbackToFullHistory = true
log.Info("Compaction handover: summarizer not wired; preserved compacted body instead", "decision_model", decisionModel)
case !canCallSummarizer:
case !gate.Allowed:
out.FallbackToFullHistory = true
log.Info("Compaction handover: summarizer skipped (tenant boundary); preserved compacted body instead", "decision_model", decisionModel)
log.Info("Compaction handover: summarizer skipped by policy; preserved compacted body instead", "skip_reason", gate.SkipReason, "sum_provider", sumProvider, "decision_model", decisionModel)
default:
summCtx := ctx
if sumCreds != nil {
summCtx = context.WithValue(ctx, CredentialsContextKey{}, sumCreds)
} else {
// Strip any request credential (e.g. subscription OAuth token) so
// this synthetic call runs on the deployment key instead of
// inheriting one that could 401 or cross a tenant boundary.
summCtx = clearCredentials(ctx)
}
summCtx := gate.summarizerContext(ctx)
start := time.Now()
summary, summaryUsage, sumErr := s.summarizer.Summarize(summCtx, env)
out.LatencyMS = time.Since(start).Milliseconds()
Expand Down
10 changes: 5 additions & 5 deletions internal/proxy/handover_internal_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ func TestProviderSummarizer_SuccessReturnsAssistantText(t *testing.T) {
respBody: canonicalAnthropicResponse,
respStatus: http.StatusOK,
}
s := NewProviderSummarizer(fake, "", 200*time.Millisecond)
s := NewProviderSummarizer(fake, providers.ProviderAnthropic, "", 200*time.Millisecond)

got, _, err := s.Summarize(context.Background(), env)
require.NoError(t, err)
Expand All @@ -107,7 +107,7 @@ func TestProviderSummarizer_TimeoutReturnsError(t *testing.T) {
// Sleep longer than the summarizer's timeout.
sleep: 200 * time.Millisecond,
}
s := NewProviderSummarizer(fake, "", 25*time.Millisecond)
s := NewProviderSummarizer(fake, providers.ProviderAnthropic, "", 25*time.Millisecond)

got, _, err := s.Summarize(context.Background(), env)
require.Error(t, err)
Expand All @@ -127,7 +127,7 @@ func TestProviderSummarizer_Non2xxReturnsError(t *testing.T) {
respBody: `{"error":"oops"}`,
respStatus: http.StatusInternalServerError,
}
s := NewProviderSummarizer(fake, "", 200*time.Millisecond)
s := NewProviderSummarizer(fake, providers.ProviderAnthropic, "", 200*time.Millisecond)

got, _, err := s.Summarize(context.Background(), env)
require.Error(t, err)
Expand All @@ -147,7 +147,7 @@ func TestProviderSummarizer_EmptyContentReturnsErrEmptySummary(t *testing.T) {
respBody: `{"id":"msg_empty","content":[]}`,
respStatus: http.StatusOK,
}
s := NewProviderSummarizer(fake, "", 200*time.Millisecond)
s := NewProviderSummarizer(fake, providers.ProviderAnthropic, "", 200*time.Millisecond)

got, _, err := s.Summarize(context.Background(), env)
require.Error(t, err)
Expand All @@ -159,7 +159,7 @@ func TestProviderSummarizer_NilEnvelopeReturnsError(t *testing.T) {
t.Parallel()

fake := &fakeHandoverProvider{}
s := NewProviderSummarizer(fake, "", 200*time.Millisecond)
s := NewProviderSummarizer(fake, providers.ProviderAnthropic, "", 200*time.Millisecond)

_, _, err := s.Summarize(context.Background(), nil)
require.Error(t, err)
Expand Down
Loading
Loading