Skip to content

feat: two-model split — tiny model for intent parse, big model for execution #17

Description

@bigmikecreates

Problem

Intent parsing with llama3.1 takes ~60 s per request. The parse step only needs to classify which of five action types the user requested — a task a much smaller model handles just as accurately.

Proposed fix

Split into two providers:

  • Parse model (llama3.2:1b) — intent classification only (~3 s)
  • Execution model (llama3.1) — summarisation, drafting, content generation (~15 s)

Implementation

apps/slack-server/src/llm.ts — add buildParseProvider():

  • Reads VIGOUR_PARSE_PROVIDER (default: same as VIGOUR_LLM_PROVIDER) and VIGOUR_PARSE_MODEL
  • Temporarily overrides the provider-specific model env var, calls createProvider(), restores (try/finally)
  • Returns null when not configured → caller falls back to main llm

apps/slack-server/src/index.ts — wire both providers:

  • const parseLlm = buildParseProvider() alongside existing const llm = buildLlmProvider()
  • resolveIntent uses parseLlm ?? llm for the parseIntent call
  • executeAction and registerConfirmationFlow keep receiving llm (big model) — unchanged

apps/slack-server/.env.example — document new vars:

VIGOUR_PARSE_PROVIDER=ollama
VIGOUR_PARSE_MODEL=llama3.2:1b

Expected latency

Command Before After
/vigour what is the time? ~60 s ~3 s
/vigour summarize my unread messages ~60 s ~18 s (3 s parse + 15 s summarise)

No changes needed

execute.ts, confirm-flow.ts, intent.ts, @vigour/intent package — untouched.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions