Skip to content

README.md

Latest commit

 

History

History
277 lines (236 loc) · 15.7 KB

README.md

File metadata and controls

277 lines (236 loc) · 15.7 KB
Raw

Tuffy

Tuffy is a direct, capable personal AI agent. It can run fully offline on local model weights, or switch to any OpenAI-compatible API model — same tools, same chat loop, same /models command either way.

Demo: walkthrough video (screenshots in demo/)

See ARCHITECTURE.md for the full design rationale, src/README.md for a map of every package, and docs/ for step-by-step configuration guides (models, MCP servers, skills, tools). Most folders below have their own README with more detail.

Features

  • Local or API models: Run fully offline on local model weights, or switch to any OpenAI-wire-format API model from the /models picker — the model registry and chat loop treat both the same way. See docs/configure-models.md.
  • Grouped tools: Tools are organized by domain — Editing & Files, Coding & Execution, Research & Lookup, System, Memory — shown with headers in /tools and the system prompt.
  • Coding & editing: edit_file for targeted changes (not just whole-file overwrites) and run_python/run_shell (sandboxed, allowlisted — no git, see src/tools/README.md) for working on real code in the workspace.
  • Skills: drop a folder with a SKILL.md into ./.tuffy/skills/<name>/ to teach Tuffy how to approach a kind of task (optionally shipping its own tools or an MCP server to connect) — no core code changes needed. See /skills and docs/configure-skills.md.
  • MCP client: connect any MCP server (filesystem, GitHub, browser, etc.) by listing it in .tuffy/mcp.json — its tools show up alongside Tuffy's native ones automatically. See /mcp and docs/configure-mcp.md.
  • Categorized CLI: /help groups commands by what they act on (Chat, Models, Network, Memory, Inspect, Session) instead of one flat list; every command prints through the same layout helpers, so padding and color mean the same thing across all of them. /status and /network info show session and connectivity state at a glance.
  • Online / offline modes: chosen at startup and switchable with /network. Offline keeps the session on local weights, hides the network-dependent tools, and blocks the memory embedder's download; picking an API model switches the session online for you. /network info prints everything the mode currently controls.
  • Clean chat with live traces: colored step-by-step traces show thoughts, tool calls, and responses as they happen, with an animated status spinner while the agent works.
  • Voice Interactive Mode: Run the CLI in full voice mode using local Whisper STT (Speech-to-Text) and Piper TTS (Text-to-Speech) for hands-free spoken conversations.
  • Long-term memory: a SQLite-backed store (via the external Elastimem library) with facts, episodic recall, session summaries, and lessons learned, extracted by a background worker gated to stay off the foreground reply path. Semantic recall runs on a small local embedding model fetched once on first online run (~67MB, cached under ~/.cache/tuffy/fastembed); offline with nothing cached, retrieval falls back to keyword search. See data/README.md.

Available Commands

Run /help inside Tuffy for the categorized list, or see docs/cli-reference.md for the same commands with full detail.

Chat

  • /new - Start a fresh conversation, keeping long-term memory intact.
  • /clear - Reset the conversation history (long-term memory is untouched — same as /new).
  • /image <path> - Attach an image file to your next message (requires a vision model).
  • /mode [text|voice] - View or switch interaction mode (text vs voice).

Models

  • /models - Arrow-key picker: every model on one row (id, name, provider, capabilities); Enter loads the highlighted one. Replaces having to type an id.
  • /models default - Same picker, and the choice is remembered as the startup default.
  • /models info - Same picker, then print that model's full card (context length, license, endpoint, API-key state, rate limits).

Network

  • /network [online|offline] - Show or switch network mode. Offline restricts the session to local models, hides the web tools, and blocks the memory embedder's download.
  • /network info - One report covering everything the mode controls: models and credentials, hidden tools, the memory embedder's state and cache path, and the voice engine.

Memory

  • /memory - Show a summary of long-term memory (facts about you, recent sessions, lessons learned).
  • /memory search <query> - Search past conversations and stored facts.
  • /memory facts <key> - Full version history of one fact key.
  • /memory forget <key> - Forget one fact key (a tombstone, not a hard delete).
  • /purge - Wipe long-term memory: archives the memory database and starts a fresh one.

Inspect

  • /status - Active model, vision support, turn count, estimated context usage, turn health, rate limits.
  • /tools - List all tools the agent can call, grouped by domain.
  • /skills - List installed skills (drop new ones in ./.tuffy/skills/<name>/).
  • /mcp - List connected MCP servers and the tools they registered.
  • /mcp add <github-url> / /mcp remove <name> - Add or remove an MCP server without hand-editing config.

Session

  • /help - Show the full command list.
  • /exit or /quit - Save session memory and close the program.

Setup & Running

1. Requirements

  • Python 3.11+
  • virtualenv (e.g. using uv or standard Python venv)

2. Installation

Set up your virtual environment and install the dependencies:

# Using uv (recommended)
uv sync

# Or to include optional voice interactive mode packages
uv sync --extra voice

# Or using standard pip
pip install -r pyproject.toml

3. Pick a model

Seven model cards ship registered: two local Qwen3-VL builds (local.py), two Groq-hosted models (api.py), and three Claude models (anthropic.py). Open the /models picker to switch between them, or /models default to persist one as your startup choice.

Out of the box the default is claude-haiku-4-5-anthropic, so a capable model drives both the agent loop and memory's background fact extraction. If it can't load — no key, no network — startup falls back to the local qwen3vl-2b-instruct-q4km rather than refusing to run, which is what lets Tuffy work fully offline with zero configuration.

Two things you supply yourself:

  • API keys, in .env at the repo root (ANTHROPIC_API_KEY=..., GROQ_API_KEY=...). The providers read that file automatically if the variable isn't already exported, so no manual export is needed. .env is gitignored.
  • Local weight files, under src/models/weights/<model_id>/. That directory is gitignored (multi-GB binaries), so a fresh clone has none and the local cards can't load until you put them there.

Adding your own model is one registry.register(...) call and nothing else. Full walkthrough for all three provider types: docs/configure-models.md.

4. (Optional) Connecting MCP servers

See docs/configure-mcp.md — create .tuffy/mcp.json (gitignored) and list any MCP servers you want connected. Tuffy connects to each one at startup and registers its tools as <server>_<tool>; a server that fails to connect is skipped with a warning and never blocks startup.

5. (Optional) Adding skills

See docs/configure-skills.md — drop a folder into ./.tuffy/skills/<name>/ with a SKILL.md (YAML frontmatter with name/description, plus a markdown body of guidance). Optionally add a tools.py or an mcp.json. See .tuffy/skills/scratchpad/ for a working example.

6. Running the Agent

Run the main script to start your chat session:

python3 main.py

To start directly in Voice Mode, pass the --voice flag:

python3 main.py --voice

7. (Optional) Running Tuffy from anywhere with a tuffy command

Add a shell function to your ~/.zshrc (or ~/.bashrc) so typing tuffy in any terminal starts the agent, regardless of your current directory:

tuffy() {
  local project_dir="/absolute/path/to/tuffy"
  local python_bin="$project_dir/.venv/bin/python3"
  [ -x "$python_bin" ] || python_bin="python3"
  (cd "$project_dir" && "$python_bin" main.py)
}

Then source ~/.zshrc (or open a new terminal tab) once. It runs in a subshell, so your working directory is untouched after Tuffy exits, and it prefers the project's own .venv python if one exists.

Note this function calls the venv's python by its direct path (.venv/bin/python3) rather than activating the venv in your shell first — nothing needs deactivating afterward, since your shell's own environment was never touched. If you instead run Tuffy by hand with source .venv/bin/activate followed by python3 main.py, you must run deactivate yourself once Tuffy exits — a subprocess (Tuffy, or anything else) can never deactivate a venv that was activated in its parent shell; environment changes only ever flow parent → child, never child → parent, so there is no way for main.py to undo source .venv/bin/activate for you. Prefer the tuffy() function above if you want a single command that never leaves your shell in an altered state.

8. (Jetson Orin) One-shot setup

If you're deploying Tuffy on a Jetson Orin (e.g. copied over via a pendrive rather than git clone), run scripts/setup_jetson.sh instead of steps 2–7 above. It is the whole of setup on that machine:

bash scripts/setup_jetson.sh

# or start straight in voice mode - arguments are forwarded to main.py
bash scripts/setup_jetson.sh --voice

What it does, in order:

  • installs uv and the apt build/audio deps (cmake, ninja, libportaudio2, libsndfile1, alsa-utils) if they're missing;
  • creates a Python 3.10 venv - not the 3.11 in .python-version, because 3.10 is the one interpreter every jetson-ai-lab wheel set builds onnxruntime-gpu for (cp311 exists nowhere on that index), and that package is what gives both Piper TTS and memory's fastembed embedder GPU execution;
  • runs uv sync --extra voice, holding back the four packages that must be built or sourced specially (llama-cpp-python, pywhispercpp, onnxruntime, piper-tts);
  • source-builds llama-cpp-python and pywhispercpp with CUDA for SM 8.7 (a wheel built on another machine won't have Jetson's GPU backend) - this is the slow part, ~20–30 min;
  • installs onnxruntime-gpu from the jetson-ai-lab index matching this board's JetPack/CUDA;
  • seeds .env from .env.example so there's a place to put ANTHROPIC_API_KEY / GROQ_API_KEY / MAYA_API_KEY;
  • checks that the local model weights are present and warns (without blocking) if they aren't - they're gitignored, so git never carries them and the script never downloads them; put Qwen3VL-2B-Instruct-Q4_K_M.gguf and the shared Q8_0 vision projector under src/models/weights/ yourself;
  • caches memory's embedding model (~67 MB) while the network is still available;
  • verifies all of the above, writes a tuffy shell function into ~/.bashrc/~/.zshrc, and launches the app.

Afterwards, tuffy starts Tuffy from any directory and tuffy --voice starts it in voice mode. The function runs in a subshell and calls the venv's interpreter directly, so your shell's working directory and environment are never touched - nothing to deactivate when you quit.

Re-run the script any time after pulling new dependencies - it's idempotent, and it skips the CUDA rebuild unless one of the four native packages' version pins (or the script itself) actually changed.

Voice mode fails to start / errors when switching to it on Jetson? The two most common causes:

  • setup_jetson.sh was never run (or its CUDA build step failed) — plain uv sync --extra voice installs CPU-only pywhispercpp/onnxruntime/piper-tts wheels, which still work but are much slower, and can fail to import at all if a previous partial install left a broken native extension. Re-run bash scripts/setup_jetson.sh and read its "Final validation" section — it prints exactly which of llama-cpp-python / pywhispercpp / onnxruntime / piper-tts is missing CUDA support.
  • If Tuffy was copied to the Jetson via pendrive rather than freshly set up, src/voice/weights/ (the cached Whisper .bin and Piper .onnx/.onnx.json files) is gitignored and must be copied along with the rest of the repo — otherwise first use of voice mode tries to download them from the network, which fails outright on an offline/air-gapped Jetson.

Either way, /mode voice (or --voice) prints the actual exception ("Error initializing voice components: ...") and falls back to text mode instead of crashing — that message is the fastest way to tell which of the two causes above applies. Once voice mode is running, a mic/ALSA failure mid-session (e.g. a USB microphone unplugged) is also caught and reported per-turn rather than ending the session.

Project Structure

main.py                 Entry point: startup wiring (skills/MCP discovery), then the input loop
src/
  cli/                  Interactive terminal chat: banner, spinner, slash commands, turn loop
  engine/               Provider-agnostic ReAct tool-calling loop (turn_engine, stream_parser, tool_dispatch)
  identity.py           Fixed self-model (name, capabilities) — never LLM-written
  memory.py             Elastimem-backed long-term memory (facts/episodic/lessons) + the `remember`/`recall` tools
  embedder.py           First-run setup for memory's semantic-recall embedding model (cache location, startup prewarm, offline gate)
  network_mode.py       Live online/offline mode for the process (read by models, tools, memory, voice)
  settings.py           Persisted user settings (.tuffy/settings.json) — default model id
  vision.py           Image encoding + IMAGE_SENTINEL protocol for vision tool results
  voice/              Local Speech-to-Text (Whisper) and Text-to-Speech (Piper) wrappers + interactive audio loop; weights/ holds gitignored voice model weight files
  llm/                   Model-provider interface + adapters (local weights, OpenAI-compatible API)
  models/                Model registry; configs/local.py + configs/api.py hold model cards; weights/ holds gitignored model weight files
  prompts/               All system-prompt text: personas.yaml + templates.py
  tools/                 Native tools by domain (editing/coding/research/system) + MCP client
  skills/                Discovery/loading for ./.tuffy/skills/*/ capability packs
.tuffy/                 Agent-owned config (gitignored where noted), industry-standard dotfolder pattern
  skills/                Droppable capability packs (SKILL.md + optional tools.py/mcp.json)
  mcp.json                MCP server config (gitignored)
docs/                   Configuration guides (models, MCP, skills, tools, CLI reference)
scripts/                Setup scripts (setup_jetson.sh — bootstrap + run on Jetson Orin)
data/memory/            JSON-backed long-term memory store
agent_workspace/        Sandboxed file I/O root for the agent's file/code tools
.env                    API keys (gitignored) — read by the API provider if not already exported
.env.example            Template for .env — copy and fill in your keys

Every folder above has its own README with more detail — start at src/README.md for the full map, docs/ for configuration how-tos, or ARCHITECTURE.md for the design rationale behind the tool/model-provider/skills systems.