AI Assistant Platform

A full-stack, multi-model AI assistant platform - chat across OpenAI, Anthropic Claude, and Google Gemini; retrieval-augmented generation over your own documents; long/short-term memory; tool & function calling; a real Model Context Protocol (MCP) server and client; multi-step agents; and integrations with Google Calendar, Gmail, Telegram, voice, and vision.

Built with FastAPI, SQLite, Pydantic v2, and vanilla HTML/CSS/JS - no frontend build step required.

---

1. Project overview

This project demonstrates an end-to-end AI engineering stack in a single, readable repository:

Layer	What it shows
Multi-model orchestration	llm_providers.py - one interface over 3 vendor SDKs
RAG	rag_service.py - chunk -> embed -> cosine search -> cited context
Memory	memory_service.py - windowed short-term + persisted long-term facts
Tool/function calling	tools.py + provider-specific tool-calling loops
Agents	agents.py - plan -> act -> observe loop with 3 example personas
MCP	mcp_server.py / mcp_client.py - real stdio MCP server + client
Integrations	Gmail, Calendar (Google OAuth2), Telegram bot, voice (Whisper/TTS), vision (GPT-4o/Claude vision)
Automation	Generic outbound webhook dispatcher for n8n / Zapier / Make

2. Architecture

Clean, mostly-flat architecture - see ARCHITECTURE.md for the full breakdown. In short:

Browser (Jinja2 templates + vanilla JS)
        │  fetch() with JWT bearer token
        ▼
FastAPI routers (routes_*.py)  ──►  Service layer (chat_service, rag_service, ...)
        │                                   │
        ▼                                   ▼
   SQLAlchemy async ORM (models.py)   llm_providers.py (OpenAI / Anthropic / Gemini)
        │
        ▼
     SQLite (data/app.db)

3. Folder structure

ai-assistant-platform/
├── main.py # FastAPI app entrypoint
├── config.py # Settings (env vars)
├── database.py # Async SQLAlchemy engine/session
├── models.py # ORM models
├── schemas.py # Pydantic request/response models
├── auth.py # JWT + password hashing
├── llm_providers.py # Multi-model abstraction (OpenAI/Anthropic/Gemini)
├── chat_service.py # Chat orchestration (memory + RAG + tools)
├── memory_service.py # Short/long-term memory
├── rag_service.py # Document chunking/embedding/search
├── tools.py # Tool/function-calling implementations
├── code_assistant.py # Code generate/explain/debug/refactor/review/test
├── agents.py # Multi-step agent loop
├── mcp_server.py # MCP server (tools, resources, prompts)
├── mcp_client.py # MCP client demo
├── google_auth_service.py # Shared Google OAuth2 flow
├── gmail_service.py # Gmail integration
├── calendar_service.py # Google Calendar integration
├── telegram_service.py # Telegram bot
├── voice_service.py # Speech-to-text / text-to-speech
├── vision_service.py # OCR / captioning / receipt analysis
├── file_search_service.py # Metadata + semantic file search
├── automation_service.py # n8n / Zapier / Make webhooks
├── rate_limiter.py # In-memory rate limit middleware
├── logging_config.py # App-wide logging
├── routes_*.py # FastAPI routers, one per feature area
├── static/ # CSS + JS (no build step)
├── templates/ # Jinja2 pages
├── tests/ # pytest suite
├── docs/ # Additional docs
├── requirements.txt
├── .env.example
├── Start App.bat # Windows one-click launcher
└── Start App (Mac).command # macOS one-click launcher

4. Technology stack

• Backend: Python 3.12+, FastAPI, Pydantic v2, SQLAlchemy 2.0 (async), SQLite
• AI SDKs: openai, anthropic, google-generativeai, langchain
• Auth: JWT (PyJWT) + bcrypt password hashing
• Frontend: HTML5, CSS3 (custom design system, no framework), vanilla JS
• Integrations: Google API Python Client (Gmail/Calendar), python-telegram-bot, MCP SDK
• Testing: pytest, pytest-asyncio, httpx

5. Installation

5.1 Prerequisites

• Python 3.12 or newer
• Git
• (Optional) Visual Studio Code

5.2 Visual Studio Code setup

1. Install the Python extension (ms-python.python).
2. Open the repository folder: File -> Open Folder...
3. Select the interpreter: Ctrl/Cmd+Shift+P -> Python: Select Interpreter -> choose the .venv created below.
4. Recommended: install the Even Better TOML and SQLite Viewer extensions.

5.3 Python installation

Download Python 3.12+ from python.org and verify:

python3 --version

5.4 Virtual environment

python3 -m venv .venv

# Activate:
# macOS/Linux:
source .venv/bin/activate
# Windows (PowerShell):
.venv\Scripts\Activate.ps1

5.5 Install dependencies

pip install -r requirements.txt

5.6 Configure environment variables

cp .env.example .env

Edit .env and fill in at least one AI provider key (see below). APP_SECRET_KEY and JWT_SECRET_KEY should be replaced with long random strings for anything beyond local testing.

5.7 Running FastAPI

uvicorn main:app --reload

Or use the one-click scripts:

• Windows: double-click Start App.bat
• macOS: double-click Start App (Mac).command (first run: chmod +x it, see script header)

Then open http://localhost:8000.

6. Configuring AI providers

OpenAI

1. Create a key at https://platform.openai.com/api-keys
2. Set OPENAI_API_KEY in .env. Adjust OPENAI_MODEL if desired (default gpt-4o-mini).

Claude (Anthropic)

1. Create a key at https://console.anthropic.com/
2. Set ANTHROPIC_API_KEY in .env. Adjust ANTHROPIC_MODEL if desired.

Gemini (Google)

1. Create a key at https://aistudio.google.com/app/apikey
2. Set GOOGLE_API_KEY in .env. Adjust GEMINI_MODEL if desired.

You can configure one, two, or all three - the model switcher in the chat UI only shows providers as "configured" once a key is present, and the app runs fine with any subset.

7. RAG (retrieval-augmented generation)

Upload a PDF, TXT, or Markdown file from the Documents page. The pipeline:

1. Extracts text (pypdf for PDFs)
2. Chunks it (~800 chars, 120 overlap)
3. Embeds each chunk (OpenAI text-embedding-3-small if OPENAI_API_KEY is set, otherwise a deterministic local hashing embedding so the feature still works with zero external dependencies - useful for demos without API keys)
4. Stores chunks + embeddings in SQLite
5. At query time, retrieves the top-K most similar chunks by cosine similarity and injects them into the system prompt with [Source N] citations

8. Memory

• Short-term: the last 20 messages of a conversation, sent verbatim as context.
• Long-term: durable facts (name, preferences, recurring constraints) that the app asks the LLM to identify after each user message; stored in memory_items and injected into every future conversation's system prompt. You can also add/edit/delete memories manually from the Memory page.

9. MCP (Model Context Protocol)

mcp_server.py is a real, standalone MCP server exposing:

• Tools: web_search, calculator (same implementations used by in-app chat)
• Resources: platform://about, platform://changelog
• Prompts: summarize_document, code_review

Run the server directly, or point any MCP-compatible client (Claude Desktop, another agent, etc.) at it:

python mcp_server.py

mcp_client.py is a working demo client that launches the server as a subprocess over stdio and exercises all three primitives:

python mcp_client.py

Note: stdio-based MCP servers reserve stdout exclusively for JSON-RPC protocol messages. mcp_server.py therefore logs to stderr only - don't route its logging through the shared stdout logger used by the web app.

10. Automation (n8n / Zapier / Make)

Set N8N_WEBHOOK_URL, ZAPIER_WEBHOOK_URL, and/or MAKE_WEBHOOK_URL in .env, then call:

POST /api/automation/{platform}/trigger?event=my_event
Body: { "any": "json payload" }

This POSTs {"event": ..., "payload": ...} to your configured webhook, letting you trigger external workflows from anywhere in the app.

11. Voice

• Speech-to-text: Whisper via the OpenAI API (/api/voice/transcribe)
• Text-to-speech: OpenAI TTS (/api/voice/speak)

Both require OPENAI_API_KEY. Try them on the Voice page (uses your browser's microphone via MediaRecorder).

12. Vision

OCR, image captioning, and receipt analysis use GPT-4o (default) or Claude's vision capabilities. Try them on the Vision page.

13. Testing

pytest

The suite uses an in-memory SQLite database and httpx's ASGI transport, so it needs no running server and no API keys. It covers auth, conversation/memory/prompt CRUD, RAG chunking/embedding math, and the tool-calling safety guardrails.

14. Deployment

For a simple single-process deployment:

uvicorn main:app --host 0.0.0.0 --port 8000 --workers 2

Put this behind a reverse proxy (nginx/Caddy) with TLS termination. For anything beyond a demo/portfolio deployment, swap:

• SQLite -> PostgreSQL (DATABASE_URL)
• The in-memory rate limiter -> Redis-backed limiting
• Local file uploads -> S3-compatible object storage

15. Troubleshooting

Symptom	Fix
ModuleNotFoundError on startup	Activate your venv, re-run pip install -r requirements.txt
Registration fails with a bcrypt error	Ensure bcrypt==4.0.1 is installed (newer bcrypt breaks passlib 1.7.4's backend detection)
Chat says "not configured"	Add at least one provider API key to .env and restart
Gmail/Calendar calls return 400	Complete the OAuth flow from Settings -> Connect Gmail/Calendar first
MCP client hangs or errors	Never log to stdout inside mcp_server.py - stdout is reserved for protocol messages
Port already in use	Change APP_PORT in .env, or stop the other process

16. Future improvements

• Swap the local hashing-embedding fallback for a small local sentence-transformer model
• Add WebSocket-based full-duplex voice streaming
• Add per-user rate limiting and usage dashboards
• Add PostgreSQL + pgvector for production-scale RAG
• Add role-based access control for multi-tenant deployments

---

See also: ARCHITECTURE.md, API_REFERENCE.md, ROADMAP.md, CONTRIBUTING.md, SECURITY.md, CHANGELOG.md.