Skip to content

Home

Jump to bottom
Jason Cravens edited this page Sep 8, 2025 · 2 revisions

LocalAgent-LLM

High-Level Overview

At its core, this is a full-stack TypeScript application designed to be a local AI assistant.

It consists of three main parts:

  • Frontend (client/): A modern React single-page application (SPA) that you interact with in your browser.
  • Backend (Inferred): A Node.js server that handles API requests, user authentication, database interactions, and communication with the AI model.
  • Shared Logic (shared/): (Inferred from drizzle.config.ts) Code that can be used by both the frontend and backend, most importantly, the database schema definitions.

The application allows you to chat with an AI, execute code, manage files, and use various tools, all orchestrated through a clean, tabbed interface.

How It Works

  1. The Frontend (Your User Interface)
    The client directory contains a React application built with Vite.

    What is Vite's role?
    Vite is your development server and build tool. When you run the application for development, Vite starts a server that serves client/index.html. It's known for being extremely fast because it serves your source files directly to the browser and uses modern browser features to handle module loading, only bundling for production. This makes for a very snappy development experience.

    Application Flow:

    • Entry Point: The app starts with client/src/main.tsx, which renders the main App component (client/src/App.tsx).
    • Routing & Auth: App.tsx sets up the core providers (QueryClientProvider for data fetching, TooltipProvider for UI) and a Router. The router uses the wouter library and a custom useAuth hook to manage authentication.
      • If you're not logged in, you're sent to the <AuthPage />.
      • If you are logged in, you're shown the <Home /> page.
      • While it's checking your auth status, it shows a loading spinner.
    • Data Fetching: The application uses TanStack Query (React Query) heavily. You can see this in the useAuth.ts and useAgent.ts hooks. This library is excellent for fetching data from your backend, managing loading and error states, and caching data to make the UI feel faster and more responsive.
    • Main UI (pages/home.tsx): This is the heart of the application. It manages the overall layout, including the tabbed interface (Chat, History, Workspace, etc.) and the various modals (Username, Model Selector, Help). It holds the primary state for the chat messages.
    • Components: The UI is built from a combination of custom components (like ChatInterface, CommandsPanel) and pre-built UI components from shadcn/ui (found in src/components/ui/), which provides a great-looking and accessible set of building blocks.

  2. API Server: This is a Node.js server that exposes a REST API. The frontend communicates with it to perform actions. Key endpoints include:

    • /api/login, /api/register, /api/logout: Handles user authentication and sessions.
    • /api/chat: The main endpoint for interacting with the AI agent.
    • /api/execute/*: Endpoints for running tools like bash, python, and search.
    • /api/commands: For managing the saved "quick commands".
    • /api/workspaces: For managing project workspaces.

    Database (PostgreSQL & Drizzle):

    • Your .env file specifies a connection to a PostgreSQL database. This is where all the application's data is stored: users, chat sessions, command history, etc.
    • What is Drizzle's role? Drizzle is your ORM (Object-Relational Mapper). It bridges the gap between your TypeScript code and your SQL database.
    • Schema Definition: You define your database tables, columns, and relationships in a TypeScript file (specified as shared/schema.ts in your drizzle.config.ts). This makes your schema type-safe and easy to manage.
    • Migrations: When you change your schema file, you use the Drizzle Kit command-line tool to automatically generate SQL migration files (stored in the drizzle/ directory). These files contain the SQL commands needed to update your database to match your new schema, ensuring your database and code are always in sync.
    • Querying: In your backend code, you would use Drizzle's query builder to write type-safe queries to fetch and manipulate data, instead of writing raw SQL strings.

  3. AI & Agent Functionality
    This is the most exciting part of the application. It's not just a simple chatbot; it's an "agent" that can use tools.

    Ollama Integration: The OLLAMA_URL in your .env file points to a running Ollama instance. Your backend server acts as a proxy, taking requests from the frontend and forwarding them to Ollama to get a response from the Large Language Model (LLM).

    The Agent's Workflow:

    • You send a message from the ChatInterface.
    • The useAgent hook's chat.mutateAsync function is called, sending the prompt to your backend's /api/chat endpoint.
    • The backend receives the request. It likely has logic to check if your prompt starts with a tool prefix (like bash: or python:).
    • Tool Execution: If a prefix is found, the backend executes that tool. For example, for bash: ls -l, it would run ls -l in a shell and capture the output.
    • Contextual Prompting: The backend then constructs a more detailed prompt for the LLM. This might include your original message, the output from any tools that were run, and possibly recent chat history for context.
    • LLM Call: This complete prompt is sent to Ollama.
    • Response Handling: The backend receives the LLM's response, saves it to the database as part of the chat session, and sends it back to the frontend to be displayed.

This "tool-using" capability is what makes it a powerful local agent, as it can interact with your system, not just talk. The CommandsPanel and ToolsPanel are UIs that give you direct access to these tool-using capabilities.

The Core Idea: Making External Functionality Available to an LLM Agent

The goal of this setup is to allow a Large Language Model (LLM) agent to use external tools (like running shell scripts, listing files, or performing OCR) as part of its reasoning and problem-solving process. Think of it like giving the LLM a set of specialized functions it can call.

Key Components and Their Roles

  1. shared/toolTypes.ts (Tool Manifest):

    • Purpose: Defines the structure of a tool. It's a blueprint that describes what each tool does, what inputs it expects, what outputs it produces, and how to execute it.
    • ToolManifest Interface:
      • name: A unique identifier for the tool (e.g., "tesseract_ocr", "ls"). This is how the LLM agent will refer to the tool.
      • description: A human-readable explanation of what the tool does. This is crucial for the LLM to understand when to use the tool.
      • inputSchema: A Zod schema that defines the expected format of the input arguments to the tool. Zod is a library for data validation. It ensures that the LLM provides the correct input.
      • outputSchema: A Zod schema that defines the format of the output that the tool will return. This helps the LLM understand the result of the tool.
      • handler: A function that actually executes the tool's logic. This is where the real work happens (e.g., running the shell script, calling the OCR library). It takes the input arguments and a context object (more on that later) and returns a promise that resolves to the output.
      • tags: Optional tags for categorizing tools.
      • mcpEndpoint: MCP for managing tools, but not immediately relevant to the core concept.

  2. server/tools/* (Tool Implementations):

    • Purpose: These files contain the actual implementations of the tools. Each file defines a ToolManifest object that describes a specific tool.
    • Examples:
      • filesystem.ts: Implements the ls tool for listing files. It uses child_process.exec to run the ls command and then parses the output.
      • tesseractOcr.ts: Implements the tesseract_ocr tool for extracting text from images. It uses child_process.execFile to run the tesseract_ocr.sh script.
      • scripts.ts: Dynamically registers shell scripts as tools. It reads the scripts from the server/scripts directory and creates a ToolManifest for each one.
    • Key Points:
      • The handler function in each tool is the most important part. It's where the tool's logic is executed.
      • The context object passed to the handler function can contain useful information, such as the workspaceRoot (the directory where the LLM agent is allowed to access files). This is important for security and sandboxing.
      • Error handling is crucial. The handler function should catch any errors that occur during execution and throw an appropriate exception.

  3. server/tools/registry.ts (Tool Registry):

    • Purpose: A central list of all the available tools. It's an array of ToolManifest objects.
    • How it Works:
      • It imports the ToolManifest objects from the individual tool files (e.g., tesseractOcrTool from tesseractOcr.ts).
      • It creates an array called toolRegistry that contains all the imported tools.
    • Why it's Important: The orchestrator.ts and routes/tools.ts files use this registry to find and access the tools.

  4. server/orchestrator.ts (Tool Orchestrator):

    • Purpose: A function that executes a tool by name. It's the bridge between the LLM agent and the actual tool implementations.
    • callToolByName Function:
      • Takes the tool name, input arguments, and context as input.
      • Looks up the tool in the toolRegistry.
      • If the tool is found, it calls the tool's handler function with the input arguments and context.
      • Returns the result of the handler function.
    • Why it's Important: The LLM agent doesn't directly call the tool implementations. Instead, it calls the callToolByName function, which handles the details of finding and executing the tool.

  5. server/routes/tools.ts (Tool API Endpoint):

    • Purpose: Provides an HTTP endpoint that the LLM agent can use to discover the available tools and their descriptions.
    • / Route (GET):
      • Returns a JSON array of tool metadata (name, description, input schema, output schema, tags).
      • Important: It does not return the handler functions. This is because the LLM agent doesn't need to execute the tools directly. It only needs to know what tools are available and how to use them.
    • Why it's Important: The LLM agent needs to know what tools are available and how to use them. This endpoint provides that information in a structured format.

  6. server/scripts/* (Shell Scripts):

    • Purpose: Contains shell scripts that can be executed as tools.
    • Example: tesseract_ocr.sh is a shell script that runs the Tesseract OCR command.
    • How They're Used: The scripts.ts file automatically registers these scripts as tools.

How It All Works Together (The Big Picture)

  1. Tool Registration: The toolRegistry.ts file imports all the tool implementations and creates a list of available tools. This is like registering the tools with the system.
  2. Tool Discovery: The LLM agent calls the /tools endpoint to get a list of available tools and their descriptions. This allows the LLM to understand what tools it can use.
  3. Tool Selection: Based on the task at hand, the LLM agent decides which tool to use. It uses the tool's description and input schema to determine if the tool is appropriate.
  4. Tool Execution: The LLM agent calls the callToolByName function in orchestrator.ts, passing the tool name, input arguments, and context.
  5. Tool Execution (Orchestrator): The callToolByName function finds the tool in the toolRegistry and calls its handler function.
  6. Tool Execution (Handler): The handler function executes the tool's logic (e.g., running a shell script, calling an API).
  7. Result Handling: The handler function returns the result of the tool execution. The callToolByName function returns this result to the LLM agent.
  8. LLM Reasoning: The LLM agent uses the result of the tool execution to continue its reasoning process.

Clone this wiki locally