Nexus — The Universal, Local AI Orchestrator

Version: 0.3.0 Status: All 4 Phases Complete

Nexus is a privacy-first, self-contained AI system that acts as the bridge between you and your digital/physical environment. No cloud required. Your data stays yours.

Core Principles

• Absolute Privacy (Local-Only): All data stays on your devices. Zero cloud dependency.
• Adaptive Footprint: Runs on everything from a smart lightbulb to a gaming PC.
• Universal Control: Connect and orchestrate any device with CPU/RAM/Storage.

Architecture

nexus/
├── core/          # Rust engine (crypto, mesh, discovery, executor, security, ffi)
├── server/        # Python orchestration layer (AI, brain, API, vision, proactive, streaming)
├── models/        # Shared model configs & prompts
├── client/        # Client apps (desktop GUI, CLI, Android)
│   ├── desktop/   # Python customtkinter GUI with Vision tab, API client, voice
│   └── android/   # Kotlin Jetpack Compose app with Vision tab, mesh, voice, skills
├── .github/       # CI/CD workflows
└── docs/          # Documentation

Quick Start

Desktop Server

cd server
pip install -e .
nexus

With GUI

pip install -e ".[full]"
nexus

Headless mode

nexus --headless --port 9090

Desktop GUI (connected to server)

cd client/desktop
python main.py --server http://localhost:9090

Android app

Requirements: Android Studio or Gradle, Android SDK 24+

cd client/android
gradle wrapper --gradle-version 8.11
./gradlew assembleDebug

Install the resulting APK on your phone, grant microphone permission, and open the Vision tab to connect to your server.

Why Nexus?

• No cloud required — on-device voice (Vosk), local LLM (Ollama/llama.cpp), and vision (YOLOv8-nano).
• Learns from you — ML-based routine detection, sequence prediction, proactive reminders, insights & streaks.
• Universal mesh — encrypted peer-to-peer device networking over TCP, mDNS, and MQTT.
• Cross-device — Android app and Linux desktop share the same API. Connect your phone to your PC's cameras and processing power.
• Extensible — Python + Rust architecture, modular design, skill system.
• Open source — MIT license.

What It Does

• Natural language commands (regex + LLM parsing)
• Distributed task execution across the mesh (Rust core)
• Real-time camera streaming with YOLO object detection (SSE + MJPEG)
• Visual search & physical item location across cameras
• Routine learning & proactive alerts
• Network security monitoring
• Encrypted peer-to-peer communication (AES-256-GCM)
• Voice input (on-device Vosk)
• QR and sound-based mesh pairing
• Downloadable skill system

Device Discovery (mDNS)

Nexus nodes find each other on the local network using mDNS (multicast DNS, aka zero-config networking). This is the first step of the mesh: mDNS answers the question "which Nexus devices are here, and where?" before any encrypted peer-to-peer link is established.

Service type

All Nexus nodes advertise and browse under a single well-known service type:

_nexus._tcp.local.

Defined as NEXUS_MDNS_SERVICE_TYPE in core/src/discovery/mod.rs, mirrored by the Python mesh layer (server/nexus_server/mesh/), the desktop client (client/desktop/mesh.py), and Android's NSD (_nexus._tcp in NexusMeshService.kt).

Service-name limit: mdns-sd enforces a 15-byte service-name limit (SERVICE_NAME_LEN_MAX_DEFAULT). Keep custom service types short — nexus (5 bytes) fits comfortably.

Advertise / browse lifecycle

The core DiscoveryService (Rust, behind the mdns feature; enable with --features full) implements the full lifecycle:

Step	API	What happens
1. Start browsing	start_mdns_browsing(type)	Creates an mDNS daemon and starts browsing _nexus._tcp.local.. Idempotent for the same type; returns an error if you try to browse a different type without stopping first.
2. Advertise	advertise_local_service(instance, host, port, props)	Registers this node as an mDNS service. enable_addr_auto() fills in real interface IPs. The props become TXT records broadcast to peers: node_id, hostname, device_class, capability_score. Returns the service fullname.
3. Poll	poll_mdns_events(timeout)	Drains queued events (up to 128 per pass, then blocks up to timeout), processing each ServiceResolved/ServiceRemoved event against the device registry. Returns the number of events handled.
4. Stop	stop_mdns_browsing()	stop_browse + daemon shutdown, clears all mDNS state. is_mdns_browsing() reports whether browsing is active.

How peers merge into the device registry

When the daemon resolves a peer, poll_mdns_events does the following:

1. Build a lightweight profile (device_from_service_info):
   • node_id — from the node_id TXT record, falling back to the instance name (fullname prefix)
   • hostname, device_class (parsed as snake_case, unknown → Unknown), and capability_score (0.0–1.0, used for task scheduling)
   • address (IPv4 preferred, any address as fallback), port, last_seen timestamp
2. Record the instance — the service fullname is mapped to the node's ID in mdns_instances so removals can be reconciled.
3. Merge into the registry — register_device() upserts the DiscoveredDevice, so it shows up in list_devices(), find_by_class(), find_by_peripheral(), and most_capable().

When the peer goes away, the daemon emits ServiceRemoved(_, fullname), which looks up the node ID in mdns_instances and removes the device from the registry.

Depth note: mDNS carries presence + TXT metadata only. Full capability scans (CPU, RAM, GPU, AI models, peripherals) happen later over the encrypted mesh handshake — mDNS is how you learn a peer exists, the handshake is how you learn what it can do.

The flow is covered by test_mdns_browse_discovers_local_service in core/src/discovery/mod.rs (register + browse on one daemon, mirroring mdns-sd's own integration test) and runs in CI as part of the Rust Core Tests workflow.

Development Status

Phase	Status
Phase 1: Core NLP & Storage	✅ Complete
Phase 2: Device Discovery & Dist. Execution	✅ Complete
Phase 3: Computer Vision Integration	✅ Complete
Phase 4: Routine Learning & Proactive Alerts	✅ Complete

Testing

Integration Test (all 4 phases)

The end-to-end integration test boots the full server headless and exercises every phase — command execution → task distribution → vision → routine learning — then prints a PASS/FAIL/WARN report.

bash scripts/integration_test.sh

Prerequisites:

• Python 3.10+ with a virtualenv at venv/ (the script auto-activates it if present; otherwise it uses the system Python)
• pip install -e server/. — the script reinstalls it automatically, so this is optional

What the script does:

1. Starts the Nexus server headless on a scratch storage dir (default port 19950)
2. Waits for it to become ready — cold starts import torch + YOLO (~25s), so it polls GET /health every second (up to 60s) instead of using a fixed sleep
3. Runs all four phases and reports results with color-coded PASS/FAIL/WARN counts
4. Stops the server and prints log highlights (errors, camera/YOLO status, routine learning)

Phase Coverage

Phase	Endpoints exercised
Phase 1 — Core Commands	GET /health, POST /api/command (what time is it, roll a dice, tell me a joke), GET /api/status
Phase 2 — Task Distribution	GET /api/network, GET /api/devices/capabilities, POST /api/tasks/submit, GET /api/tasks
Phase 3 — Computer Vision	GET /api/vision/status, GET /api/vision/cameras, POST /api/vision/snapshot, POST /api/vision/search, POST /api/vision/locate
Phase 4 — Routine Learning	GET /api/routines, GET /api/suggestions, GET /api/routines/predict, GET /api/insights, GET /api/streaks, POST /api/reminders/test, GET /api/reminders

Options & Notes

• Custom port: NEXUS_TEST_PORT=19951 bash scripts/integration_test.sh — the log and storage files are scoped per port under /tmp/ so concurrent runs don't collide
• Exit code: 0 when all critical checks pass, 1 if any fail (WARNs don't fail the run)
• First run is slower: YOLOv8-nano weights download on first use, plus torch import; subsequent runs are faster
• The server is always cleaned up when the script finishes

Contributing

This project is early. The most helpful contributions right now are:

1. Real-world testing and bug reports
2. Adding useful built-in commands
3. Improving the UI/UX and documentation
4. Writing tests in server/ and client/android/src/test/

Open an issue or pull request on GitHub.

License

MIT — See LICENSE