A high-performance API server that provides OpenAI-compatible endpoints for MLX models. Developed using Python and powered by the FastAPI framework, it provides an efficient, scalable, and user-friendly solution for running MLX-based vision and language models locally with an OpenAI-compatible interface.

A High-Performance LLM Inference Engine with vLLM-Style Continuous Batching

An Efficient and Versatile Inference Engine for Distributed LLM Serving

Reproducible llama.cpp CPU inference profiling and a deterministic LLM serving simulator with continuous batching, KV cache, prefix caching, and workload-driven latency analysis.

A lightweight, educational LLM inference engine for studying continuous batching, paged KV cache, chunked prefill, and online serving.

Continuous batching for TTS — like vLLM, but for voice. Serve 10+ simultaneous text-to-speech requests on a single GPU.

🌱 A tiny, readable LLM serving engine with vLLM/SGLang-style features.

An AI inference engine from scratch. Like nanoGPT, but for serving.

Real PyTorch inference server comparing FCFS, EagerContBatch, and ChunkedPrefill on RTX 2070. Key finding: ChunkedPrefill is not a throughput optimizer but a fairness mechanism — it reduces worst-case decode stalls from 58.6ms to 34.1ms while EagerContBatch maximizes mean TTFT and throughput.

An educational LLM inference engine built from scratch: custom decoding loop, KV cache, continuous batching, paged memory with preemption, prefix caching, speculative decoding, OpenAI-compatible API, real benchmarks.

High-Performance LLM Inference Engine with PagedAttention & Continuous Batching in Rust

A production-grade, native Rust speculative inference engine for Apple Silicon with Metal GPU acceleration and paged attention.

Local-first LLM stack on a single RTX 5090: QLoRA fine-tuning, exact speculative decoding, paged KV-cache, and continuous batching — served via FastAPI with a live React dashboard.

End-to-end LLM serving simulator integrating scheduling, prefix caching, tensor allocation, and KV-cache management. 168-run sweep (72 baseline + 96 pressure). Key finding: ChunkedPrefill + LFU cache achieves 41% lower TTFT p95 and 94% prefix hit rate, but hits OOM first under memory pressure.

Fork of OpenAI and Anthropic compatible server for Apple Silicon. Native MLX backend, 500+ tok/s. Run LLMs and vision-language models with continuous batching, MCP tool calling, and multimodal support.

A from scratch LLM inference engine build in PyTorch with custom GPT2 transformers, kv cache, paged kv cache, continuous batching and A100 benchmarks

Minimal LLM inference server with KV cache reuse and continuous batching. Built from scratch to understand vLLM mechanisms.

Discrete-event simulator of LLM inference scheduling policies, comparing FCFS, continuous batching, priority, SLO-aware, and chunked prefill under memory pressure and latency SLOs.

A minimal LLM inference server with a real paged KV-cache, continuous batching and speculative decoding, by sarmalinux

Mini-SGLang port for Ascend NPUs with FIA, paged KV cache, ragged continuous batching, and request lifecycle safety.