nanoserve

A small, readable LLM server for Apple Silicon: an OpenAI-style chat API with SSE streaming, a continuous-batching engine, and a benchmark suite that times every request from the client's side. It serves pretrained Qwen models through mlx-lm and the models I train from scratch in nanotrain.

I built it to find out where the time goes when a 14B model answers over HTTP on a laptop: how long the prompt takes, how fast tokens come out, what the KV cache costs, what the server itself adds, and what happens when several people ask at once.

Results

As of 2026-10-17 on an Apple M4 Pro (48 GB), mlx-lm 0.31.3, Qwen3-14B at 4-bit, measured through the real server over localhost. The Mac was on battery in macOS Low Power Mode for every run, which caps GPU clocks; plugged in, the absolute speeds will be higher. Ratios between runs are the more portable numbers.

	measured
Decode, 153-token prompt	18.1 tok/s
Decode, 16k-token prompt	12.6 tok/s (30% slower: the KV cache is reread every step)
Time to first token at 537 / 4,121 / 16,409 prompt tokens	5.0 s / 32.4 s / 139 s
Prefill rate, 153 to 16k tokens	107 to 134 tok/s
Peak memory at 16k context	10.8 GiB (7.7 GiB of weights)
HTTP + SSE cost per request (client clock minus engine clock, same request)	about 5 ms
nanoserve vs raw mlx-lm, 16 interleaved pairs	decode 3% slower through the server (13 of 16 pairs)
8 simultaneous streams (281-token prompts, 128 new tokens each), aggregate throughput	12.7 tok/s one at a time, 27.7 tok/s batched (2.2x)
8 simultaneous streams, slowest time to first token	73 s one at a time, 16 s batched

Smaller models through the same server (512-token prompt, 128 new tokens):

model	weights	decode	TTFT at 4k tokens	peak memory at 4k
Qwen2.5-0.5B 4-bit	0.26 GiB	342 tok/s	0.99 s	1.1 GiB
Qwen2.5-0.5B 8-bit	0.49 GiB	251 tok/s	1.02 s	1.3 GiB
Qwen2.5-0.5B bf16	0.92 GiB	167 tok/s	0.89 s	1.7 GiB
Qwen3-1.7B 4-bit	0.90 GiB	152 tok/s	3.24 s	2.1 GiB
Qwen3-4B 4-bit	2.11 GiB	64 tok/s	8.22 s	3.4 GiB
Qwen3-8B 4-bit	4.29 GiB	35 tok/s	14.59 s	5.6 GiB
Qwen3-14B 4-bit	7.74 GiB	16 tok/s	32.45 s	9.0 GiB

Medians of 3 runs after a warmup (2 runs at 16k). The Qwen3-14B row reuses the context sweep's 512- and 4,096-token runs. Every row behind these numbers is in results/suite/ as JSONL, with the date, chip, git sha, and power state; summary.md is the generated version of these tables.

How it works

flowchart LR
    C["client<br/>curl, OpenAI SDK, client.py"] -->|"POST /v1/chat/completions"| S["server.py<br/>FastAPI + SSE"]
    S --> E{"engine.py"}
    E -->|"mlx: one request at a time"| M["mlx-lm stream_generate<br/>Qwen3, 4-bit"]
    E -->|"mlx-batch"| B["batching.py<br/>scheduler thread +<br/>mlx-lm BatchGenerator"]
    E -->|"nanotrain"| N["nanotrain checkpoint<br/>byte-level GPT"]
    S -->|"data: token chunks,<br/>then usage + stats"| C

Loading

Every engine implements one method, _stream, which yields (text, tokens) pairs. BaseEngine.stream_stats wraps it once for all engines and measures time to first token, decode rate, and peak MLX memory. The server turns that stream into OpenAI-style SSE chunks; the last chunk carries usage and the engine's own stats.

The mlx engine holds a lock for each generation, so concurrent requests queue and the wait counts toward their TTFT. The mlx-batch engine runs one scheduler thread that owns the model: requests drop their prompts into an inbox, the scheduler adds them to mlx-lm's BatchGenerator between decode steps, and each generated token is routed back to its request's queue. That is continuous batching in about 140 lines (batching.py).

client.py starts its clock before the request leaves and records when every chunk arrives. Comparing that with the engine's clock, and with mlx-lm called directly, separates model time from serving time.

Quickstart

python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[mac,bench,dev]"
make test                                   # mock engine, no weights needed

python -m nanoserve.server                  # Qwen3-14B 4-bit on :8000 (auto-detects MLX)
curl localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":64,"stream":true}'

NANOSERVE_ENGINE=mlx-batch python -m nanoserve.server     # continuous batching
NANOSERVE_ENGINE=mock python -m nanoserve.server          # no weights at all

Reproduce the numbers

make weights   # Qwen3 1.7B/4B/8B/14B at 4-bit, Qwen2.5-0.5B at 4/8/16-bit (~17 GB)
make bench     # every measurement through a live server (about 45 min on battery)
make charts    # redraw docs/*.png and results/suite/summary.md from the JSONL

make bench runs python -m nanoserve.suite all. Each phase starts the server in a subprocess, waits for /health, warms it up, and writes one row per request. Phases can run on their own: context, raw, overhead, concurrency, models, and nanotrain.

More results

Time to first token is the prompt, almost entirely

On this machine the 14B model prefills about 120 tokens per second, so a 4,000-token prompt waits half a minute before the first token. Decode speed barely matters for a long-prompt request; prefill does.

Prefill throughput is flat across a 100x range of prompt lengths. I expected attention's quadratic cost to pull it down at 16k. My guess is that at this model size the weight matmuls still dominate; either way, the run-to-run spread (bars) is larger than any trend.

Decode slows as context grows

Each decode step reads all the weights plus the whole KV cache. From 153 to 16k tokens of context, decode fell 30%; reading 2.7 GB more per step predicts 24%. The measured speed sits at about half of what the 273 GB/s memory spec would allow. I have not measured it plugged in, so I can't say how much of that gap is Low Power Mode and how much is the kernels.

What the KV cache costs

Qwen3-14B stores 160 KiB of keys and values per token (40 layers, 8 KV heads, head dim 128, fp16). The measured peak follows the formula plus a small, growing allowance for prefill activations.

Model size and precision

Decode speed tracks weight bytes, not parameter count: Qwen2.5-0.5B in bf16 and Qwen3-1.7B at 4-bit both read about 0.9 GiB per token and land at 167 and 152 tok/s. From 0.5 GiB of weights up, every model moves 132 to 165 GB/s, about half the 273 GB/s spec. The smallest model manages only 95 GB/s, because fixed per-token work that does not shrink with the model starts to dominate. The 14B point comes from the context sweep, run about two hours earlier. Within that sweep, repeats of the same request spread by up to 23% in decode speed, so its lower figure (134 GB/s against 162 for the 8B) may be drift rather than size.

What the server costs

I measured it two ways. On a single request, the client's clock minus the engine's clock is the time spent in HTTP, JSON, SSE framing, and thread hand-offs: about 5 ms (left). Against raw mlx-lm I interleaved 16 raw and served requests in ABBA order. Time to first token differed by anything from -200 to +900 ms within a pair (middle), so that comparison is dominated by the machine, not the server. Decode ran a median 3% slower through the server, and 13 of 16 pairs were slower, so that cost looks real: every token is handed from the generation thread to the event loop before it goes out on the socket.

The typical gap between tokens is the same through the server (57.9 vs 58.0 ms at the median, 1,024-token prompt). The tail is not: p99 is 149 ms through the server against 119 ms raw, and the server also shows more very short gaps. From the client, tokens sometimes arrive in pairs after a pause, which is what event-loop scheduling and socket buffering look like.

Concurrency

One at a time, the eighth request in a burst waits for seven others to finish: 73 s before its first token. Batched, all eight share the prefill and then the decode steps, and the worst first token comes at 16 s. Aggregate throughput rises 2.2x, not 8x: this workload has 281-token prompts, and prefill is compute-bound, so batching it barely helps. During decode alone, eight streams together produce about 49 tok/s against 17 for one.

Serve a model you trained in nanotrain

pip install -e ../nanotrain
NANOSERVE_ENGINE=nanotrain NANOSERVE_MODEL=../nanotrain/checkpoints/007-size-11m-lr5e-4 \
  python -m nanoserve.server

The scratch model is a byte-level base model, so the prompt is the raw message text with no chat template. nanotrain owns the model and checkpoint format; nanoserve owns HTTP, streaming, and measurement. The nanotrain README has the serving numbers for its checkpoints.

What I learned

• Time to first token is the number users feel, and on a laptop it is set by prefill. A 16k-token prompt took over two minutes before anything streamed.
• Decode is a memory-bandwidth problem, and the arithmetic works: bytes per token predicted both the model-size ranking and most of the slowdown with context.
• Separate sweeps are not comparable at the millisecond level. Repeats of the same request differed by up to 29% in time to first token (7.3 to 9.4 s for a 1,024-token prompt), while the server's own cost is about 5 ms. Interleaving raw and served requests in pairs, and comparing two clocks on the same request, were the only ways to see the real overhead.
• The lock was not a crash fix. Two threads driving stream_generate on one model at once gave the same greedy text as running alone, in three trials (scripts/threads_check.py). The lock is there so requests queue predictably, waiting shows up in TTFT, and peak memory belongs to one request.
• Batching helps decode, not prefill. With long prompts the 8-stream gain was 2.2x; with short prompts and long answers it would be larger.

Limitations

• One machine, on battery in Low Power Mode. Absolute speeds are lower than the same Mac plugged in.
• Three runs per point (two at 16k). The bars show min to max; some points overlap.
• No quality evaluation. The quantization rows say nothing about answer quality.
• The batched engine does not report per-request peak memory, has no request cancellation, and uses a fixed maximum batch.
• No prefix caching: a repeated system prompt is prefilled every time. That is the next mechanism worth adding.
• The chat API covers the common subset of OpenAI's (messages, max_tokens, temperature, stream). It is a learning server, not a gateway.

Layout

nanoserve/
  engine.py     Engine interface; mock, mlx, and nanotrain engines; timing and memory
  batching.py   continuous batching: scheduler thread over mlx-lm's BatchGenerator
  server.py     OpenAI-style chat API, SSE streaming, /health, /metrics
  config.py     NANOSERVE_* environment settings
  client.py     streaming client that timestamps every chunk
  suite.py      the measurement suite behind this README
  charts.py     JSONL -> docs/*.png and results/suite/summary.md
  bench.py, lab.py, report.py, plot.py, memory.py   the original experiment tools
education/      guides, the prediction-first lab, interactive visualizations
experiments/    lab experiment specs
results/suite/  raw JSONL and environment for every README number
docs/           the charts

The education/ folder has the original learning material: a getting-started guide, the prediction-first lab (python -m nanoserve.lab), and two interactive visualizations.

License

MIT