feat(server): add bmoe-server HTTP binary with OpenAI-compatible API - #161
Open
Bjynt wants to merge 9 commits into
Open
feat(server): add bmoe-server HTTP binary with OpenAI-compatible API#161Bjynt wants to merge 9 commits into
Bjynt wants to merge 9 commits into
Conversation
Add a new bmoe-server binary that loads a model once and serves it over HTTP, preserving the Session-based SSD streaming benefits between requests. Exposes GET /v1/models, POST /v1/completions, and POST /v1/chat/completions with SSE streaming via stream=true. All bmoe-cli streaming flags are supported. Tested on-device with MiniCPM (505M) and Ornith (22GB MoE).
- Always enable chatml so model chat template wraps messages arrays
- Handle 'content' as message content array ([{type:'text',text:'...'}])
as sent by the OpenAI/Node SDK (pi uses this format)
- Support 'max_completion_tokens' field in addition to 'max_tokens'
- Disable render_text for streaming to eliminate O(n^2) per-token
parsing overhead with large prompts (53s -> 0.5s for 5 tokens)
- Set greq.think from --no-think flag (was only on RunConfig, not passed
through to GenerateRequest)
- Update CHANGELOG and README with fixes
OpenAI-compatible SDKs (OpenAI/JS, OpenAI/Python) use fetch() and expect chunked transfer encoding for streaming responses. Without it, Connection: close causes the SDK to buffer the entire body before parsing, which deadlocks on streaming responses. Add proper chunked encoding format: hex size prefix per chunk, and a terminating zero-size chunk after [DONE].
- Add mmproj_path to SessionConfig, RunConfig, ServerConfig - Initialize MTMD context in Session::open() from mmproj_path - Add --mmproj CLI flag to bmoe-server - Add extract_images() to parse OpenAI-compatible image_url format - Add images field to GenerateRequest, wired through server handlers - Server loads mmproj and logs 'mmproj loaded' on startup - Update CHANGELOG.md and README.md with vision support docs
…e buffer scaling - Add n_batch to RunConfig and wire through to SessionConfig - Add --batch-size CLI flag to bmoe-server (default 512) - Fix compute buffer scaling: was O(n_ctx), now O(n_batch) - Enables large context (8192+) with small compute buffer (490 MiB) - Verified: 8192 context with 512 batch = 490 MiB compute buffer
…e buffer scaling - Add n_batch to RunConfig and wire through to SessionConfig - Add --batch-size CLI flag to bmoe-server (default 512) - Fix compute buffer scaling: was O(n_ctx), now O(n_batch) - Enables large context (8192+) with small compute buffer (490 MiB) - Verified: 8192 context with 512 batch = 490 MiB compute buffer
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I wanted to run this with Pi in Termux, but seems to be missing the server, so..
Added a new bmoe-server binary that loads a model once and serves it over HTTP, preserving the Session-based SSD streaming benefits between requests.
Exposes:
All bmoe-cli streaming flags are supported (--moe-stream, --cache-mb, --prefetch, etc.).
Tested on-device with MiniCPM-V 4.6 (505M) and Ornith 35B (22GB Qwen35MoE, 256 experts, 8 used/token with SSD streaming).
!! Code implemented by Laguna-S-2.1 !!