murakumo vs. vLLM, SGLang and llama.cpp — a measured comparison, including the parts where we lose

For engineers evaluating whether murakumo is a serious alternative to vLLM / SGLang / llama.cpp — measured 2026-09-06, unflattering numbers included

The honest way to start a comparison like this is to say what is being compared, because the four things in the title are not the same layer, and a single ranking across layers is meaningless. vLLM, SGLang and llama.cpp are inference engines. Most of what we build is not.

What each of these actually is

Worth saying explicitly because the name invites the wrong guess: kotoba-lang/vllm in this workspace is a client for vLLM servers — inference requests rendered as EDN data — not an engine and not a reimplementation of one.

Scale, measured

Measured 2026-09-06 through the GitHub REST API. Stars and forks come from GET /repos/{owner}/{repo}; contributor and commit counts come from the Link: … rel="last" page number on /contributors and /commits at per_page=1, which is the count rather than an estimate of it.

All four of ours are public repositories. Forks: zero. Watchers: zero. External contributors: zero. That is not a rhetorical flourish — it is the single most important number on this page for anyone deciding what to depend on. Four hundred and fifty people review a change to vLLM. One reviews a change to murakumo.

The decisive measurement: we ran distributed serving, and it lost

Verified 2026-07-05, recorded in kotoba-lang/murakumo's README. The same model, two ways:

A real GPU beats network-distributed CPU by roughly five times on this hardware. We built the distributed path, we measured it against the single-node path, and the single-node path won. Publishing the ratio is the point: a distributed-inference project that only shows you the distributed number is not telling you what it costs.

The reason is physical, and it is also measured. ADR-260815 put a measurement plane in front of the parallelism chooser. Nine of eleven inventoried nodes answered; every one reports 1000baseT full-duplex, and a verified transfer between two minis moved 67,108,864 bytes in 575 ms — 933 Mbps, i.e. the 1 GbE wire saturated. At that link speed the planner correctly returns pipeline, not tensor. Tensor parallelism is not a feature we are missing here; it is a feature that would make things worse on this wire.

What would overturn that is one purchase away and has not been made. Each Mac mini exposes three Thunderbolt ports, already enrolled as members of bridge0 by macOS default, with bridge0 status: inactive on every node — 27 idle ports across the nine reachable nodes, and zero cables. Nothing on the node side has to change for that to become a fabric. Nothing on the node side has changed, either. Until the cables exist, the ports are a plan, not a capability.

The discipline that makes those numbers worth reading

The same ADR is the reason we are willing to publish the losing number. Its rule is that :link-gbps is zero unless every rank boundary carries a verified transfer, and the plan reports :evidence as one of :measured / :partial / :none / :unverified — so a zero cannot silently mean four different things. Before that, a fleet nobody measured and a fleet measured at 0.5 Gbps arrived at the chooser as the same value.

The probe's first implementation reported 9,587–14,913 Mbps across a 1 GbE fleet — fifteen times the physical ceiling. The measured cause: macOS nc abandons the connection when the send buffer fills, so 133,120 bytes of a 16,777,216-byte transfer arrived, the sender exited 0, and the 8 ms it took to fail became the denominator. Exit status, elapsed time and the absence of any error all reported success. That is the failure mode this whole page is written against: a measurement that could not run returning what a measurement that ran and found nothing wrong returns.

What the router actually has

The public boundary is api.murakumo.cloud. Its router does implement capacity admission control, landed 2026-09-01 — this corrects an earlier statement of ours that it was contract-only:

• owned-slot-admission returns :slot-available / :all-slots-busy / :capacity-unmeasured. The unmeasured case is admitted while reporting :enforced false — an unmeasured probe is explicitly not treated as an empty pool.
• merge-pool-capacity asks the whole owned pool rather than one member. That exists because of a measured bug on 2026-09-01: the probe asked only one head, got connection_refused, read that as unmeasured, and admitted 8 concurrent requests that queued to 27.7 s.
• demote-busy-endpoints moves a head measured full behind the others without re-sorting, because rotating the leader splits llama.cpp's prefix cache — an 18k-token transcript re-evaluates from zero at about 83 s instead of hitting the cache at 7–13 s.
• The policy is to return an early 429 rather than queue, because a queued request and a fast one both answer 200 and nobody can tell them apart until the latency arrives.

Continuous batching is delegated to llama.cpp's own slots (--parallel), which is where it belongs. What murakumo does not have is a vLLM-style kernel-level scheduler, and at this layer it does not need one. "Delegated" is a design decision about which layer owns a concern; it is not the same claim as "implemented", and it is not the same claim as "missing" either.

Where the own stack really stands

This is kotoba-lang/inference, the one of the four that is an engine, as recorded in its own verify/maturity.edn (updated 2026-08-28).

Genuinely working. GGUF Q4_K / Q6_K dequantisation. A real 42-layer Gemma4-E4B forward pass. JDK-FFM native K-dot and Deno WebGPU→Metal K-dot both matching the official ggml Q8_K/Q4_K/Q6_K oracle. Token-level agreement with live Ollama: the prompt The capital of France is produces token 9079, " Paris".

Not working yet, in its own words. The gap :distributed-llm-vllm-like is marked :contract-only: "Distributed transformer specs now carry tensor/pipeline parallelism, shard strategy, and paged KV cache as EDN, but no vLLM-like scheduler, admission control, continuous batching, or remote worker execution has been implemented yet."

Speed. Warm decode is 8.4 s/token on Metal, and 6.9–15.5 s/token on a real prompt. Those are seconds per token, not tokens per second. An engine at that speed is a correctness result, not a serving result, and we do not serve production traffic from it.

Live capacity today, and what that number actually measures

Self-declared at GET /v1/models, measured 2026-09-05 for murakumo-main (Qwen3.8-27B, Q4_K_M, int4, Japan):

That was published as a thirty-five-fold degradation, with server-side queueing on the b70 head as the suspected cause, and flagged for the operator rather than papered over. Two corrections followed on this page, and both were wrong. They are kept below the current text, because the shape of the mistakes is the more useful part.

What is verified as of 2026-09-07. murakumo-main is served by gad's port 8090. The unit holding that port is named murakumo-ring.service and its unit file really does define a ten-participant llama.cpp RPC ring — --rpc against nine Mac minis, --split-mode layer --tensor-split 3,3,3,3,4,3,3,3,3,12. A systemd drop-in blanks that ExecStart= and substitutes something else. The effective command, read with systemctl show -p ExecStart, is a single-node Vulkan server: Qwen3.8-27B Q4_K_M with an mmproj, -ngl 999 -c 524288 --parallel 2 --flash-attn on --spec-type ngram-cache,ngram-simple. There is no --rpc in it. The nine RPC workers are still listening on :50052 and have zero connections.

So the distributed ring is not serving, and has not been while any of these numbers were taken. gad is one machine with a very large context window; b70 is another machine with a small one. They differ by about four times per token, and the reason is not distribution — it is different hardware and a sixteen-fold difference in context budget. The seven-day median of 10.38 tok/s quoted in the previous version of this paragraph is real but is a median across a mixture of configurations: that unit restarted twenty-three times on 09-07 alone. Since the current configuration started there are four decode samples, which is not a distribution, and this page will not print one.

What survives all of it: the published sum(completion_tokens) / wall_clock is a real defect, because prefill sits inside the wall clock and a 13-second uncached prefill in front of sixteen tokens publishes as 1.2 tok/s whatever the hardware is doing. And the pool's members genuinely differ by more than four times. If you are evaluating murakumo as a serving endpoint, ask for the decode rate and the prefill rate separately, and ask which endpoint answered.

Two things did get fixed this week, both on b70. Its slots went from one to two at unchanged total context — the same KV budget split in half, and of 3,795 logged requests none exceeded the resulting 16,384-token window. Measured three ways, the result disagreed with itself: a same-shape burst got 8% worse (4,636 → 5,052 ms median), sustained aggregate decode got 8% better (32.3 → 35.5 tok/s), and a short request queued behind a long one went from 16.54 s to 5.12 s with non-overlapping ranges — while the long generation itself paid 12.95 → 20.2 s. Nothing there makes the head faster; it stops one request monopolising it. And prefix reuse turns out to have been working the whole time: n_prompt_tokens_cache reports the current task and reads 0 on an idle slot, which is what was misread as a cache miss. The per-request figure is timings.cache_n, and a nonce-prefixed control shows 10,059 of 12,111 tokens reused on the second turn — prefill 12.1 s down to 2.7 s.

Corrections, 2026-09-07. The first version of this section said the fleet was degraded. The second said it was not, and that the number was an accounting artefact — measured on b70, which is not the endpoint the number came from. The third said the endpoint was a live RPC ring, read off a unit file whose name and first ExecStart both say so, without asking systemd for the effective command. Each correction was measured. Each measured the wrong thing: the wrong machine, then the wrong layer. They are left standing here rather than replaced, because a page whose argument is that operators should publish their own unflattering measurements does not get to quietly tidy up its own.

Feature comparison

Read the murakumo column as a control plane. "Delegated" there means the concern is owned by llama.cpp underneath, which is a placement decision, not a gap. The num/torch/inference column is the own stack, which is early.

Provenance. The vLLM, SGLang and llama.cpp columns were read off those projects' official documentation and repositories on 2026-09-06 — links are in the notes below the table — and were not benchmarked by us. The murakumo and num/torch/inference columns are our own measurements and our own source. Where we could not confirm a cell from a primary source we say so rather than guess.

Notes on the upstream cells

• llama.cpp tensor parallelism is the cell most often stated wrongly, including by us. Its multi-GPU documentation distinguishes three split modes: the default layer is "Pipeline parallelism. Each GPU holds a contiguous slice of layers"; row is an "older row-split tensor-parallel path with comparatively poor performance" that the docs say to avoid in new deployments; and tensor is "tensor parallelism that splits both weights and KV across the participating GPUs", marked experimental. --tensor-split itself is the per-GPU proportion knob ("how much of the model goes to each GPU"), applied under whichever split mode is active — it is not by itself tensor parallelism. The RPC backend distributes weights and KV in proportion to each device's available memory and does not use either term in its own README.
• llama.cpp prefill/decode disaggregation is an open feature request (issue #21266), not a shipped flag. We found no server documentation describing it as available, so the cell reads "not shipped" on the strength of the tracking issue rather than an explicit statement of absence.
• vLLM: parallelism and scaling, speculative decoding, disaggregated prefilling (the page's own title carries "(experimental)").
• SGLang: the project README for continuous batching, RadixAttention and tensor/pipeline/expert/data parallelism; PD disaggregation.
• The two dashes in the admission-control row are not a claim that these engines lack rate limiting. They are engines rather than hosted boundaries, so the row does not apply to them in the same sense; we did not find a comparable per-pool admission decision to cite, and we are not inventing one to fill the cell.

The honest reading

If you need to serve a large model fast today, use vLLM or SGLang on real GPUs, or llama.cpp if you are on Apple Silicon or CPU. That is not modesty; it is what our own measurements say. Four hundred and fifty contributors, twenty thousand commits and a saturated NVLink domain beat one contributor, a thousand commits and 933 Mbps of Ethernet, and no amount of architecture makes that arithmetic different.

What murakumo is actually for is the layer above: taking heterogeneous machines you already own, deciding what can run where from measured memory rather than from a configuration file, emitting the engine's own commands instead of replacing the engine, and refusing to answer when it has not measured enough to answer. The parts of that which are done are done and measured. The parts that are not are listed above with the same specificity.

The reason to publish the 12.7 against the 61.5, and to go and measure the head behind the 1.04 rather than publish it as a collapse, is that a comparison page which only contains the flattering half of the operator's own measurements is not evidence about the system — it is evidence about the operator. Neither is a page that leaves an unexamined bad number standing because it sounds suitably humble.