lai_automatisierung/benchmarks/inference/REPORT.md

Benchmark report: Ollama, vLLM-Metal, NVFP4, and MXFP8

Date: 2026-08-31

Result

For this Apple M4 Max and the tested Qwen3.8-27B packages:

• Interactive/single-request use: Ollama NVFP4 wins. It delivered 1.83x the vLLM-Metal NVFP4 median decode speed. It was also 1.50x faster than vLLM-Metal for MXFP8.
• Four concurrent requests: vLLM-Metal wins. NVFP4 delivered 1.62x and MXFP8 1.73x Ollama's median aggregate output throughput, with much lower tail TTFT.
• Quantization: NVFP4 wins decisively for speed and memory. MXFP8 approximately doubles weight storage and reduced median single-stream decode speed by 50% in Ollama and 39% in vLLM-Metal.
• Complex batch-1 coding: Ollama wins every measured task. Depending on quantization and task, vLLM-Metal took 1.53x to 2.37x as long.
• Practical choice: use Ollama NVFP4 for interactive chat, coding, and one agent at a time. Use vLLM-Metal NVFP4 when several users or jobs generate concurrently and aggregate capacity or tail TTFT matters. MXFP8 is not attractive for speed or memory on this machine; use it only if its output quality is proven better for the target workload.

All 50 measured requests succeeded. Warmups are excluded from the table.

Both serving stacks were conclusively verified as native Apple GPU implementations. Ollama logged an MLX runner, MLX engine 0.32.0-213-gadf21de, device=gpu, and Metal. vLLM-Metal activated its Metal plugin, set MLX to Device(gpu, 0), set PyTorch to MPS, and loaded native paged-attention Metal kernels. The 0.28.0+cpu vLLM version string describes the portable core wheel, not the active compute backend.

Measurements

Workload and metric	Ollama	vLLM-Metal	Result
Concurrency 1, TTFT p50	0.119 s	0.372 s	Ollama 3.12x faster
Concurrency 1, TTFT p95	0.393 s	0.420 s	Ollama 1.07x faster
Concurrency 1, total latency p50	2.754 s	5.455 s	Ollama 1.98x faster
Concurrency 1, total latency p95	3.451 s	5.562 s	Ollama 1.61x faster
Concurrency 1, decode p50	45.73 tok/s	24.96 tok/s	Ollama 1.83x faster
Concurrency 1, aggregate output p50	44.19 tok/s	23.46 tok/s	Ollama 1.88x faster
Concurrency 4, TTFT p50	5.078 s	1.169 s	vLLM-Metal 4.34x faster
Concurrency 4, TTFT p95	10.691 s	1.236 s	vLLM-Metal 8.65x faster
Concurrency 4, total latency p50	8.619 s	9.273 s	Ollama 1.08x faster
Concurrency 4, total latency p95	14.521 s	9.591 s	vLLM-Metal 1.51x faster
Concurrency 4, aggregate output p50	33.92 tok/s	54.89 tok/s	vLLM-Metal 1.62x faster

The concurrency-4 median individual latency does not represent group completion: Ollama returned some requests early while queueing others. Its p95 TTFT reached 10.69 seconds. Aggregate output rate and p95 latency are the relevant capacity metrics here.

MXFP8 measurements

All 50 measured MXFP8 requests succeeded. The protocol is identical to the NVFP4 test.

Workload and metric	Ollama	vLLM-Metal	Result
Concurrency 1, TTFT p50	0.259 s	0.417 s	Ollama 1.61x faster
Concurrency 1, TTFT p95	0.583 s	0.453 s	vLLM-Metal 1.29x faster
Concurrency 1, total latency p50	5.272 s	8.839 s	Ollama 1.68x faster
Concurrency 1, total latency p95	5.728 s	8.999 s	Ollama 1.57x faster
Concurrency 1, decode p50	22.74 tok/s	15.13 tok/s	Ollama 1.50x faster
Concurrency 1, aggregate output p50	22.22 tok/s	14.39 tok/s	Ollama 1.54x faster
Concurrency 4, TTFT p50	9.394 s	1.314 s	vLLM-Metal 7.15x faster
Concurrency 4, TTFT p95	18.583 s	1.419 s	vLLM-Metal 13.10x faster
Concurrency 4, total latency p50	14.902 s	12.702 s	vLLM-Metal 1.17x faster
Concurrency 4, total latency p95	24.082 s	13.642 s	vLLM-Metal 1.77x faster
Concurrency 4, aggregate output p50	20.18 tok/s	34.96 tok/s	vLLM-Metal 1.73x faster

Complex coding tasks

These runs used native chat templates, thinking disabled, temperature 0, seed 42, an 8,192-token context, and a 3,072-token output cap. Complete generated responses are retained with their hashes.

NVFP4

Task	Ollama total	Ollama decode	vLLM total	vLLM decode	Result by total time
Python async TTL cache	56.41 s	55.38 tok/s	133.44 s	23.20 tok/s	Ollama 2.37x faster
Laravel idempotent transfer	78.72 s	39.58 tok/s	166.01 s	18.92 tok/s	Ollama 2.11x faster*
TypeScript DAG scheduler	96.54 s	32.30 tok/s	152.22 s	20.38 tok/s	Ollama 1.58x faster

* The vLLM PHP response stopped naturally at 2,840 tokens; the Ollama response reached 3,072. Its normalized decode rate was still 2.09x lower.

MXFP8

Task	Ollama total	Ollama decode	vLLM total	vLLM decode	Result by total time
Python async TTL cache	149.82 s	20.64 tok/s	247.71 s	12.46 tok/s	Ollama 1.65x faster
Laravel idempotent transfer	149.77 s	20.72 tok/s	242.90 s	12.73 tok/s	Ollama 1.62x faster
TypeScript DAG scheduler	147.28 s	21.06 tok/s	224.71 s	13.76 tok/s	Ollama 1.53x faster

Except for the vLLM NVFP4 PHP response, every coding run reached the 3,072-token cap. These figures therefore measure time to produce a fixed amount of long-form code, not time to a guaranteed complete or correct implementation. The generated code was retained as evidence but was not compiled as a cross-model quality benchmark because the model artifacts produced different implementations.

Exact-checkpoint direct MLX control

The direct control removes HTTP, vLLM scheduling, paged-cache management, and request-state overhead. It loads the exact same Hugging Face artifact used by vLLM through MLX-LM 0.31.3. It uses the same raw prompts, two warmups, five measured requests, temperature 0, seed 42, and 128 output tokens.

Quantization	Direct MLX decode p50	vLLM decode p50	Difference	Direct MLX peak memory
NVFP4	28.53 tok/s	24.96 tok/s	vLLM 12.5% lower	15.48 GB
MXFP8	15.42 tok/s	15.13 tok/s	vLLM 1.9% lower	28.06 GB

This isolates part of the NVFP4 gap as serving overhead. It also shows that short MXFP8 batch-1 decode is almost identical between direct MLX-LM and vLLM-Metal. It does not make the Ollama comparison artifact-identical: Ollama's packed weights differ. Long hybrid-GDN sequences still incur additional vLLM state and paging work that this short control does not represent.

A fresh back-to-back NVFP4 rerun, with Ollama unloaded before direct MLX-LM started, confirmed the practical batch-1 comparison:

Metric	Ollama	Direct MLX-LM	Result
TTFT p50	0.120 s	0.503 s	Ollama 4.20x faster
Decode p50	45.28 tok/s	27.87 tok/s	Ollama 1.62x faster
Request throughput p50	44.57 tok/s	25.36 tok/s	Ollama 1.76x faster

The direct MLX-LM rerun was 38.4% below Ollama's median decode throughput. This remains a practical-package comparison rather than a byte-identical runtime comparison.

Memory and package sizes

The previously observed approximately 32 GB for vLLM NVFP4 was not a 32-GB model. It was model memory plus a large cache reservation.

Item	NVFP4	MXFP8
Ollama package label	about 18 GB	about 31 GB
Hugging Face checkpoint	16.1 GB	28.7 GB
Direct MLX peak memory	15.48 GB	28.06 GB
vLLM logged model memory	15.13 GB	27.74 GB
vLLM runtime overhead	1.16 GB	1.17 GB
vLLM logical KV/cache budget	15.05 GB	18.51 GB

For MXFP8, vLLM reported 8.94 GB as the concrete MPS KV-cache capacity inside the larger 18.51-GB logical cache budget. Its default 0.92 memory-utilization target provisioned capacity for roughly ten concurrent 8K requests, although the benchmark used at most four short requests. That reservation is serving capacity, not weight size.

Why vLLM-Metal is slower at batch 1

• Ollama uses a tightly integrated Qwen-specific MLX runner and a newer/custom MLX engine revision.
• vLLM-Metal adds scheduling, paged KV-cache management, request state, block-table translation, and hybrid Gated DeltaNet state management.
• Qwen3.8 has 16 SDPA and 48 linear/GDN layers. vLLM-Metal labels this hybrid paged-attention path experimental and translates its large logical blocks into Metal-kernel-compatible blocks.
• Some shapes can use lazy/fused Metal kernels; unsupported cases use less favorable convolution or recurrent fallback paths.
• These costs buy batching: vLLM loses single-stream latency but wins aggregate throughput and tail TTFT at concurrency four.
• The direct exact-checkpoint result confirms measurable vLLM overhead for NVFP4. The much larger Ollama advantage in long coding runs also includes different packed artifacts and different runtime integration, so it cannot be assigned exclusively to the scheduler.

Token-count check

The runtimes did not always stop at the same token count because the packed artifacts generate different output and encounter EOS differently:

Workload	Ollama output tokens	vLLM-Metal output tokens
Five measured requests, concurrency 1	576	640
Twenty measured requests, concurrency 4	2,340	2,556

The fixed short prompt ended at 96–98 tokens in Ollama and at 124–128 tokens in vLLM-Metal. Therefore raw total latency alone is not a fair speed metric. Decode tokens per second and aggregate output tokens per second normalize for this difference and support the same conclusions.

Test protocol

• Host: MacBook Pro, Apple M4 Max, 64 GB unified memory, macOS 15.6, arm64
• Ollama: 0.32.14, native MLX execution
• vLLM core: 0.28.0+cpu with native vllm-metal 0.4.0 development plugin; logs confirmed Device(gpu, 0) and MPS
• Context limit: 8,192 tokens
• Sampling: temperature 0, seed 42
• Output limit: 128 tokens
• Two warmup groups, then five measured groups at concurrency 1 and 4
• Three fixed raw-completion prompts
• Ollama speculative drafting explicitly disabled
• Runtimes loaded and tested sequentially to prevent unified-memory contention

The +cpu suffix belongs to the vLLM core wheel. In this setup, the Metal plugin supplied GPU execution; this was not a CPU benchmark.

Model-equivalence limitation

• Ollama package: qwen3.8:27b-mlx
• vLLM-Metal package: mlx-community/Qwen3.8-27B-nvfp4
• Hugging Face revision: 5ff8ef173ad0d7c3aae92f0be43031a6ab8067c6
• Both are based on Qwen3.8-27B and declare NVFP4 group-size-16 quantization.
• They are not byte-identical. A corresponding 44,564,480-byte packed weight tensor had different SHA-256 hashes.

This is a comparison of the closest practical native runtime/model packages, not a pure runtime-overhead comparison using identical packed weights. Quantization/conversion differences can account for some of the speed difference.

Cold start

Cold startup was measured separately after both model packages were cached locally:

Measurement	Ollama	vLLM-Metal
Model-weight load	3.245 s	3.25 s
Operational readiness	3.455 s for a cold one-token request	approximately 28.5 s from process launch to API ready

The model-weight load itself was effectively tied. vLLM-Metal additionally spent 9.71 seconds profiling, allocating its 15-GB KV cache, and warming kernels; Python process and plugin startup account for the remaining launch time. Ollama's daemon was already running, so its value measures unloaded-model-to-response rather than daemon process launch. These operational measurements are useful but not perfectly symmetric.

The first vLLM setup took about 12 minutes 45 seconds to download and reconstruct the 16.1-GB checkpoint. That one-time network cost is excluded from the cached cold-start and all warm-inference results.

Caveats

• Prompt-throughput values from Ollama were excluded because repeated short prompts produced cache-hit durations near zero; vLLM's completion stream did not expose equivalent per-request prefill duration.
• Five groups establish a clear directional result, but longer multi-run tests are appropriate before capacity planning.
• Both runtimes used prefix caching as shipped. That reflects practical warmed operation, not cache-disabled kernel isolation.
• Total latency is influenced by differing EOS positions; normalized token rates are more comparable.
• Ollama and Hugging Face model pairs share the Qwen3.8-27B base and nominal quantization mode but are not byte-identical. Results compare practical deployable packages, not only runtime code.
• Direct MLX controls are batch-1 controls only; MLX-LM does not reproduce vLLM's concurrent serving objective.

Evidence

• vLLM-Metal: results/20260831T114727Z/metadata.json, raw.json, raw.csv, and summary.json
• Ollama: results/20260831T114928Z/metadata.json, raw.json, raw.csv, and summary.json
• Cold start: results/cold_start.json
• Backend proof: results/backend_verification.json
• Ollama MXFP8: results/20260831T121855Z/
• vLLM-Metal MXFP8: results/20260831T132015Z/
• NVFP4 coding: coding-results/20260831T120914Z-ollama-qwen3.8_27b-mlx/ and coding-results/20260831T121823Z-vllm-metal-qwen3.8-27b-nvfp4/
• MXFP8 coding: coding-results/20260831T123006Z-ollama-qwen3.8_27b-mxfp8/ and coding-results/20260831T133452Z-vllm-metal-qwen3.8-27b-mxfp8/
• Direct MLX controls: direct-results/20260831T133636Z-qwen3.8-27b-nvfp4/ and direct-results/20260831T133721Z-qwen3.8-27b-mxfp8/
• Back-to-back rerun: results/20260831T134334Z/ and direct-results/20260831T134407Z-qwen3.8-27b-nvfp4-rerun/