lai_automatisierung/benchmarks/inference

Ollama versus vLLM-Metal benchmark

This benchmark compares native Apple-Silicon execution through Ollama and vLLM-Metal. It does not change the Laravel application, the automation worker, or their model configuration.

The completed 2026-08-31 comparison and recommendation are documented in REPORT.md.

Model classification

• Ollama: qwen3.8:27b-mlx
• vLLM-Metal: mlx-community/Qwen3.8-27B-nvfp4, revision 5ff8ef173ad0d7c3aae92f0be43031a6ab8067c6
• Both derive from Qwen3.8-27B and use nominal NVFP4 group-size-16 quantization.
• They are not byte-identical artifacts. A complete corresponding 44,564,480-byte packed weight tensor produced different SHA-256 hashes. Results therefore compare deployable runtime/model packages, not runtime overhead with identical bytes.

The MXFP8 comparison uses Ollama qwen3.8:27b-mxfp8 and mlx-community/Qwen3.8-27B-mxfp8, revision d48d163bcdf24acaf656474854ab88ea17d65bd1. Both declare group-size-32 MXFP8 but are also not byte-identical.

Controlled protocol

1. Run both services natively on the same otherwise-idle Apple Silicon host.
2. Use a context limit of 8,192, temperature 0, seed 42, and 128 maximum output tokens. Use raw text completions and disable Ollama speculative drafting so neither runtime has a draft model.
3. Run two warmup rounds before measured rounds.
4. Run five measured rounds at concurrency 1 and five groups at concurrency 4.
5. Alternate three fixed prompts to reduce prefix-cache bias.
6. Compare client-observed TTFT and total latency, warmed decode speed, and aggregate output throughput.
7. Preserve metadata, prompts, per-request measurements, errors, and summaries as JSON and CSV.
8. Treat startup/model loading as a separate cold-load observation; do not mix it into warm inference percentiles.

benchmark.py uses only the Python standard library. It expects Ollama at 127.0.0.1:11434 and the vLLM OpenAI-compatible API at 127.0.0.1:8001.

Complex coding suite

coding_benchmark.py sends the same three complex implementation tasks through each runtime's native chat API: an asynchronous Python cache, an idempotent Laravel money transfer, and a concurrent TypeScript DAG scheduler. Thinking and speculative drafting are disabled, temperature is zero, and full generated responses are retained for inspection. The summary records TTFT, total latency, output tokens, decode rate, and finish reason.

When every response reaches the configured output limit, the suite measures time for a fixed amount of long-form coding output rather than time to a naturally completed solution. This distinction must be preserved in reports.

Direct MLX control

direct_mlx_benchmark.py loads a Hugging Face checkpoint directly with MLX-LM and measures the same raw prompts at concurrency one. Running it against the exact NVFP4 or MXFP8 checkpoint used by vLLM isolates short-generation MLX performance from HTTP, scheduling, paged-cache, and request-state overhead. It is not a concurrent-serving benchmark and cannot isolate Ollama because Ollama's packed artifact is different.

Metrics

• TTFT: request start to the first non-empty generated text chunk at the client.
• Total latency: request start to completion of the response stream.
• Decode tokens/s: Ollama's server-reported evaluation rate; for vLLM, output tokens divided by post-TTFT client time.
• Request tokens/s: output tokens divided by total client latency, including prefill and network overhead.
• Aggregate output tokens/s: all successful generated tokens in a concurrent group divided by group wall time.
• p50/p95: linearly interpolated percentiles over measured requests or groups; warmups are excluded.

Prompt throughput is reported when the server exposes both prompt token count and prompt duration. A missing value is not estimated.

Result files

Each run creates a timestamped directory below results/ containing:

• metadata.json: environment, arguments, model metadata, prompts, and summary
• raw.json: complete per-request evidence
• raw.csv: tabular per-request evidence
• summary.json: grouped statistics

The coding suite stores equivalent evidence below coding-results/. Direct MLX controls store metadata.json, raw.json, and summary.json below direct-results/.

Do not compare runs performed under materially different memory pressure, thermal conditions, model revisions, or runtime versions.