77 lines
4.3 KiB
Markdown
77 lines
4.3 KiB
Markdown
# Ollama versus vLLM-Metal benchmark
|
|
|
|
This benchmark compares native Apple-Silicon execution through Ollama and vLLM-Metal. It does not change the Laravel application, the automation worker, or their model configuration.
|
|
|
|
The completed 2026-08-31 comparison and recommendation are documented in `REPORT.md`.
|
|
|
|
## Model classification
|
|
|
|
- Ollama: `qwen3.8:27b-mlx`
|
|
- vLLM-Metal: `mlx-community/Qwen3.8-27B-nvfp4`, revision `5ff8ef173ad0d7c3aae92f0be43031a6ab8067c6`
|
|
- Both derive from Qwen3.8-27B and use nominal NVFP4 group-size-16 quantization.
|
|
- They are **not byte-identical artifacts**. A complete corresponding 44,564,480-byte packed weight tensor produced different SHA-256 hashes. Results therefore compare deployable runtime/model packages, not runtime overhead with identical bytes.
|
|
|
|
The MXFP8 comparison uses Ollama `qwen3.8:27b-mxfp8` and
|
|
`mlx-community/Qwen3.8-27B-mxfp8`, revision
|
|
`d48d163bcdf24acaf656474854ab88ea17d65bd1`. Both declare group-size-32 MXFP8
|
|
but are also not byte-identical.
|
|
|
|
## Controlled protocol
|
|
|
|
1. Run both services natively on the same otherwise-idle Apple Silicon host.
|
|
2. Use a context limit of 8,192, temperature 0, seed 42, and 128 maximum output tokens.
|
|
Use raw text completions and disable Ollama speculative drafting so neither runtime has a draft model.
|
|
3. Run two warmup rounds before measured rounds.
|
|
4. Run five measured rounds at concurrency 1 and five groups at concurrency 4.
|
|
5. Alternate three fixed prompts to reduce prefix-cache bias.
|
|
6. Compare client-observed TTFT and total latency, warmed decode speed, and aggregate output throughput.
|
|
7. Preserve metadata, prompts, per-request measurements, errors, and summaries as JSON and CSV.
|
|
8. Treat startup/model loading as a separate cold-load observation; do not mix it into warm inference percentiles.
|
|
|
|
`benchmark.py` uses only the Python standard library. It expects Ollama at `127.0.0.1:11434` and the vLLM OpenAI-compatible API at `127.0.0.1:8001`.
|
|
|
|
## Complex coding suite
|
|
|
|
`coding_benchmark.py` sends the same three complex implementation tasks through each runtime's
|
|
native chat API: an asynchronous Python cache, an idempotent Laravel money transfer, and a
|
|
concurrent TypeScript DAG scheduler. Thinking and speculative drafting are disabled, temperature is
|
|
zero, and full generated responses are retained for inspection. The summary records TTFT, total
|
|
latency, output tokens, decode rate, and finish reason.
|
|
|
|
When every response reaches the configured output limit, the suite measures time for a fixed amount
|
|
of long-form coding output rather than time to a naturally completed solution. This distinction must
|
|
be preserved in reports.
|
|
|
|
## Direct MLX control
|
|
|
|
`direct_mlx_benchmark.py` loads a Hugging Face checkpoint directly with MLX-LM and measures the
|
|
same raw prompts at concurrency one. Running it against the exact NVFP4 or MXFP8 checkpoint used by
|
|
vLLM isolates short-generation MLX performance from HTTP, scheduling, paged-cache, and request-state
|
|
overhead. It is not a concurrent-serving benchmark and cannot isolate Ollama because Ollama's packed
|
|
artifact is different.
|
|
|
|
## Metrics
|
|
|
|
- **TTFT:** request start to the first non-empty generated text chunk at the client.
|
|
- **Total latency:** request start to completion of the response stream.
|
|
- **Decode tokens/s:** Ollama's server-reported evaluation rate; for vLLM, output tokens divided by post-TTFT client time.
|
|
- **Request tokens/s:** output tokens divided by total client latency, including prefill and network overhead.
|
|
- **Aggregate output tokens/s:** all successful generated tokens in a concurrent group divided by group wall time.
|
|
- **p50/p95:** linearly interpolated percentiles over measured requests or groups; warmups are excluded.
|
|
|
|
Prompt throughput is reported when the server exposes both prompt token count and prompt duration. A missing value is not estimated.
|
|
|
|
## Result files
|
|
|
|
Each run creates a timestamped directory below `results/` containing:
|
|
|
|
- `metadata.json`: environment, arguments, model metadata, prompts, and summary
|
|
- `raw.json`: complete per-request evidence
|
|
- `raw.csv`: tabular per-request evidence
|
|
- `summary.json`: grouped statistics
|
|
|
|
The coding suite stores equivalent evidence below `coding-results/`. Direct MLX controls store
|
|
`metadata.json`, `raw.json`, and `summary.json` below `direct-results/`.
|
|
|
|
Do not compare runs performed under materially different memory pressure, thermal conditions, model revisions, or runtime versions.
|