# Ollama versus vLLM-Metal benchmark

This benchmark compares native Apple-Silicon execution through Ollama and vLLM-Metal. It does not change the Laravel application, the automation worker, or their model configuration.

The completed 2026-08-31 comparison and recommendation are documented in `REPORT.md`.

## Model classification

- Ollama: `qwen3.8:27b-mlx`
- vLLM-Metal: `mlx-community/Qwen3.8-27B-nvfp4`, revision `5ff8ef173ad0d7c3aae92f0be43031a6ab8067c6`
- Both derive from Qwen3.8-27B and use nominal NVFP4 group-size-16 quantization.
- They are **not byte-identical artifacts**. A complete corresponding 44,564,480-byte packed weight tensor produced different SHA-256 hashes. Results therefore compare deployable runtime/model packages, not runtime overhead with identical bytes.

The MXFP8 comparison uses Ollama `qwen3.8:27b-mxfp8` and
`mlx-community/Qwen3.8-27B-mxfp8`, revision
`d48d163bcdf24acaf656474854ab88ea17d65bd1`. Both declare group-size-32 MXFP8
but are also not byte-identical.

## Controlled protocol

1. Run both services natively on the same otherwise-idle Apple Silicon host.
2. Use a context limit of 8,192, temperature 0, seed 42, and 128 maximum output tokens.
	Use raw text completions and disable Ollama speculative drafting so neither runtime has a draft model.
3. Run two warmup rounds before measured rounds.
4. Run five measured rounds at concurrency 1 and five groups at concurrency 4.
5. Alternate three fixed prompts to reduce prefix-cache bias.
6. Compare client-observed TTFT and total latency, warmed decode speed, and aggregate output throughput.
7. Preserve metadata, prompts, per-request measurements, errors, and summaries as JSON and CSV.
8. Treat startup/model loading as a separate cold-load observation; do not mix it into warm inference percentiles.

`benchmark.py` uses only the Python standard library. It expects Ollama at `127.0.0.1:11434` and the vLLM OpenAI-compatible API at `127.0.0.1:8001`.

## Complex coding suite

`coding_benchmark.py` sends the same three complex implementation tasks through each runtime's
native chat API: an asynchronous Python cache, an idempotent Laravel money transfer, and a
concurrent TypeScript DAG scheduler. Thinking and speculative drafting are disabled, temperature is
zero, and full generated responses are retained for inspection. The summary records TTFT, total
latency, output tokens, decode rate, and finish reason.

When every response reaches the configured output limit, the suite measures time for a fixed amount
of long-form coding output rather than time to a naturally completed solution. This distinction must
be preserved in reports.

## Direct MLX control

`direct_mlx_benchmark.py` loads a Hugging Face checkpoint directly with MLX-LM and measures the
same raw prompts at concurrency one. Running it against the exact NVFP4 or MXFP8 checkpoint used by
vLLM isolates short-generation MLX performance from HTTP, scheduling, paged-cache, and request-state
overhead. It is not a concurrent-serving benchmark and cannot isolate Ollama because Ollama's packed
artifact is different.

## Metrics

- **TTFT:** request start to the first non-empty generated text chunk at the client.
- **Total latency:** request start to completion of the response stream.
- **Decode tokens/s:** Ollama's server-reported evaluation rate; for vLLM, output tokens divided by post-TTFT client time.
- **Request tokens/s:** output tokens divided by total client latency, including prefill and network overhead.
- **Aggregate output tokens/s:** all successful generated tokens in a concurrent group divided by group wall time.
- **p50/p95:** linearly interpolated percentiles over measured requests or groups; warmups are excluded.

Prompt throughput is reported when the server exposes both prompt token count and prompt duration. A missing value is not estimated.

## Result files

Each run creates a timestamped directory below `results/` containing:

- `metadata.json`: environment, arguments, model metadata, prompts, and summary
- `raw.json`: complete per-request evidence
- `raw.csv`: tabular per-request evidence
- `summary.json`: grouped statistics

The coding suite stores equivalent evidence below `coding-results/`. Direct MLX controls store
`metadata.json`, `raw.json`, and `summary.json` below `direct-results/`.

Do not compare runs performed under materially different memory pressure, thermal conditions, model revisions, or runtime versions.