feat: add native MLX inference benchmarks

benchmarks/inference/README.md

@@ -0,0 +1,76 @@
+# Ollama versus vLLM-Metal benchmark
+
+This benchmark compares native Apple-Silicon execution through Ollama and vLLM-Metal. It does not change the Laravel application, the automation worker, or their model configuration.
+
+The completed 2026-08-31 comparison and recommendation are documented in `REPORT.md`.
+
+## Model classification
+
+- Ollama: `qwen3.8:27b-mlx`
+- vLLM-Metal: `mlx-community/Qwen3.8-27B-nvfp4`, revision `5ff8ef173ad0d7c3aae92f0be43031a6ab8067c6`
+- Both derive from Qwen3.8-27B and use nominal NVFP4 group-size-16 quantization.
+- They are **not byte-identical artifacts**. A complete corresponding 44,564,480-byte packed weight tensor produced different SHA-256 hashes. Results therefore compare deployable runtime/model packages, not runtime overhead with identical bytes.
+
+The MXFP8 comparison uses Ollama `qwen3.8:27b-mxfp8` and
+`mlx-community/Qwen3.8-27B-mxfp8`, revision
+`d48d163bcdf24acaf656474854ab88ea17d65bd1`. Both declare group-size-32 MXFP8
+but are also not byte-identical.
+
+## Controlled protocol
+
+1. Run both services natively on the same otherwise-idle Apple Silicon host.
+2. Use a context limit of 8,192, temperature 0, seed 42, and 128 maximum output tokens.
+	Use raw text completions and disable Ollama speculative drafting so neither runtime has a draft model.
+3. Run two warmup rounds before measured rounds.
+4. Run five measured rounds at concurrency 1 and five groups at concurrency 4.
+5. Alternate three fixed prompts to reduce prefix-cache bias.
+6. Compare client-observed TTFT and total latency, warmed decode speed, and aggregate output throughput.
+7. Preserve metadata, prompts, per-request measurements, errors, and summaries as JSON and CSV.
+8. Treat startup/model loading as a separate cold-load observation; do not mix it into warm inference percentiles.
+
+`benchmark.py` uses only the Python standard library. It expects Ollama at `127.0.0.1:11434` and the vLLM OpenAI-compatible API at `127.0.0.1:8001`.
+
+## Complex coding suite
+
+`coding_benchmark.py` sends the same three complex implementation tasks through each runtime's
+native chat API: an asynchronous Python cache, an idempotent Laravel money transfer, and a
+concurrent TypeScript DAG scheduler. Thinking and speculative drafting are disabled, temperature is
+zero, and full generated responses are retained for inspection. The summary records TTFT, total
+latency, output tokens, decode rate, and finish reason.
+
+When every response reaches the configured output limit, the suite measures time for a fixed amount
+of long-form coding output rather than time to a naturally completed solution. This distinction must
+be preserved in reports.
+
+## Direct MLX control
+
+`direct_mlx_benchmark.py` loads a Hugging Face checkpoint directly with MLX-LM and measures the
+same raw prompts at concurrency one. Running it against the exact NVFP4 or MXFP8 checkpoint used by
+vLLM isolates short-generation MLX performance from HTTP, scheduling, paged-cache, and request-state
+overhead. It is not a concurrent-serving benchmark and cannot isolate Ollama because Ollama's packed
+artifact is different.
+
+## Metrics
+
+- **TTFT:** request start to the first non-empty generated text chunk at the client.
+- **Total latency:** request start to completion of the response stream.
+- **Decode tokens/s:** Ollama's server-reported evaluation rate; for vLLM, output tokens divided by post-TTFT client time.
+- **Request tokens/s:** output tokens divided by total client latency, including prefill and network overhead.
+- **Aggregate output tokens/s:** all successful generated tokens in a concurrent group divided by group wall time.
+- **p50/p95:** linearly interpolated percentiles over measured requests or groups; warmups are excluded.
+
+Prompt throughput is reported when the server exposes both prompt token count and prompt duration. A missing value is not estimated.
+
+## Result files
+
+Each run creates a timestamped directory below `results/` containing:
+
+- `metadata.json`: environment, arguments, model metadata, prompts, and summary
+- `raw.json`: complete per-request evidence
+- `raw.csv`: tabular per-request evidence
+- `summary.json`: grouped statistics
+
+The coding suite stores equivalent evidence below `coding-results/`. Direct MLX controls store
+`metadata.json`, `raw.json`, and `summary.json` below `direct-results/`.
+
+Do not compare runs performed under materially different memory pressure, thermal conditions, model revisions, or runtime versions.