christian.meyer/lai_automatisierung
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: christian.meyer:1097fa7ae23937af4601bb147f2ff16fe29f7656
Branches Tags
christian.meyer:main christian.meyer:feature/mini-swe-agent-worker
...
compare: christian.meyer:407af8099b35e7e729c5c1b31f4cc2b1eda123fa
Branches Tags
christian.meyer:main christian.meyer:feature/mini-swe-agent-worker

2 Commits
1097fa7ae2 ... 407af8099b

Author SHA1 Message Date
Christian Meyer
407af8099b docs: record Qwen MLX benchmark results
Some checks failed
CI / test (push) Has been cancelled
Details
CI / test (pull_request) Has been cancelled
Details
2026-08-31 16:39:37 +02:00
Christian Meyer
75b2c8958c feat: add native MLX inference benchmarks 2026-08-31 16:39:33 +02:00
45 changed files with 7188 additions and 0 deletions
Expand all files Collapse all files

2
benchmarks/.gitignore vendored Normal file
View File

@@ -0,0 +1,2 @@
__pycache__/
*.py[cod]

76
benchmarks/inference/README.md Normal file
View File

@@ -0,0 +1,76 @@
# Ollama versus vLLM-Metal benchmark
This benchmark compares native Apple-Silicon execution through Ollama and vLLM-Metal. It does not change the Laravel application, the automation worker, or their model configuration.
The completed 2026-08-31 comparison and recommendation are documented in `REPORT.md`.
## Model classification
- Ollama: `qwen3.8:27b-mlx`
- vLLM-Metal: `mlx-community/Qwen3.8-27B-nvfp4`, revision `5ff8ef173ad0d7c3aae92f0be43031a6ab8067c6`
- Both derive from Qwen3.8-27B and use nominal NVFP4 group-size-16 quantization.
- They are **not byte-identical artifacts**. A complete corresponding 44,564,480-byte packed weight tensor produced different SHA-256 hashes. Results therefore compare deployable runtime/model packages, not runtime overhead with identical bytes.
The MXFP8 comparison uses Ollama `qwen3.8:27b-mxfp8` and
`mlx-community/Qwen3.8-27B-mxfp8`, revision
`d48d163bcdf24acaf656474854ab88ea17d65bd1`. Both declare group-size-32 MXFP8
but are also not byte-identical.
## Controlled protocol
1. Run both services natively on the same otherwise-idle Apple Silicon host.
2. Use a context limit of 8,192, temperature 0, seed 42, and 128 maximum output tokens.
Use raw text completions and disable Ollama speculative drafting so neither runtime has a draft model.
3. Run two warmup rounds before measured rounds.
4. Run five measured rounds at concurrency 1 and five groups at concurrency 4.
5. Alternate three fixed prompts to reduce prefix-cache bias.
6. Compare client-observed TTFT and total latency, warmed decode speed, and aggregate output throughput.
7. Preserve metadata, prompts, per-request measurements, errors, and summaries as JSON and CSV.
8. Treat startup/model loading as a separate cold-load observation; do not mix it into warm inference percentiles.
`benchmark.py` uses only the Python standard library. It expects Ollama at `127.0.0.1:11434` and the vLLM OpenAI-compatible API at `127.0.0.1:8001`.
## Complex coding suite
`coding_benchmark.py` sends the same three complex implementation tasks through each runtime's
native chat API: an asynchronous Python cache, an idempotent Laravel money transfer, and a
concurrent TypeScript DAG scheduler. Thinking and speculative drafting are disabled, temperature is
zero, and full generated responses are retained for inspection. The summary records TTFT, total
latency, output tokens, decode rate, and finish reason.
When every response reaches the configured output limit, the suite measures time for a fixed amount
of long-form coding output rather than time to a naturally completed solution. This distinction must
be preserved in reports.
## Direct MLX control
`direct_mlx_benchmark.py` loads a Hugging Face checkpoint directly with MLX-LM and measures the
same raw prompts at concurrency one. Running it against the exact NVFP4 or MXFP8 checkpoint used by
vLLM isolates short-generation MLX performance from HTTP, scheduling, paged-cache, and request-state
overhead. It is not a concurrent-serving benchmark and cannot isolate Ollama because Ollama's packed
artifact is different.
## Metrics
- **TTFT:** request start to the first non-empty generated text chunk at the client.
- **Total latency:** request start to completion of the response stream.
- **Decode tokens/s:** Ollama's server-reported evaluation rate; for vLLM, output tokens divided by post-TTFT client time.
- **Request tokens/s:** output tokens divided by total client latency, including prefill and network overhead.
- **Aggregate output tokens/s:** all successful generated tokens in a concurrent group divided by group wall time.
- **p50/p95:** linearly interpolated percentiles over measured requests or groups; warmups are excluded.
Prompt throughput is reported when the server exposes both prompt token count and prompt duration. A missing value is not estimated.
## Result files
Each run creates a timestamped directory below `results/` containing:
- `metadata.json`: environment, arguments, model metadata, prompts, and summary
- `raw.json`: complete per-request evidence
- `raw.csv`: tabular per-request evidence
- `summary.json`: grouped statistics
The coding suite stores equivalent evidence below `coding-results/`. Direct MLX controls store
`metadata.json`, `raw.json`, and `summary.json` below `direct-results/`.
Do not compare runs performed under materially different memory pressure, thermal conditions, model revisions, or runtime versions.

193
benchmarks/inference/REPORT.md Normal file
View File

@@ -0,0 +1,193 @@
# Benchmark report: Ollama, vLLM-Metal, NVFP4, and MXFP8
Date: 2026-08-31
## Result
For this Apple M4 Max and the tested Qwen3.8-27B packages:
- **Interactive/single-request use: Ollama NVFP4 wins.** It delivered 1.83x the vLLM-Metal NVFP4 median decode speed. It was also 1.50x faster than vLLM-Metal for MXFP8.
- **Four concurrent requests: vLLM-Metal wins.** NVFP4 delivered 1.62x and MXFP8 1.73x Ollama's median aggregate output throughput, with much lower tail TTFT.
- **Quantization: NVFP4 wins decisively for speed and memory.** MXFP8 approximately doubles weight storage and reduced median single-stream decode speed by 50% in Ollama and 39% in vLLM-Metal.
- **Complex batch-1 coding: Ollama wins every measured task.** Depending on quantization and task, vLLM-Metal took 1.53x to 2.37x as long.
- **Practical choice:** use Ollama NVFP4 for interactive chat, coding, and one agent at a time. Use vLLM-Metal NVFP4 when several users or jobs generate concurrently and aggregate capacity or tail TTFT matters. MXFP8 is not attractive for speed or memory on this machine; use it only if its output quality is proven better for the target workload.
All 50 measured requests succeeded. Warmups are excluded from the table.
Both serving stacks were conclusively verified as native Apple GPU implementations. Ollama logged an MLX runner, MLX engine `0.32.0-213-gadf21de`, `device=gpu`, and Metal. vLLM-Metal activated its Metal plugin, set MLX to `Device(gpu, 0)`, set PyTorch to MPS, and loaded native paged-attention Metal kernels. The `0.28.0+cpu` vLLM version string describes the portable core wheel, not the active compute backend.
## Measurements
| Workload and metric | Ollama | vLLM-Metal | Result |
|---|---:|---:|---|
| Concurrency 1, TTFT p50 | 0.119 s | 0.372 s | Ollama 3.12x faster |
| Concurrency 1, TTFT p95 | 0.393 s | 0.420 s | Ollama 1.07x faster |
| Concurrency 1, total latency p50 | 2.754 s | 5.455 s | Ollama 1.98x faster |
| Concurrency 1, total latency p95 | 3.451 s | 5.562 s | Ollama 1.61x faster |
| Concurrency 1, decode p50 | 45.73 tok/s | 24.96 tok/s | Ollama 1.83x faster |
| Concurrency 1, aggregate output p50 | 44.19 tok/s | 23.46 tok/s | Ollama 1.88x faster |
| Concurrency 4, TTFT p50 | 5.078 s | 1.169 s | vLLM-Metal 4.34x faster |
| Concurrency 4, TTFT p95 | 10.691 s | 1.236 s | vLLM-Metal 8.65x faster |
| Concurrency 4, total latency p50 | 8.619 s | 9.273 s | Ollama 1.08x faster |
| Concurrency 4, total latency p95 | 14.521 s | 9.591 s | vLLM-Metal 1.51x faster |
| Concurrency 4, aggregate output p50 | 33.92 tok/s | 54.89 tok/s | vLLM-Metal 1.62x faster |
The concurrency-4 median individual latency does not represent group completion: Ollama returned some requests early while queueing others. Its p95 TTFT reached 10.69 seconds. Aggregate output rate and p95 latency are the relevant capacity metrics here.
## MXFP8 measurements
All 50 measured MXFP8 requests succeeded. The protocol is identical to the NVFP4 test.
| Workload and metric | Ollama | vLLM-Metal | Result |
|---|---:|---:|---|
| Concurrency 1, TTFT p50 | 0.259 s | 0.417 s | Ollama 1.61x faster |
| Concurrency 1, TTFT p95 | 0.583 s | 0.453 s | vLLM-Metal 1.29x faster |
| Concurrency 1, total latency p50 | 5.272 s | 8.839 s | Ollama 1.68x faster |
| Concurrency 1, total latency p95 | 5.728 s | 8.999 s | Ollama 1.57x faster |
| Concurrency 1, decode p50 | 22.74 tok/s | 15.13 tok/s | Ollama 1.50x faster |
| Concurrency 1, aggregate output p50 | 22.22 tok/s | 14.39 tok/s | Ollama 1.54x faster |
| Concurrency 4, TTFT p50 | 9.394 s | 1.314 s | vLLM-Metal 7.15x faster |
| Concurrency 4, TTFT p95 | 18.583 s | 1.419 s | vLLM-Metal 13.10x faster |
| Concurrency 4, total latency p50 | 14.902 s | 12.702 s | vLLM-Metal 1.17x faster |
| Concurrency 4, total latency p95 | 24.082 s | 13.642 s | vLLM-Metal 1.77x faster |
| Concurrency 4, aggregate output p50 | 20.18 tok/s | 34.96 tok/s | vLLM-Metal 1.73x faster |
## Complex coding tasks
These runs used native chat templates, thinking disabled, temperature 0, seed 42, an 8,192-token context, and a 3,072-token output cap. Complete generated responses are retained with their hashes.
### NVFP4
| Task | Ollama total | Ollama decode | vLLM total | vLLM decode | Result by total time |
|---|---:|---:|---:|---:|---|
| Python async TTL cache | 56.41 s | 55.38 tok/s | 133.44 s | 23.20 tok/s | Ollama 2.37x faster |
| Laravel idempotent transfer | 78.72 s | 39.58 tok/s | 166.01 s | 18.92 tok/s | Ollama 2.11x faster* |
| TypeScript DAG scheduler | 96.54 s | 32.30 tok/s | 152.22 s | 20.38 tok/s | Ollama 1.58x faster |
\* The vLLM PHP response stopped naturally at 2,840 tokens; the Ollama response reached 3,072. Its normalized decode rate was still 2.09x lower.
### MXFP8
| Task | Ollama total | Ollama decode | vLLM total | vLLM decode | Result by total time |
|---|---:|---:|---:|---:|---|
| Python async TTL cache | 149.82 s | 20.64 tok/s | 247.71 s | 12.46 tok/s | Ollama 1.65x faster |
| Laravel idempotent transfer | 149.77 s | 20.72 tok/s | 242.90 s | 12.73 tok/s | Ollama 1.62x faster |
| TypeScript DAG scheduler | 147.28 s | 21.06 tok/s | 224.71 s | 13.76 tok/s | Ollama 1.53x faster |
Except for the vLLM NVFP4 PHP response, every coding run reached the 3,072-token cap. These figures therefore measure time to produce a fixed amount of long-form code, not time to a guaranteed complete or correct implementation. The generated code was retained as evidence but was not compiled as a cross-model quality benchmark because the model artifacts produced different implementations.
## Exact-checkpoint direct MLX control
The direct control removes HTTP, vLLM scheduling, paged-cache management, and request-state overhead. It loads the exact same Hugging Face artifact used by vLLM through MLX-LM 0.31.3. It uses the same raw prompts, two warmups, five measured requests, temperature 0, seed 42, and 128 output tokens.
| Quantization | Direct MLX decode p50 | vLLM decode p50 | Difference | Direct MLX peak memory |
|---|---:|---:|---:|---:|
| NVFP4 | 28.53 tok/s | 24.96 tok/s | vLLM 12.5% lower | 15.48 GB |
| MXFP8 | 15.42 tok/s | 15.13 tok/s | vLLM 1.9% lower | 28.06 GB |
This isolates part of the NVFP4 gap as serving overhead. It also shows that short MXFP8 batch-1 decode is almost identical between direct MLX-LM and vLLM-Metal. It does not make the Ollama comparison artifact-identical: Ollama's packed weights differ. Long hybrid-GDN sequences still incur additional vLLM state and paging work that this short control does not represent.
A fresh back-to-back NVFP4 rerun, with Ollama unloaded before direct MLX-LM started, confirmed the practical batch-1 comparison:
| Metric | Ollama | Direct MLX-LM | Result |
|---|---:|---:|---|
| TTFT p50 | 0.120 s | 0.503 s | Ollama 4.20x faster |
| Decode p50 | 45.28 tok/s | 27.87 tok/s | Ollama 1.62x faster |
| Request throughput p50 | 44.57 tok/s | 25.36 tok/s | Ollama 1.76x faster |
The direct MLX-LM rerun was 38.4% below Ollama's median decode throughput. This remains a practical-package comparison rather than a byte-identical runtime comparison.
## Memory and package sizes
The previously observed approximately 32 GB for vLLM NVFP4 was not a 32-GB model. It was model memory plus a large cache reservation.
| Item | NVFP4 | MXFP8 |
|---|---:|---:|
| Ollama package label | about 18 GB | about 31 GB |
| Hugging Face checkpoint | 16.1 GB | 28.7 GB |
| Direct MLX peak memory | 15.48 GB | 28.06 GB |
| vLLM logged model memory | 15.13 GB | 27.74 GB |
| vLLM runtime overhead | 1.16 GB | 1.17 GB |
| vLLM logical KV/cache budget | 15.05 GB | 18.51 GB |
For MXFP8, vLLM reported 8.94 GB as the concrete MPS KV-cache capacity inside the larger 18.51-GB logical cache budget. Its default 0.92 memory-utilization target provisioned capacity for roughly ten concurrent 8K requests, although the benchmark used at most four short requests. That reservation is serving capacity, not weight size.
## Why vLLM-Metal is slower at batch 1
- Ollama uses a tightly integrated Qwen-specific MLX runner and a newer/custom MLX engine revision.
- vLLM-Metal adds scheduling, paged KV-cache management, request state, block-table translation, and hybrid Gated DeltaNet state management.
- Qwen3.8 has 16 SDPA and 48 linear/GDN layers. vLLM-Metal labels this hybrid paged-attention path experimental and translates its large logical blocks into Metal-kernel-compatible blocks.
- Some shapes can use lazy/fused Metal kernels; unsupported cases use less favorable convolution or recurrent fallback paths.
- These costs buy batching: vLLM loses single-stream latency but wins aggregate throughput and tail TTFT at concurrency four.
- The direct exact-checkpoint result confirms measurable vLLM overhead for NVFP4. The much larger Ollama advantage in long coding runs also includes different packed artifacts and different runtime integration, so it cannot be assigned exclusively to the scheduler.
## Token-count check
The runtimes did not always stop at the same token count because the packed artifacts generate different output and encounter EOS differently:
| Workload | Ollama output tokens | vLLM-Metal output tokens |
|---|---:|---:|
| Five measured requests, concurrency 1 | 576 | 640 |
| Twenty measured requests, concurrency 4 | 2,340 | 2,556 |
The fixed short prompt ended at 96–98 tokens in Ollama and at 124–128 tokens in vLLM-Metal. Therefore raw total latency alone is not a fair speed metric. Decode tokens per second and aggregate output tokens per second normalize for this difference and support the same conclusions.
## Test protocol
- Host: MacBook Pro, Apple M4 Max, 64 GB unified memory, macOS 15.6, arm64
- Ollama: 0.32.14, native MLX execution
- vLLM core: 0.28.0+cpu with native `vllm-metal` 0.4.0 development plugin; logs confirmed `Device(gpu, 0)` and MPS
- Context limit: 8,192 tokens
- Sampling: temperature 0, seed 42
- Output limit: 128 tokens
- Two warmup groups, then five measured groups at concurrency 1 and 4
- Three fixed raw-completion prompts
- Ollama speculative drafting explicitly disabled
- Runtimes loaded and tested sequentially to prevent unified-memory contention
The `+cpu` suffix belongs to the vLLM core wheel. In this setup, the Metal plugin supplied GPU execution; this was not a CPU benchmark.
## Model-equivalence limitation
- Ollama package: `qwen3.8:27b-mlx`
- vLLM-Metal package: `mlx-community/Qwen3.8-27B-nvfp4`
- Hugging Face revision: `5ff8ef173ad0d7c3aae92f0be43031a6ab8067c6`
- Both are based on Qwen3.8-27B and declare NVFP4 group-size-16 quantization.
- They are not byte-identical. A corresponding 44,564,480-byte packed weight tensor had different SHA-256 hashes.
This is a comparison of the closest practical native runtime/model packages, not a pure runtime-overhead comparison using identical packed weights. Quantization/conversion differences can account for some of the speed difference.
## Cold start
Cold startup was measured separately after both model packages were cached locally:
| Measurement | Ollama | vLLM-Metal |
|---|---:|---:|
| Model-weight load | 3.245 s | 3.25 s |
| Operational readiness | 3.455 s for a cold one-token request | approximately 28.5 s from process launch to API ready |
The model-weight load itself was effectively tied. vLLM-Metal additionally spent 9.71 seconds profiling, allocating its 15-GB KV cache, and warming kernels; Python process and plugin startup account for the remaining launch time. Ollama's daemon was already running, so its value measures unloaded-model-to-response rather than daemon process launch. These operational measurements are useful but not perfectly symmetric.
The first vLLM setup took about 12 minutes 45 seconds to download and reconstruct the 16.1-GB checkpoint. That one-time network cost is excluded from the cached cold-start and all warm-inference results.
## Caveats
- Prompt-throughput values from Ollama were excluded because repeated short prompts produced cache-hit durations near zero; vLLM's completion stream did not expose equivalent per-request prefill duration.
- Five groups establish a clear directional result, but longer multi-run tests are appropriate before capacity planning.
- Both runtimes used prefix caching as shipped. That reflects practical warmed operation, not cache-disabled kernel isolation.
- Total latency is influenced by differing EOS positions; normalized token rates are more comparable.
- Ollama and Hugging Face model pairs share the Qwen3.8-27B base and nominal quantization mode but are not byte-identical. Results compare practical deployable packages, not only runtime code.
- Direct MLX controls are batch-1 controls only; MLX-LM does not reproduce vLLM's concurrent serving objective.
## Evidence
- vLLM-Metal: `results/20260831T114727Z/metadata.json`, `raw.json`, `raw.csv`, and `summary.json`
- Ollama: `results/20260831T114928Z/metadata.json`, `raw.json`, `raw.csv`, and `summary.json`
- Cold start: `results/cold_start.json`
- Backend proof: `results/backend_verification.json`
- Ollama MXFP8: `results/20260831T121855Z/`
- vLLM-Metal MXFP8: `results/20260831T132015Z/`
- NVFP4 coding: `coding-results/20260831T120914Z-ollama-qwen3.8_27b-mlx/` and `coding-results/20260831T121823Z-vllm-metal-qwen3.8-27b-nvfp4/`
- MXFP8 coding: `coding-results/20260831T123006Z-ollama-qwen3.8_27b-mxfp8/` and `coding-results/20260831T133452Z-vllm-metal-qwen3.8-27b-mxfp8/`
- Direct MLX controls: `direct-results/20260831T133636Z-qwen3.8-27b-nvfp4/` and `direct-results/20260831T133721Z-qwen3.8-27b-mxfp8/`
- Back-to-back rerun: `results/20260831T134334Z/` and `direct-results/20260831T134407Z-qwen3.8-27b-nvfp4-rerun/`

500
benchmarks/inference/benchmark.py Normal file
View File

@@ -0,0 +1,500 @@
#!/usr/bin/env python3
"""Reproducible HTTP benchmark for Ollama and OpenAI-compatible runtimes."""
from __future__ import annotations
import argparse
import concurrent.futures
import csv
import json
import math
import platform
import statistics
import subprocess
import sys
import threading
import time
import urllib.error
import urllib.request
import uuid
from dataclasses import asdict, dataclass
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
PROMPTS = {
"short": (
"In exactly four concise bullet points, explain why database transactions are useful. "
"Do not use a heading."
),
"medium": (
"You are reviewing a Laravel service that transfers money between two accounts. "
"Explain the failure modes involving concurrency, partial writes, retries, and audit logs. "
"Then provide a robust implementation plan. Keep the response technical and concise."
),
"code": (
"Write a Python 3.12 function named merge_intervals that accepts a list of integer pairs, "
"validates each pair, merges overlapping or adjacent intervals, and returns sorted tuples. "
"Include type hints, a short docstring, and five assert-based examples. Output only code."
),
}
CSV_FIELDS = (
"run_id",
"runtime",
"model",
"phase",
"concurrency",
"round",
"slot",
"request_id",
"prompt_name",
"prompt_chars",
"max_tokens",
"started_at",
"ttft_s",
"total_s",
"prompt_tokens",
"output_tokens",
"prompt_tps",
"decode_tps",
"request_tps",
"group_elapsed_s",
"group_output_tps",
"error",
)
@dataclass(frozen=True)
class RuntimeConfig:
name: str
kind: str
base_url: str
model: str
@dataclass
class Measurement:
run_id: str
runtime: str
model: str
phase: str
concurrency: int
round: int
slot: int
request_id: str
prompt_name: str
prompt_chars: int
max_tokens: int
started_at: str
ttft_s: float | None = None
total_s: float | None = None
prompt_tokens: int | None = None
output_tokens: int | None = None
prompt_tps: float | None = None
decode_tps: float | None = None
request_tps: float | None = None
group_elapsed_s: float | None = None
group_output_tps: float | None = None
error: str | None = None
def utc_now() -> str:
return datetime.now(UTC).isoformat()
def percentile(values: list[float], fraction: float) -> float | None:
if not values:
return None
ordered = sorted(values)
position = (len(ordered) - 1) * fraction
lower = math.floor(position)
upper = math.ceil(position)
if lower == upper:
return ordered[lower]
return ordered[lower] + (ordered[upper] - ordered[lower]) * (position - lower)
def safe_version(command: list[str]) -> str | None:
try:
result = subprocess.run(command, capture_output=True, text=True, timeout=15, check=False)
except (OSError, subprocess.SubprocessError):
return None
value = (result.stdout or result.stderr).strip()
return value or None
def http_json(url: str, timeout: float = 15) -> dict[str, Any]:
request = urllib.request.Request(url, headers={"Accept": "application/json"})
with urllib.request.urlopen(request, timeout=timeout) as response:
return json.load(response)
def read_ndjson(response: Any, started: float) -> tuple[float | None, dict[str, Any], int]:
first_token_at: float | None = None
output_chars = 0
terminal: dict[str, Any] = {}
for raw_line in response:
if not raw_line.strip():
continue
event = json.loads(raw_line)
content = event.get("response", "")
if content:
output_chars += len(content)
if first_token_at is None:
first_token_at = time.perf_counter()
if event.get("done"):
terminal = event
return (first_token_at - started if first_token_at else None, terminal, output_chars)
def read_sse(response: Any, started: float) -> tuple[float | None, dict[str, Any], int]:
first_token_at: float | None = None
output_chars = 0
terminal: dict[str, Any] = {}
for raw_line in response:
line = raw_line.decode("utf-8").strip()
if not line.startswith("data:"):
continue
payload = line[5:].strip()
if payload == "[DONE]":
break
event = json.loads(payload)
terminal = event
choices = event.get("choices") or []
if choices:
text = choices[0].get("text") or choices[0].get("delta", {}).get("content") or ""
if text:
output_chars += len(text)
if first_token_at is None:
first_token_at = time.perf_counter()
return (first_token_at - started if first_token_at else None, terminal, output_chars)
def benchmark_request(
runtime: RuntimeConfig,
prompt_name: str,
prompt: str,
max_tokens: int,
seed: int,
timeout: float,
run_id: str,
phase: str,
concurrency: int,
round_number: int,
slot: int,
) -> Measurement:
measurement = Measurement(
run_id=run_id,
runtime=runtime.name,
model=runtime.model,
phase=phase,
concurrency=concurrency,
round=round_number,
slot=slot,
request_id=str(uuid.uuid4()),
prompt_name=prompt_name,
prompt_chars=len(prompt),
max_tokens=max_tokens,
started_at=utc_now(),
)
headers = {"Content-Type": "application/json", "Accept": "text/event-stream"}
if runtime.kind == "ollama":
url = f"{runtime.base_url.rstrip('/')}/api/generate"
payload = {
"model": runtime.model,
"prompt": prompt,
"stream": True,
"raw": True,
"keep_alive": "30m",
"options": {
"temperature": 0,
"seed": seed,
"num_predict": max_tokens,
"num_ctx": 8192,
"draft_num_predict": 0,
},
}
else:
url = f"{runtime.base_url.rstrip('/')}/v1/completions"
payload = {
"model": runtime.model,
"prompt": prompt,
"stream": True,
"stream_options": {"include_usage": True},
"temperature": 0,
"seed": seed,
"max_tokens": max_tokens,
}
started = time.perf_counter()
try:
request = urllib.request.Request(
url,
data=json.dumps(payload).encode("utf-8"),
headers=headers,
method="POST",
)
with urllib.request.urlopen(request, timeout=timeout) as response:
if runtime.kind == "ollama":
ttft, terminal, output_chars = read_ndjson(response, started)
measurement.prompt_tokens = terminal.get("prompt_eval_count")
measurement.output_tokens = terminal.get("eval_count")
prompt_ns = terminal.get("prompt_eval_duration")
decode_ns = terminal.get("eval_duration")
if prompt_ns and measurement.prompt_tokens:
measurement.prompt_tps = measurement.prompt_tokens / (prompt_ns / 1e9)
if decode_ns and measurement.output_tokens:
measurement.decode_tps = measurement.output_tokens / (decode_ns / 1e9)
else:
ttft, terminal, output_chars = read_sse(response, started)
usage = terminal.get("usage") or {}
measurement.prompt_tokens = usage.get("prompt_tokens")
measurement.output_tokens = usage.get("completion_tokens")
measurement.ttft_s = ttft
measurement.total_s = time.perf_counter() - started
if measurement.output_tokens and measurement.total_s:
measurement.request_tps = measurement.output_tokens / measurement.total_s
if measurement.decode_tps is None and measurement.ttft_s is not None:
decode_time = measurement.total_s - measurement.ttft_s
if decode_time > 0:
measurement.decode_tps = measurement.output_tokens / decode_time
if output_chars == 0:
measurement.error = "stream contained no generated text"
except urllib.error.HTTPError as error:
detail = error.read().decode("utf-8", errors="replace")[:1000]
measurement.error = f"HTTP {error.code}: {detail}"
measurement.total_s = time.perf_counter() - started
except (OSError, ValueError, json.JSONDecodeError) as error:
measurement.error = f"{type(error).__name__}: {error}"
measurement.total_s = time.perf_counter() - started
return measurement
def run_round(
runtime: RuntimeConfig,
prompt_items: list[tuple[str, str]],
max_tokens: int,
seed: int,
timeout: float,
run_id: str,
phase: str,
concurrency: int,
round_number: int,
) -> tuple[list[Measurement], float]:
barrier = threading.Barrier(concurrency)
def execute(slot: int) -> Measurement:
prompt_name, prompt = prompt_items[(round_number * concurrency + slot) % len(prompt_items)]
barrier.wait()
return benchmark_request(
runtime,
prompt_name,
prompt,
max_tokens,
seed,
timeout,
run_id,
phase,
concurrency,
round_number,
slot,
)
group_started = time.perf_counter()
with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as executor:
measurements = list(executor.map(execute, range(concurrency)))
group_elapsed = time.perf_counter() - group_started
successful_tokens = sum(item.output_tokens or 0 for item in measurements if not item.error)
group_tps = successful_tokens / group_elapsed if group_elapsed else 0.0
for item in measurements:
item.group_elapsed_s = group_elapsed
item.group_output_tps = group_tps
return measurements, group_elapsed
def summarize(measurements: list[Measurement]) -> list[dict[str, Any]]:
groups: dict[tuple[str, int], list[Measurement]] = {}
for measurement in measurements:
if measurement.phase != "measured":
continue
groups.setdefault((measurement.runtime, measurement.concurrency), []).append(measurement)
summaries: list[dict[str, Any]] = []
for (runtime, concurrency), items in sorted(groups.items()):
successful = [item for item in items if not item.error]
summary: dict[str, Any] = {
"runtime": runtime,
"model": items[0].model,
"concurrency": concurrency,
"requests": len(items),
"successes": len(successful),
"failures": len(items) - len(successful),
}
for field in ("ttft_s", "total_s", "prompt_tps", "decode_tps", "request_tps"):
values = [
float(value)
for item in successful
if (value := getattr(item, field)) is not None
]
summary[f"{field}_mean"] = statistics.fmean(values) if values else None
summary[f"{field}_p50"] = percentile(values, 0.50)
summary[f"{field}_p95"] = percentile(values, 0.95)
group_tps = [
float(item.group_output_tps)
for item in successful
if item.slot == 0 and item.group_output_tps is not None
]
summary["aggregate_output_tps_mean"] = statistics.fmean(group_tps) if group_tps else None
summary["aggregate_output_tps_p50"] = percentile(group_tps, 0.50)
summary["aggregate_output_tps_p95"] = percentile(group_tps, 0.95)
summaries.append(summary)
return summaries
def write_csv(path: Path, measurements: list[Measurement]) -> None:
with path.open("w", newline="", encoding="utf-8") as handle:
writer = csv.DictWriter(handle, fieldnames=CSV_FIELDS, lineterminator="\n")
writer.writeheader()
for measurement in measurements:
writer.writerow(asdict(measurement))
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output-dir", type=Path, default=Path(__file__).parent / "results")
parser.add_argument("--runtime", choices=("all", "ollama", "vllm"), default="all")
parser.add_argument("--ollama-url", default="http://127.0.0.1:11434")
parser.add_argument("--ollama-model", default="qwen3.8:27b-mlx")
parser.add_argument("--vllm-url", default="http://127.0.0.1:8001")
parser.add_argument("--vllm-model", default="qwen3.8-27b-nvfp4")
parser.add_argument("--warmups", type=int, default=2)
parser.add_argument("--rounds", type=int, default=5)
parser.add_argument("--concurrency", type=int, nargs="+", default=(1, 4))
parser.add_argument("--max-tokens", type=int, default=128)
parser.add_argument("--seed", type=int, default=42)
parser.add_argument("--timeout", type=float, default=600)
return parser.parse_args()
def main() -> int:
args = parse_args()
if args.warmups < 0 or args.rounds < 1 or any(value < 1 for value in args.concurrency):
raise SystemExit("warmups must be >= 0; rounds and concurrency must be >= 1")
runtimes = {
"ollama": RuntimeConfig("ollama", "ollama", args.ollama_url, args.ollama_model),
"vllm": RuntimeConfig("vllm-metal", "openai", args.vllm_url, args.vllm_model),
}
selected = list(runtimes.values()) if args.runtime == "all" else [runtimes[args.runtime]]
for runtime in selected:
health_url = (
f"{runtime.base_url.rstrip('/')}/api/tags"
if runtime.kind == "ollama"
else f"{runtime.base_url.rstrip('/')}/v1/models"
)
try:
http_json(health_url)
except Exception as error:
raise SystemExit(f"{runtime.name} is not ready at {health_url}: {error}") from error
run_id = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
output_dir = args.output_dir / run_id
output_dir.mkdir(parents=True, exist_ok=False)
metadata = {
"run_id": run_id,
"created_at": utc_now(),
"host": {
"platform": platform.platform(),
"machine": platform.machine(),
"python": platform.python_version(),
},
"arguments": vars(args) | {"output_dir": str(args.output_dir)},
"runtimes": [asdict(runtime) for runtime in selected],
"versions": {
"ollama": safe_version(["ollama", "--version"]),
"vllm": safe_version([str(Path.home() / ".venv-vllm-metal/bin/vllm"), "--version"]),
},
"model_metadata": {},
"prompts": PROMPTS,
"measurement_notes": {
"ttft": (
"Client wall time from request start until first non-empty generated text chunk."
),
"total": "Client wall time from request start until stream completion.",
"decode_tps": "Ollama server eval metric; otherwise output tokens / (total - TTFT).",
"aggregate_output_tps": "Sum of successful output tokens / concurrent group wall time.",
},
}
for runtime in selected:
url = (
f"{runtime.base_url.rstrip('/')}/api/show"
if runtime.kind == "ollama"
else f"{runtime.base_url.rstrip('/')}/v1/models"
)
try:
if runtime.kind == "ollama":
request = urllib.request.Request(
url,
data=json.dumps({"model": runtime.model}).encode(),
headers={"Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(request, timeout=30) as response:
metadata["model_metadata"][runtime.name] = json.load(response)
else:
metadata["model_metadata"][runtime.name] = http_json(url)
except Exception as error:
metadata["model_metadata"][runtime.name] = {"metadata_error": str(error)}
prompt_items = list(PROMPTS.items())
measurements: list[Measurement] = []
total_groups = len(selected) * sum(args.warmups + args.rounds for _ in args.concurrency)
completed_groups = 0
for runtime in selected:
for concurrency in args.concurrency:
for phase, rounds in (("warmup", args.warmups), ("measured", args.rounds)):
for round_number in range(rounds):
items, elapsed = run_round(
runtime,
prompt_items,
args.max_tokens,
args.seed,
args.timeout,
run_id,
phase,
concurrency,
round_number,
)
measurements.extend(items)
completed_groups += 1
errors = sum(bool(item.error) for item in items)
print(
f"[{completed_groups}/{total_groups}] {runtime.name} phase={phase} "
f"concurrency={concurrency} round={round_number + 1}/{rounds} "
f"elapsed={elapsed:.3f}s errors={errors}",
flush=True,
)
summaries = summarize(measurements)
metadata["completed_at"] = utc_now()
metadata["summary"] = summaries
(output_dir / "raw.json").write_text(
json.dumps([asdict(item) | {
"group_elapsed_s": getattr(item, "group_elapsed_s", None),
"group_output_tps": getattr(item, "group_output_tps", None),
} for item in measurements], indent=2),
encoding="utf-8",
)
(output_dir / "metadata.json").write_text(
json.dumps(metadata, indent=2, default=str), encoding="utf-8"
)
(output_dir / "summary.json").write_text(json.dumps(summaries, indent=2), encoding="utf-8")
write_csv(output_dir / "raw.csv", measurements)
print(json.dumps(summaries, indent=2))
print(f"Results: {output_dir}")
return 1 if any(item.error for item in measurements if item.phase == "measured") else 0
if __name__ == "__main__":
sys.exit(main())

111
benchmarks/inference/coding-results/20260831T120914Z-ollama-qwen3.8_27b-mlx/results.json Normal file
View File

File diff suppressed because one or more lines are too long

53
benchmarks/inference/coding-results/20260831T120914Z-ollama-qwen3.8_27b-mlx/summary.json Normal file
View File

@@ -0,0 +1,53 @@
[
{
"task": "python_async_cache",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"started_at": "2026-08-31T12:05:23.192111+00:00",
"prompt_chars": 857,
"max_tokens": 3072,
"ttft_s": 1.021859125001356,
"total_s": 56.40731879102532,
"prompt_tokens": 211,
"output_tokens": 3072,
"decode_tps": 55.37840341105521,
"finish_reason": "length",
"output_chars": 12165,
"output_sha256": "a3355a481519820f9021b6846e3f784fba8aa922d3317c5a84d6762447d9dd97",
"error": null
},
{
"task": "php_idempotent_transfer",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"started_at": "2026-08-31T12:06:19.598953+00:00",
"prompt_chars": 756,
"max_tokens": 3072,
"ttft_s": 1.2263128339545801,
"total_s": 78.72023624996655,
"prompt_tokens": 182,
"output_tokens": 3072,
"decode_tps": 39.58218973701569,
"finish_reason": "length",
"output_chars": 12654,
"output_sha256": "070acfea480b76f0913ce21686f2031f7a32421f5645f28eebe0c2ef9d2728ec",
"error": null
},
{
"task": "typescript_dag_scheduler",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"started_at": "2026-08-31T12:07:38.318694+00:00",
"prompt_chars": 773,
"max_tokens": 3072,
"ttft_s": 1.5556320829782635,
"total_s": 96.54028320801444,
"prompt_tokens": 183,
"output_tokens": 3072,
"decode_tps": 32.29928081838954,
"finish_reason": "length",
"output_chars": 11211,
"output_sha256": "f97ad0b443b4fe7ae9a5b2a3f0cc3c8ad2b71bc2ba4d5849bd679b498488c654",
"error": null
}
]

111
benchmarks/inference/coding-results/20260831T121823Z-vllm-metal-qwen3.8-27b-nvfp4/results.json Normal file
View File

File diff suppressed because one or more lines are too long

53
benchmarks/inference/coding-results/20260831T121823Z-vllm-metal-qwen3.8-27b-nvfp4/summary.json Normal file
View File

@@ -0,0 +1,53 @@
[
{
"task": "python_async_cache",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"started_at": "2026-08-31T12:10:51.466896+00:00",
"prompt_chars": 857,
"max_tokens": 3072,
"ttft_s": 1.0361592500121333,
"total_s": 133.43742824997753,
"prompt_tokens": 211,
"output_tokens": 3072,
"decode_tps": 23.20219453486358,
"finish_reason": "length",
"output_chars": 14058,
"output_sha256": "a18e80f1852ecc0c70b987f244870b8a2dc20db8ad42505280162a4b17dbd057",
"error": null
},
{
"task": "php_idempotent_transfer",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"started_at": "2026-08-31T12:13:04.907109+00:00",
"prompt_chars": 756,
"max_tokens": 3072,
"ttft_s": 15.930970875022467,
"total_s": 166.00547937501688,
"prompt_tokens": 182,
"output_tokens": 2840,
"decode_tps": 18.923933374068692,
"finish_reason": "stop",
"output_chars": 11403,
"output_sha256": "bb758808663b88bb3fbacc52df49598eb21aa8bc7fa0ee8466410f0ae13e8b0d",
"error": null
},
{
"task": "typescript_dag_scheduler",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"started_at": "2026-08-31T12:15:50.913910+00:00",
"prompt_chars": 773,
"max_tokens": 3072,
"ttft_s": 1.4814700839924626,
"total_s": 152.22010587499244,
"prompt_tokens": 183,
"output_tokens": 3072,
"decode_tps": 20.379645761550783,
"finish_reason": "length",
"output_chars": 10869,
"output_sha256": "f700d51c471a59acfcf85016d416f0c5354662552fea44b48f56d80c3afdcb59",
"error": null
}
]

111
benchmarks/inference/coding-results/20260831T123006Z-ollama-qwen3.8_27b-mxfp8/results.json Normal file
View File

File diff suppressed because one or more lines are too long

53
benchmarks/inference/coding-results/20260831T123006Z-ollama-qwen3.8_27b-mxfp8/summary.json Normal file
View File

@@ -0,0 +1,53 @@
[
{
"task": "python_async_cache",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"started_at": "2026-08-31T12:22:39.260933+00:00",
"prompt_chars": 857,
"max_tokens": 3072,
"ttft_s": 1.1107976250350475,
"total_s": 149.8235780420364,
"prompt_tokens": 211,
"output_tokens": 3072,
"decode_tps": 20.63948064778098,
"finish_reason": "length",
"output_chars": 13492,
"output_sha256": "3e5ca7654fc69293040b7cc1b554ccddbd1a99b82710d361074fee260803182f",
"error": null
},
{
"task": "php_idempotent_transfer",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"started_at": "2026-08-31T12:25:09.083177+00:00",
"prompt_chars": 756,
"max_tokens": 3072,
"ttft_s": 1.6688512500259094,
"total_s": 149.76781020802446,
"prompt_tokens": 182,
"output_tokens": 3072,
"decode_tps": 20.71737613932144,
"finish_reason": "length",
"output_chars": 12642,
"output_sha256": "71ee449c3f573828250542e04f7a63db5a8d23087bdec89f0a9a0bb31d52a3b5",
"error": null
},
{
"task": "typescript_dag_scheduler",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"started_at": "2026-08-31T12:27:38.904048+00:00",
"prompt_chars": 773,
"max_tokens": 3072,
"ttft_s": 1.6032058749697171,
"total_s": 147.27845333400182,
"prompt_tokens": 183,
"output_tokens": 3072,
"decode_tps": 21.060449812696582,
"finish_reason": "length",
"output_chars": 11368,
"output_sha256": "a3d74aa8a8183babda562bc53b32dda3b1b3ad318788d2384cc8babff3df3223",
"error": null
}
]

111
benchmarks/inference/coding-results/20260831T133452Z-vllm-metal-qwen3.8-27b-mxfp8/results.json Normal file
View File

File diff suppressed because one or more lines are too long

53
benchmarks/inference/coding-results/20260831T133452Z-vllm-metal-qwen3.8-27b-mxfp8/summary.json Normal file
View File

@@ -0,0 +1,53 @@
[
{
"task": "python_async_cache",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"started_at": "2026-08-31T13:22:57.038624+00:00",
"prompt_chars": 857,
"max_tokens": 3072,
"ttft_s": 1.1419034169521183,
"total_s": 247.70563520898577,
"prompt_tokens": 211,
"output_tokens": 3072,
"decode_tps": 12.459253344652916,
"finish_reason": "length",
"output_chars": 13028,
"output_sha256": "699045091c4f5bc19d0f8f0bbc676b164512c8e3e7dec0f95acb13b5bef2f4ba",
"error": null
},
{
"task": "php_idempotent_transfer",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"started_at": "2026-08-31T13:27:04.745948+00:00",
"prompt_chars": 756,
"max_tokens": 3072,
"ttft_s": 1.6236594580113888,
"total_s": 242.9034315830213,
"prompt_tokens": 182,
"output_tokens": 3072,
"decode_tps": 12.73210751545455,
"finish_reason": "length",
"output_chars": 11719,
"output_sha256": "64e2a1896826c7e6a38540d8e7d2650e66be4c3df39a3a63ca49261d90eeea17",
"error": null
},
{
"task": "typescript_dag_scheduler",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"started_at": "2026-08-31T13:31:07.651240+00:00",
"prompt_chars": 773,
"max_tokens": 3072,
"ttft_s": 1.4923370420001447,
"total_s": 224.71278991695726,
"prompt_tokens": 183,
"output_tokens": 3072,
"decode_tps": 13.76217976638934,
"finish_reason": "length",
"output_chars": 10877,
"output_sha256": "bb305dfdb203bcf6d40071cb30ae92561f0b39bf38f52973d3f27e45cd5ff0a7",
"error": null
}
]

364
benchmarks/inference/coding_benchmark.py Normal file
View File

@@ -0,0 +1,364 @@
#!/usr/bin/env python3
"""Measure end-to-end latency for deterministic, complex programming tasks."""
from __future__ import annotations
import argparse
import hashlib
import json
import platform
import subprocess
import sys
import time
import urllib.error
import urllib.request
from dataclasses import asdict, dataclass
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
TASKS = {
"python_async_cache": """
Act as a senior Python engineer. Implement a production-ready, single-file Python 3.12 module
containing a generic AsyncTTLCache[K, V]. Requirements: weighted LRU capacity; per-entry TTL based
on time.monotonic; single-flight get_or_create so concurrent misses for one key invoke the async
factory exactly once; cancellation of one waiter must not cancel the shared factory; exceptions are
never cached; invalidating a key while its factory is running must prevent stale publication; never
hold the cache lock while awaiting user code; clear(), invalidate(), and immutable hit/miss/eviction
statistics. Include concise pytest-asyncio tests covering races, cancellation, expiry, weighted
capacity, factory failure, and invalidation during creation. Use only the standard library plus
pytest. Output only the complete module, without Markdown fences or prose.
""".strip(),
"php_idempotent_transfer": """
Act as a senior Laravel 12 engineer. Produce one self-contained PHP file that demonstrates an
idempotent money-transfer service and its PHPUnit tests. Store money as integer cents. Requirements:
validate positive amount and distinct accounts; use a database transaction; lock both account rows
in deterministic ID order; reject insufficient funds; use a unique client idempotency key so retries
return the original transfer; retry deadlocks without duplicating effects; persist an outbox event
in the same transaction; and prove with tests that duplicate and competing requests preserve
balances, create one transfer, and create one outbox event. Include minimal model/table assumptions
as comments.
Output only valid PHP, without Markdown fences or prose.
""".strip(),
"typescript_dag_scheduler": """
Act as a senior TypeScript engineer. Implement a strict-mode, dependency-free DAG task scheduler in
one TypeScript file. Each task has an ID, dependency IDs, and an async run(signal) function.
Requirements: validate unknown dependencies and duplicate IDs; report an explicit cycle path; stable
input-order scheduling; configurable maximum concurrency; AbortSignal support; fail fast after the
first error while awaiting already-running tasks; never start dependents of failed tasks; return a
Map of typed results on success; and avoid unhandled promise rejections. Include a compact test
harness covering a diamond graph, concurrency enforcement, stable order, cycle detection, external
cancellation, and rejection. Output only TypeScript, without Markdown fences or prose.
""".strip(),
}
@dataclass(frozen=True)
class Runtime:
name: str
kind: str
base_url: str
model: str
@dataclass
class Result:
task: str
runtime: str
model: str
started_at: str
prompt_chars: int
max_tokens: int
ttft_s: float | None = None
total_s: float | None = None
prompt_tokens: int | None = None
output_tokens: int | None = None
decode_tps: float | None = None
finish_reason: str | None = None
output_chars: int = 0
output_sha256: str | None = None
output: str = ""
error: str | None = None
def utc_now() -> str:
return datetime.now(UTC).isoformat()
def post_json(url: str, payload: dict[str, Any], timeout: float) -> dict[str, Any]:
request = urllib.request.Request(
url,
data=json.dumps(payload).encode("utf-8"),
headers={"Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(request, timeout=timeout) as response:
return json.load(response)
def health(runtime: Runtime) -> None:
path = "/api/tags" if runtime.kind == "ollama" else "/v1/models"
with urllib.request.urlopen(f"{runtime.base_url.rstrip('/')}{path}", timeout=15):
pass
def consume_ollama(response: Any, started: float) -> tuple[float | None, str, dict[str, Any]]:
first_at: float | None = None
chunks: list[str] = []
terminal: dict[str, Any] = {}
for line in response:
if not line.strip():
continue
event = json.loads(line)
text = event.get("message", {}).get("content") or event.get("response") or ""
if text:
if first_at is None:
first_at = time.perf_counter()
chunks.append(text)
if event.get("done"):
terminal = event
return (first_at - started if first_at else None, "".join(chunks), terminal)
def consume_sse(
response: Any, started: float
) -> tuple[float | None, str, dict[str, Any], str | None]:
first_at: float | None = None
chunks: list[str] = []
terminal: dict[str, Any] = {}
finish_reason: str | None = None
for raw_line in response:
line = raw_line.decode("utf-8").strip()
if not line.startswith("data:"):
continue
payload = line[5:].strip()
if payload == "[DONE]":
break
event = json.loads(payload)
terminal = event
for choice in event.get("choices") or []:
text = choice.get("text") or choice.get("delta", {}).get("content") or ""
if text:
if first_at is None:
first_at = time.perf_counter()
chunks.append(text)
finish_reason = choice.get("finish_reason") or finish_reason
return (first_at - started if first_at else None, "".join(chunks), terminal, finish_reason)
def request_payload(
runtime: Runtime, prompt: str, max_tokens: int, seed: int
) -> tuple[str, dict[str, Any]]:
if runtime.kind == "ollama":
return f"{runtime.base_url.rstrip('/')}/api/chat", {
"model": runtime.model,
"messages": [
{
"role": "system",
"content": (
"You are a senior software engineer. Follow every output constraint "
"exactly and return a complete solution."
),
},
{"role": "user", "content": prompt},
],
"think": False,
"stream": True,
"keep_alive": "30m",
"options": {
"temperature": 0,
"seed": seed,
"num_predict": max_tokens,
"num_ctx": 8192,
"draft_num_predict": 0,
},
}
return f"{runtime.base_url.rstrip('/')}/v1/chat/completions", {
"model": runtime.model,
"messages": [
{
"role": "system",
"content": (
"You are a senior software engineer. Follow every output constraint exactly "
"and return a complete solution."
),
},
{"role": "user", "content": prompt},
],
"stream": True,
"stream_options": {"include_usage": True},
"chat_template_kwargs": {"enable_thinking": False},
"temperature": 0,
"seed": seed,
"max_tokens": max_tokens,
}
def run_task(
runtime: Runtime,
task_name: str,
prompt: str,
max_tokens: int,
seed: int,
timeout: float,
) -> Result:
result = Result(
task=task_name,
runtime=runtime.name,
model=runtime.model,
started_at=utc_now(),
prompt_chars=len(prompt),
max_tokens=max_tokens,
)
url, payload = request_payload(runtime, prompt, max_tokens, seed)
request = urllib.request.Request(
url,
data=json.dumps(payload).encode("utf-8"),
headers={"Content-Type": "application/json", "Accept": "text/event-stream"},
method="POST",
)
started = time.perf_counter()
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
if runtime.kind == "ollama":
result.ttft_s, result.output, terminal = consume_ollama(response, started)
result.prompt_tokens = terminal.get("prompt_eval_count")
result.output_tokens = terminal.get("eval_count")
result.finish_reason = terminal.get("done_reason")
decode_ns = terminal.get("eval_duration")
if decode_ns and result.output_tokens:
result.decode_tps = result.output_tokens / (decode_ns / 1e9)
else:
result.ttft_s, result.output, terminal, result.finish_reason = consume_sse(
response, started
)
usage = terminal.get("usage") or {}
result.prompt_tokens = usage.get("prompt_tokens")
result.output_tokens = usage.get("completion_tokens")
result.total_s = time.perf_counter() - started
if result.decode_tps is None and result.output_tokens and result.ttft_s is not None:
decode_s = result.total_s - result.ttft_s
if decode_s > 0:
result.decode_tps = result.output_tokens / decode_s
result.output_chars = len(result.output)
result.output_sha256 = hashlib.sha256(result.output.encode()).hexdigest()
if not result.output:
result.error = "empty generated output"
except urllib.error.HTTPError as error:
detail = error.read().decode("utf-8", errors="replace")[:2000]
result.error = f"HTTP {error.code}: {detail}"
result.total_s = time.perf_counter() - started
except (OSError, ValueError, json.JSONDecodeError) as error:
result.error = f"{type(error).__name__}: {error}"
result.total_s = time.perf_counter() - started
return result
def package_versions() -> dict[str, str | None]:
script = (
"import importlib.metadata as m,json; "
"print(json.dumps({p:m.version(p) for p in "
"['vllm','vllm-metal','mlx','mlx-lm','mlx-vlm']}))"
)
executable = Path.home() / ".venv-vllm-metal/bin/python"
try:
process = subprocess.run(
[str(executable), "-c", script], capture_output=True, text=True, timeout=30, check=True
)
return json.loads(process.stdout)
except (OSError, subprocess.SubprocessError, json.JSONDecodeError):
return {}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--runtime", choices=("ollama", "vllm"), required=True)
parser.add_argument("--ollama-url", default="http://127.0.0.1:11434")
parser.add_argument("--ollama-model", default="qwen3.8:27b-mlx")
parser.add_argument("--vllm-url", default="http://127.0.0.1:8001")
parser.add_argument("--vllm-model", default="qwen3.8-27b-nvfp4")
parser.add_argument("--max-tokens", type=int, default=1536)
parser.add_argument("--seed", type=int, default=42)
parser.add_argument("--timeout", type=float, default=900)
parser.add_argument("--output-dir", type=Path, default=Path(__file__).parent / "coding-results")
return parser.parse_args()
def main() -> int:
args = parse_args()
if args.max_tokens < 1:
raise SystemExit("max-tokens must be positive")
runtime = (
Runtime("ollama", "ollama", args.ollama_url, args.ollama_model)
if args.runtime == "ollama"
else Runtime("vllm-metal", "openai", args.vllm_url, args.vllm_model)
)
try:
health(runtime)
except Exception as error:
raise SystemExit(f"{runtime.name} is unavailable: {error}") from error
warmup = run_task(runtime, "warmup", "Output only the word READY.", 8, args.seed, args.timeout)
if warmup.error:
raise SystemExit(f"warmup failed: {warmup.error}")
print(
f"warmup runtime={runtime.name} total={warmup.total_s:.3f}s "
f"tokens={warmup.output_tokens}",
flush=True,
)
results: list[Result] = []
for index, (task_name, prompt) in enumerate(TASKS.items(), start=1):
result = run_task(runtime, task_name, prompt, args.max_tokens, args.seed, args.timeout)
results.append(result)
print(
f"[{index}/{len(TASKS)}] task={task_name} total={result.total_s:.3f}s "
f"ttft={result.ttft_s:.3f}s tokens={result.output_tokens} "
f"decode={result.decode_tps:.2f}tok/s finish={result.finish_reason} "
f"error={result.error or 'none'}",
flush=True,
)
run_id = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
output_dir = args.output_dir / f"{run_id}-{runtime.name}-{runtime.model.replace(':', '_')}"
output_dir.mkdir(parents=True, exist_ok=False)
raw = {
"run_id": run_id,
"created_at": utc_now(),
"runtime": asdict(runtime),
"host": {
"platform": platform.platform(),
"machine": platform.machine(),
"python": platform.python_version(),
},
"package_versions": package_versions(),
"settings": {
"temperature": 0,
"seed": args.seed,
"max_tokens": args.max_tokens,
"context_tokens": 8192,
"raw_completion": False,
"chat_template": True,
"thinking": False,
"ollama_speculative_drafting": False,
},
"warmup": asdict(warmup),
"tasks": TASKS,
"results": [asdict(result) for result in results],
}
(output_dir / "results.json").write_text(json.dumps(raw, indent=2), encoding="utf-8")
summary = [
{
key: value
for key, value in asdict(result).items()
if key not in {"output"}
}
for result in results
]
(output_dir / "summary.json").write_text(json.dumps(summary, indent=2), encoding="utf-8")
print(f"Results: {output_dir}")
return 1 if any(result.error for result in results) else 0
if __name__ == "__main__":
sys.exit(main())

48
benchmarks/inference/direct-results/20260831T133636Z-qwen3.8-27b-nvfp4/metadata.json Normal file
View File

@@ -0,0 +1,48 @@
{
"run_id": "20260831T133636Z",
"created_at": "2026-08-31T13:37:14.110488+00:00",
"host": {
"platform": "macOS-15.6-arm64-arm-64bit",
"machine": "arm64",
"python": "3.12.8"
},
"arguments": {
"model": "mlx-community/Qwen3.8-27B-nvfp4",
"label": "qwen3.8-27b-nvfp4",
"warmups": 2,
"rounds": 5,
"max_tokens": 128,
"seed": 42,
"output_dir": "/Users/christianmeyer/Documents/git/lai_automatisierung/benchmarks/inference/direct-results"
},
"load_s": 3.066661207994912,
"mlx_version": "0.32.0",
"device": "Device(gpu, 0)",
"prompts": {
"short": "In exactly four concise bullet points, explain why database transactions are useful. Do not use a heading.",
"medium": "You are reviewing a Laravel service that transfers money between two accounts. Explain the failure modes involving concurrency, partial writes, retries, and audit logs. Then provide a robust implementation plan. Keep the response technical and concise.",
"code": "Write a Python 3.12 function named merge_intervals that accepts a list of integer pairs, validates each pair, merges overlapping or adjacent intervals, and returns sorted tuples. Include type hints, a short docstring, and five assert-based examples. Output only code."
},
"summary": {
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4",
"requests": 5,
"successes": 5,
"ttft_s_mean": 0.4445558497915044,
"ttft_s_p50": 0.48620820802170783,
"ttft_s_p95": 0.518837732996326,
"total_s_mean": 4.9355954667902555,
"total_s_p50": 4.966667582979426,
"total_s_p95": 5.034370008390397,
"prompt_tps_mean": 107.94239735907567,
"prompt_tps_p50": 107.53363787342614,
"prompt_tps_p95": 135.27062378239268,
"decode_tps_mean": 28.570950408778582,
"decode_tps_p50": 28.53361993265224,
"decode_tps_p95": 28.826484879691197,
"request_tps_mean": 25.943347049499913,
"request_tps_p50": 25.771807325831702,
"request_tps_p95": 26.64316963019791,
"peak_memory_gb": 15.479377792
}
}

128
benchmarks/inference/direct-results/20260831T133636Z-qwen3.8-27b-nvfp4/raw.json Normal file
View File

@@ -0,0 +1,128 @@
[
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4",
"source": "mlx-community/Qwen3.8-27B-nvfp4",
"phase": "warmup",
"round": 1,
"prompt_name": "short",
"prompt_tokens": 20,
"output_tokens": 128,
"ttft_s": 0.6381363340187818,
"total_s": 5.067599917005282,
"prompt_tps": 46.64172306513033,
"decode_tps": 28.964400213729846,
"request_tps": 25.258505425906254,
"peak_memory_gb": 15.399916034,
"finish_reason": "length",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4",
"source": "mlx-community/Qwen3.8-27B-nvfp4",
"phase": "warmup",
"round": 2,
"prompt_name": "medium",
"prompt_tokens": 43,
"output_tokens": 128,
"ttft_s": 0.46635333297308534,
"total_s": 4.885915416001808,
"prompt_tps": 122.76309757245242,
"decode_tps": 29.032294926274737,
"request_tps": 26.19775192603388,
"peak_memory_gb": 15.453767792,
"finish_reason": "length",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4",
"source": "mlx-community/Qwen3.8-27B-nvfp4",
"phase": "measured",
"round": 1,
"prompt_name": "short",
"prompt_tokens": 20,
"output_tokens": 128,
"ttft_s": 0.3432638749945909,
"total_s": 4.786967416992411,
"prompt_tps": 90.81617646556052,
"decode_tps": 28.873430911706816,
"request_tps": 26.73926702438696,
"peak_memory_gb": 15.453767792,
"finish_reason": "length",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4",
"source": "mlx-community/Qwen3.8-27B-nvfp4",
"phase": "measured",
"round": 2,
"prompt_name": "medium",
"prompt_tokens": 43,
"output_tokens": 128,
"ttft_s": 0.48620820802170783,
"total_s": 4.966667582979426,
"prompt_tps": 118.1052752991172,
"decode_tps": 28.63870075162873,
"request_tps": 25.771807325831702,
"peak_memory_gb": 15.453784176,
"finish_reason": "length",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4",
"source": "mlx-community/Qwen3.8-27B-nvfp4",
"phase": "measured",
"round": 3,
"prompt_name": "code",
"prompt_tokens": 54,
"output_tokens": 128,
"ttft_s": 0.5125948329805396,
"total_s": 5.009093041997403,
"prompt_tps": 139.56196090321154,
"decode_tps": 28.53361993265224,
"request_tps": 25.553528139089888,
"peak_memory_gb": 15.479377792,
"finish_reason": "length",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4",
"source": "mlx-community/Qwen3.8-27B-nvfp4",
"phase": "measured",
"round": 4,
"prompt_name": "short",
"prompt_tokens": 20,
"output_tokens": 128,
"ttft_s": 0.36031387496041134,
"total_s": 4.874560041993391,
"prompt_tps": 83.69493625406292,
"decode_tps": 28.424103574919425,
"request_tps": 26.258780053441704,
"peak_memory_gb": 15.479377792,
"finish_reason": "length",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4",
"source": "mlx-community/Qwen3.8-27B-nvfp4",
"phase": "measured",
"round": 5,
"prompt_name": "medium",
"prompt_tokens": 43,
"output_tokens": 128,
"ttft_s": 0.5203984580002725,
"total_s": 5.040689249988645,
"prompt_tps": 107.53363787342614,
"decode_tps": 28.384896872985717,
"request_tps": 25.393352704749322,
"peak_memory_gb": 15.479377792,
"finish_reason": "length",
"error": null
}
]

22
benchmarks/inference/direct-results/20260831T133636Z-qwen3.8-27b-nvfp4/summary.json Normal file
View File

@@ -0,0 +1,22 @@
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4",
"requests": 5,
"successes": 5,
"ttft_s_mean": 0.4445558497915044,
"ttft_s_p50": 0.48620820802170783,
"ttft_s_p95": 0.518837732996326,
"total_s_mean": 4.9355954667902555,
"total_s_p50": 4.966667582979426,
"total_s_p95": 5.034370008390397,
"prompt_tps_mean": 107.94239735907567,
"prompt_tps_p50": 107.53363787342614,
"prompt_tps_p95": 135.27062378239268,
"decode_tps_mean": 28.570950408778582,
"decode_tps_p50": 28.53361993265224,
"decode_tps_p95": 28.826484879691197,
"request_tps_mean": 25.943347049499913,
"request_tps_p50": 25.771807325831702,
"request_tps_p95": 26.64316963019791,
"peak_memory_gb": 15.479377792
}

48
benchmarks/inference/direct-results/20260831T133721Z-qwen3.8-27b-mxfp8/metadata.json Normal file
View File

@@ -0,0 +1,48 @@
{
"run_id": "20260831T133721Z",
"created_at": "2026-08-31T13:38:28.661722+00:00",
"host": {
"platform": "macOS-15.6-arm64-arm-64bit",
"machine": "arm64",
"python": "3.12.8"
},
"arguments": {
"model": "mlx-community/Qwen3.8-27B-mxfp8",
"label": "qwen3.8-27b-mxfp8",
"warmups": 2,
"rounds": 5,
"max_tokens": 128,
"seed": 42,
"output_dir": "/Users/christianmeyer/Documents/git/lai_automatisierung/benchmarks/inference/direct-results"
},
"load_s": 5.387068083975464,
"mlx_version": "0.32.0",
"device": "Device(gpu, 0)",
"prompts": {
"short": "In exactly four concise bullet points, explain why database transactions are useful. Do not use a heading.",
"medium": "You are reviewing a Laravel service that transfers money between two accounts. Explain the failure modes involving concurrency, partial writes, retries, and audit logs. Then provide a robust implementation plan. Keep the response technical and concise.",
"code": "Write a Python 3.12 function named merge_intervals that accepts a list of integer pairs, validates each pair, merges overlapping or adjacent intervals, and returns sorted tuples. Include type hints, a short docstring, and five assert-based examples. Output only code."
},
"summary": {
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-mxfp8",
"requests": 5,
"successes": 5,
"ttft_s_mean": 0.6405866249930113,
"ttft_s_p50": 0.6006166669540107,
"ttft_s_p95": 0.993492508213967,
"total_s_mean": 8.97830267478712,
"total_s_p50": 8.952084249991458,
"total_s_p95": 9.543902474397328,
"prompt_tps_mean": 77.25518114106583,
"prompt_tps_p50": 69.803885982221,
"prompt_tps_p95": 108.54351756783554,
"decode_tps_mean": 15.141694584402668,
"decode_tps_p50": 15.417877788139895,
"decode_tps_p95": 15.90376325022775,
"request_tps_mean": 14.036390549836813,
"request_tps_p50": 14.298346220336581,
"request_tps_p95": 14.998850485179887,
"peak_memory_gb": 28.05746427
}
}

128
benchmarks/inference/direct-results/20260831T133721Z-qwen3.8-27b-mxfp8/raw.json Normal file
View File

@@ -0,0 +1,128 @@
[
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-mxfp8",
"source": "mlx-community/Qwen3.8-27B-mxfp8",
"phase": "warmup",
"round": 1,
"prompt_name": "short",
"prompt_tokens": 20,
"output_tokens": 122,
"ttft_s": 1.4950010409811512,
"total_s": 8.873082707985304,
"prompt_tps": 65.2268651572191,
"decode_tps": 16.569121716474896,
"request_tps": 13.749449206667087,
"peak_memory_gb": 27.989241503,
"finish_reason": "stop",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-mxfp8",
"source": "mlx-community/Qwen3.8-27B-mxfp8",
"phase": "warmup",
"round": 2,
"prompt_name": "medium",
"prompt_tokens": 43,
"output_tokens": 128,
"ttft_s": 0.5392710830201395,
"total_s": 8.420522083004471,
"prompt_tps": 104.9724920224018,
"decode_tps": 16.272217242824215,
"request_tps": 15.200957700514595,
"peak_memory_gb": 28.031514318,
"finish_reason": "length",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-mxfp8",
"source": "mlx-community/Qwen3.8-27B-mxfp8",
"phase": "measured",
"round": 1,
"prompt_name": "short",
"prompt_tokens": 20,
"output_tokens": 122,
"ttft_s": 0.41662891599116847,
"total_s": 8.074155374953989,
"prompt_tps": 69.803885982221,
"decode_tps": 15.96651063653789,
"request_tps": 15.109939595470719,
"peak_memory_gb": 28.031514318,
"finish_reason": "stop",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-mxfp8",
"source": "mlx-community/Qwen3.8-27B-mxfp8",
"phase": "measured",
"round": 2,
"prompt_name": "medium",
"prompt_tokens": 43,
"output_tokens": 128,
"ttft_s": 0.6006166669540107,
"total_s": 8.794534499989823,
"prompt_tps": 94.30109779838891,
"decode_tps": 15.652773704987183,
"request_tps": 14.554494044016558,
"peak_memory_gb": 28.031531726,
"finish_reason": "length",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-mxfp8",
"source": "mlx-community/Qwen3.8-27B-mxfp8",
"phase": "measured",
"round": 3,
"prompt_name": "code",
"prompt_tokens": 54,
"output_tokens": 128,
"ttft_s": 0.6342117090243846,
"total_s": 8.952084249991458,
"prompt_tps": 112.1041225101972,
"decode_tps": 15.417877788139895,
"request_tps": 14.298346220336581,
"peak_memory_gb": 28.05746427,
"finish_reason": "length",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-mxfp8",
"source": "mlx-community/Qwen3.8-27B-mxfp8",
"phase": "measured",
"round": 4,
"prompt_name": "short",
"prompt_tokens": 20,
"output_tokens": 122,
"ttft_s": 0.46816312498413026,
"total_s": 9.549591040995438,
"prompt_tps": 61.572468489546665,
"decode_tps": 13.468195538794124,
"request_tps": 12.775416190731752,
"peak_memory_gb": 28.05746427,
"finish_reason": "stop",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-mxfp8",
"source": "mlx-community/Qwen3.8-27B-mxfp8",
"phase": "measured",
"round": 5,
"prompt_name": "medium",
"prompt_tokens": 43,
"output_tokens": 128,
"ttft_s": 1.0833127080113627,
"total_s": 9.521148208004888,
"prompt_tps": 48.494330924975344,
"decode_tps": 15.203115253554254,
"request_tps": 13.443756698628452,
"peak_memory_gb": 28.05746427,
"finish_reason": "length",
"error": null
}
]

22
benchmarks/inference/direct-results/20260831T133721Z-qwen3.8-27b-mxfp8/summary.json Normal file
View File

@@ -0,0 +1,22 @@
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-mxfp8",
"requests": 5,
"successes": 5,
"ttft_s_mean": 0.6405866249930113,
"ttft_s_p50": 0.6006166669540107,
"ttft_s_p95": 0.993492508213967,
"total_s_mean": 8.97830267478712,
"total_s_p50": 8.952084249991458,
"total_s_p95": 9.543902474397328,
"prompt_tps_mean": 77.25518114106583,
"prompt_tps_p50": 69.803885982221,
"prompt_tps_p95": 108.54351756783554,
"decode_tps_mean": 15.141694584402668,
"decode_tps_p50": 15.417877788139895,
"decode_tps_p95": 15.90376325022775,
"request_tps_mean": 14.036390549836813,
"request_tps_p50": 14.298346220336581,
"request_tps_p95": 14.998850485179887,
"peak_memory_gb": 28.05746427
}

48
benchmarks/inference/direct-results/20260831T134407Z-qwen3.8-27b-nvfp4-rerun/metadata.json Normal file
View File

@@ -0,0 +1,48 @@
{
"run_id": "20260831T134407Z",
"created_at": "2026-08-31T13:44:45.718463+00:00",
"host": {
"platform": "macOS-15.6-arm64-arm-64bit",
"machine": "arm64",
"python": "3.12.8"
},
"arguments": {
"model": "mlx-community/Qwen3.8-27B-nvfp4",
"label": "qwen3.8-27b-nvfp4-rerun",
"warmups": 2,
"rounds": 5,
"max_tokens": 128,
"seed": 42,
"output_dir": "/Users/christianmeyer/Documents/git/lai_automatisierung/benchmarks/inference/direct-results"
},
"load_s": 3.153125083015766,
"mlx_version": "0.32.0",
"device": "Device(gpu, 0)",
"prompts": {
"short": "In exactly four concise bullet points, explain why database transactions are useful. Do not use a heading.",
"medium": "You are reviewing a Laravel service that transfers money between two accounts. Explain the failure modes involving concurrency, partial writes, retries, and audit logs. Then provide a robust implementation plan. Keep the response technical and concise.",
"code": "Write a Python 3.12 function named merge_intervals that accepts a list of integer pairs, validates each pair, merges overlapping or adjacent intervals, and returns sorted tuples. Include type hints, a short docstring, and five assert-based examples. Output only code."
},
"summary": {
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4-rerun",
"requests": 5,
"successes": 5,
"ttft_s_mean": 0.46268870839849113,
"ttft_s_p50": 0.5028267079615034,
"ttft_s_p95": 0.5507681754184887,
"total_s_mean": 5.0599139665835535,
"total_s_p50": 5.047160332964268,
"total_s_p95": 5.222982274997048,
"prompt_tps_mean": 103.80325950757997,
"prompt_tps_p50": 102.6553925326314,
"prompt_tps_p95": 128.99882432098076,
"decode_tps_mean": 27.91928963114296,
"decode_tps_p50": 27.87385628056727,
"decode_tps_p95": 28.51407145210707,
"request_tps_mean": 25.315032232739288,
"request_tps_p50": 25.360795289977208,
"request_tps_p95": 26.258409367849122,
"peak_memory_gb": 15.479377792
}
}

128
benchmarks/inference/direct-results/20260831T134407Z-qwen3.8-27b-nvfp4-rerun/raw.json Normal file
View File

@@ -0,0 +1,128 @@
[
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4-rerun",
"source": "mlx-community/Qwen3.8-27B-nvfp4",
"phase": "warmup",
"round": 1,
"prompt_name": "short",
"prompt_tokens": 20,
"output_tokens": 128,
"ttft_s": 0.46846333297435194,
"total_s": 4.887791082961485,
"prompt_tps": 84.84884256090237,
"decode_tps": 29.034596267228054,
"request_tps": 26.18769866130316,
"peak_memory_gb": 15.399948802,
"finish_reason": "length",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4-rerun",
"source": "mlx-community/Qwen3.8-27B-nvfp4",
"phase": "warmup",
"round": 2,
"prompt_name": "medium",
"prompt_tokens": 43,
"output_tokens": 128,
"ttft_s": 0.476554416003637,
"total_s": 4.906530833046418,
"prompt_tps": 122.50184376172285,
"decode_tps": 28.96545575207384,
"request_tps": 26.0876787195335,
"peak_memory_gb": 15.453767792,
"finish_reason": "length",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4-rerun",
"source": "mlx-community/Qwen3.8-27B-nvfp4",
"phase": "measured",
"round": 1,
"prompt_name": "short",
"prompt_tokens": 20,
"output_tokens": 128,
"ttft_s": 0.34607333299936727,
"total_s": 4.835503207985312,
"prompt_tps": 90.02367046337802,
"decode_tps": 28.58315107058229,
"request_tps": 26.470874797192113,
"peak_memory_gb": 15.453767792,
"finish_reason": "length",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4-rerun",
"source": "mlx-community/Qwen3.8-27B-nvfp4",
"phase": "measured",
"round": 2,
"prompt_name": "medium",
"prompt_tokens": 43,
"output_tokens": 128,
"ttft_s": 0.5028267079615034,
"total_s": 5.047160332964268,
"prompt_tps": 113.11164547028159,
"decode_tps": 28.2377529782062,
"request_tps": 25.360795289977208,
"peak_memory_gb": 15.453784176,
"finish_reason": "length",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4-rerun",
"source": "mlx-community/Qwen3.8-27B-nvfp4",
"phase": "measured",
"round": 3,
"prompt_name": "code",
"prompt_tokens": 54,
"output_tokens": 128,
"ttft_s": 0.5305347089888528,
"total_s": 5.134004874969833,
"prompt_tps": 132.97061903365554,
"decode_tps": 27.87385628056727,
"request_tps": 24.93180336155254,
"peak_memory_gb": 15.479377792,
"finish_reason": "length",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4-rerun",
"source": "mlx-community/Qwen3.8-27B-nvfp4",
"phase": "measured",
"round": 4,
"prompt_name": "short",
"prompt_tokens": 20,
"output_tokens": 128,
"ttft_s": 0.3781822500168346,
"total_s": 5.037674791994505,
"prompt_tps": 80.25497003795336,
"decode_tps": 27.538176565956825,
"request_tps": 25.408547650477164,
"peak_memory_gb": 15.479377792,
"finish_reason": "length",
"error": null
},
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4-rerun",
"source": "mlx-community/Qwen3.8-27B-nvfp4",
"phase": "measured",
"round": 5,
"prompt_name": "medium",
"prompt_tokens": 43,
"output_tokens": 128,
"ttft_s": 0.5558265420258977,
"total_s": 5.245226625003852,
"prompt_tps": 102.6553925326314,
"decode_tps": 27.363511260402223,
"request_tps": 24.403140064497403,
"peak_memory_gb": 15.479377792,
"finish_reason": "length",
"error": null
}
]

22
benchmarks/inference/direct-results/20260831T134407Z-qwen3.8-27b-nvfp4-rerun/summary.json Normal file
View File

@@ -0,0 +1,22 @@
{
"runtime": "mlx-lm-direct",
"model": "qwen3.8-27b-nvfp4-rerun",
"requests": 5,
"successes": 5,
"ttft_s_mean": 0.46268870839849113,
"ttft_s_p50": 0.5028267079615034,
"ttft_s_p95": 0.5507681754184887,
"total_s_mean": 5.0599139665835535,
"total_s_p50": 5.047160332964268,
"total_s_p95": 5.222982274997048,
"prompt_tps_mean": 103.80325950757997,
"prompt_tps_p50": 102.6553925326314,
"prompt_tps_p95": 128.99882432098076,
"decode_tps_mean": 27.91928963114296,
"decode_tps_p50": 27.87385628056727,
"decode_tps_p95": 28.51407145210707,
"request_tps_mean": 25.315032232739288,
"request_tps_p50": 25.360795289977208,
"request_tps_p95": 26.258409367849122,
"peak_memory_gb": 15.479377792
}

155
benchmarks/inference/direct_mlx_benchmark.py Normal file
View File

@@ -0,0 +1,155 @@
#!/usr/bin/env python3
"""Measure direct MLX-LM generation without an HTTP serving layer."""
from __future__ import annotations
import argparse
import json
import math
import platform
import statistics
import time
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
import mlx.core as mx
from benchmark import PROMPTS
from mlx_lm import load, stream_generate
from mlx_lm.sample_utils import make_sampler
def percentile(values: list[float], fraction: float) -> float:
ordered = sorted(values)
position = (len(ordered) - 1) * fraction
lower = math.floor(position)
upper = math.ceil(position)
if lower == upper:
return ordered[lower]
return ordered[lower] + (ordered[upper] - ordered[lower]) * (position - lower)
def summarize(rows: list[dict[str, Any]]) -> dict[str, Any]:
measured = [row for row in rows if row["phase"] == "measured"]
result: dict[str, Any] = {
"runtime": "mlx-lm-direct",
"model": measured[0]["model"],
"requests": len(measured),
"successes": sum(row["error"] is None for row in measured),
}
for field in ("ttft_s", "total_s", "prompt_tps", "decode_tps", "request_tps"):
values = [float(row[field]) for row in measured if row[field] is not None]
result[f"{field}_mean"] = statistics.fmean(values)
result[f"{field}_p50"] = percentile(values, 0.5)
result[f"{field}_p95"] = percentile(values, 0.95)
result["peak_memory_gb"] = max(float(row["peak_memory_gb"]) for row in measured)
return result
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--model", required=True)
parser.add_argument("--label", required=True)
parser.add_argument("--warmups", type=int, default=2)
parser.add_argument("--rounds", type=int, default=5)
parser.add_argument("--max-tokens", type=int, default=128)
parser.add_argument("--seed", type=int, default=42)
parser.add_argument(
"--output-dir", type=Path, default=Path(__file__).parent / "direct-results"
)
return parser.parse_args()
def main() -> int:
args = parse_args()
if args.warmups < 0 or args.rounds < 1 or args.max_tokens < 1:
raise SystemExit("warmups must be >= 0; rounds and max-tokens must be >= 1")
run_id = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
output_dir = args.output_dir / f"{run_id}-{args.label}"
output_dir.mkdir(parents=True, exist_ok=False)
load_started = time.perf_counter()
model, tokenizer = load(args.model, trust_remote_code=True)
load_s = time.perf_counter() - load_started
sampler = make_sampler(temp=0.0)
prompt_items = list(PROMPTS.items())
rows: list[dict[str, Any]] = []
for phase, count in (("warmup", args.warmups), ("measured", args.rounds)):
for round_number in range(count):
prompt_name, prompt = prompt_items[round_number % len(prompt_items)]
mx.random.seed(args.seed)
started = time.perf_counter()
first_token_at: float | None = None
final = None
error = None
try:
for response in stream_generate(
model,
tokenizer,
prompt,
max_tokens=args.max_tokens,
sampler=sampler,
):
if first_token_at is None:
first_token_at = time.perf_counter()
final = response
except Exception as exception: # Preserve evidence before failing the run.
error = f"{type(exception).__name__}: {exception}"
total_s = time.perf_counter() - started
tokens = final.generation_tokens if final is not None else 0
row = {
"runtime": "mlx-lm-direct",
"model": args.label,
"source": args.model,
"phase": phase,
"round": round_number + 1,
"prompt_name": prompt_name,
"prompt_tokens": final.prompt_tokens if final is not None else None,
"output_tokens": tokens,
"ttft_s": first_token_at - started if first_token_at is not None else None,
"total_s": total_s,
"prompt_tps": final.prompt_tps if final is not None else None,
"decode_tps": final.generation_tps if final is not None else None,
"request_tps": tokens / total_s if tokens else None,
"peak_memory_gb": final.peak_memory if final is not None else None,
"finish_reason": final.finish_reason if final is not None else None,
"error": error,
}
rows.append(row)
print(
f"{phase} {round_number + 1}/{count} prompt={prompt_name} "
f"total={total_s:.3f}s tokens={tokens} "
f"decode={row['decode_tps'] or 0:.2f}tok/s error={error or 'none'}",
flush=True,
)
summary = summarize(rows)
metadata = {
"run_id": run_id,
"created_at": datetime.now(UTC).isoformat(),
"host": {
"platform": platform.platform(),
"machine": platform.machine(),
"python": platform.python_version(),
},
"arguments": vars(args) | {"output_dir": str(args.output_dir)},
"load_s": load_s,
"mlx_version": getattr(mx, "__version__", None),
"device": str(mx.default_device()),
"prompts": PROMPTS,
"summary": summary,
}
(output_dir / "raw.json").write_text(json.dumps(rows, indent=2), encoding="utf-8")
(output_dir / "metadata.json").write_text(
json.dumps(metadata, indent=2, default=str), encoding="utf-8"
)
(output_dir / "summary.json").write_text(json.dumps(summary, indent=2), encoding="utf-8")
print(json.dumps(summary, indent=2))
print(f"Results: {output_dir}")
return 1 if any(row["error"] for row in rows if row["phase"] == "measured") else 0
if __name__ == "__main__":
raise SystemExit(main())

136
benchmarks/inference/results/20260831T114727Z/metadata.json Normal file
View File

@@ -0,0 +1,136 @@
{
"run_id": "20260831T114727Z",
"created_at": "2026-08-31T11:47:27.622260+00:00",
"host": {
"platform": "macOS-15.6-arm64-arm-64bit",
"machine": "arm64",
"python": "3.12.8"
},
"arguments": {
"output_dir": "/Users/christianmeyer/Documents/git/lai_automatisierung/benchmarks/inference/results",
"runtime": "vllm",
"ollama_url": "http://127.0.0.1:11434",
"ollama_model": "qwen3.8:27b-mlx",
"vllm_url": "http://127.0.0.1:8001",
"vllm_model": "qwen3.8-27b-nvfp4",
"warmups": 2,
"rounds": 5,
"concurrency": [
1,
4
],
"max_tokens": 128,
"seed": 42,
"timeout": 600
},
"runtimes": [
{
"name": "vllm-metal",
"kind": "openai",
"base_url": "http://127.0.0.1:8001",
"model": "qwen3.8-27b-nvfp4"
}
],
"versions": {
"ollama": "ollama version is 0.32.14",
"vllm": "INFO 08-31 13:47:28 [__init__.py:52] Available plugins for group vllm.platform_plugins:\nINFO 08-31 13:47:28 [__init__.py:54] - metal -> vllm_metal:register\nINFO 08-31 13:47:28 [__init__.py:57] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.\nINFO 08-31 13:47:30 [__init__.py:254] Platform plugin metal is activated\nINFO 08-31 13:47:30 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.\n0.28.0+cpu"
},
"model_metadata": {
"vllm-metal": {
"object": "list",
"data": [
{
"id": "qwen3.8-27b-nvfp4",
"object": "model",
"created": 1788176852,
"owned_by": "vllm",
"root": "mlx-community/Qwen3.8-27B-nvfp4",
"parent": null,
"max_model_len": 8192,
"permission": [
{
"id": "modelperm-956fc45cb39100d2",
"object": "model_permission",
"created": 1788176852,
"allow_create_engine": false,
"allow_sampling": true,
"allow_logprobs": true,
"allow_search_indices": false,
"allow_view": true,
"allow_fine_tuning": false,
"organization": "*",
"group": null,
"is_blocking": false
}
]
}
]
}
},
"prompts": {
"short": "In exactly four concise bullet points, explain why database transactions are useful. Do not use a heading.",
"medium": "You are reviewing a Laravel service that transfers money between two accounts. Explain the failure modes involving concurrency, partial writes, retries, and audit logs. Then provide a robust implementation plan. Keep the response technical and concise.",
"code": "Write a Python 3.12 function named merge_intervals that accepts a list of integer pairs, validates each pair, merges overlapping or adjacent intervals, and returns sorted tuples. Include type hints, a short docstring, and five assert-based examples. Output only code."
},
"measurement_notes": {
"ttft": "Client wall time from request start until first non-empty generated text chunk.",
"total": "Client wall time from request start until stream completion.",
"decode_tps": "Ollama server eval metric; otherwise output tokens / (total - TTFT).",
"aggregate_output_tps": "Sum of successful output tokens / concurrent group wall time."
},
"completed_at": "2026-08-31T11:49:15.455115+00:00",
"summary": [
{
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"concurrency": 1,
"requests": 5,
"successes": 5,
"failures": 0,
"ttft_s_mean": 0.3338013501954265,
"ttft_s_p50": 0.37227433401858434,
"ttft_s_p95": 0.4201282836031169,
"total_s_mean": 5.444907250197138,
"total_s_p50": 5.454897999996319,
"total_s_p95": 5.562144016823732,
"prompt_tps_mean": null,
"prompt_tps_p50": null,
"prompt_tps_p95": null,
"decode_tps_mean": 25.04504255080693,
"decode_tps_p50": 24.962500989743333,
"decode_tps_p95": 25.32382313060655,
"request_tps_mean": 23.517575676354507,
"request_tps_p50": 23.465150035818517,
"request_tps_p95": 24.196276605031606,
"aggregate_output_tps_mean": 23.516827822391825,
"aggregate_output_tps_p50": 23.464397444683396,
"aggregate_output_tps_p95": 24.195480643338048
},
{
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"concurrency": 4,
"requests": 20,
"successes": 20,
"failures": 0,
"ttft_s_mean": 1.1358489938022103,
"ttft_s_p50": 1.1691114795103204,
"ttft_s_p95": 1.2360623297834537,
"total_s_mean": 9.343862856202758,
"total_s_p50": 9.272835896001197,
"total_s_p95": 9.591099269420374,
"prompt_tps_mean": null,
"prompt_tps_p50": null,
"prompt_tps_p95": null,
"decode_tps_mean": 15.572524640705884,
"decode_tps_p50": 15.580143217705817,
"decode_tps_p95": 15.920671319893613,
"request_tps_mean": 13.679827878866371,
"request_tps_p50": 13.803600866803595,
"request_tps_p95": 13.831739950302437,
"aggregate_output_tps_mean": 54.66256367620249,
"aggregate_output_tps_p50": 54.89183177619698,
"aggregate_output_tps_p95": 55.24148726503248
}
]
}

36
benchmarks/inference/results/20260831T114727Z/raw.csv Normal file
View File

@@ -0,0 +1,36 @@
run_id,runtime,model,phase,concurrency,round,slot,request_id,prompt_name,prompt_chars,max_tokens,started_at,ttft_s,total_s,prompt_tokens,output_tokens,prompt_tps,decode_tps,request_tps,group_elapsed_s,group_output_tps,error
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,warmup,1,0,0,8b712536-91f2-4ded-b334-6db253435ba3,short,106,128,2026-08-31T11:47:32.567050+00:00,0.561656582984142,5.602756957989186,20,128,,25.39128175956463,22.845895504619364,5.6035958750289865,22.842475234589944,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,warmup,1,1,0,be5a9e94-abd6-4cbc-8029-7a55e3ad129c,medium,252,128,2026-08-31T11:47:38.170087+00:00,0.3621360409888439,5.426666249986738,43,128,,25.273815086064428,23.587225398339694,5.426845958980266,23.586444311762243,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,1,0,0,e0ac985d-818e-45ec-a2c1-7e077fe0f1cd,short,106,128,2026-08-31T11:47:43.597014+00:00,0.21451262495247647,5.262066499970388,20,128,,25.358817987761604,24.325044162919706,5.262241083022673,24.324237141654443,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,1,1,0,428bad62-8fbf-41a1-9b36-07912506da11,medium,252,128,2026-08-31T11:47:48.859322+00:00,0.37227433401858434,5.454897999996319,43,128,,25.18384370198632,23.465150035818517,5.455072959011886,23.464397444683396,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,1,2,0,b2de9565-6ff0-47a1-91e6-847b83994f6c,code,267,128,2026-08-31T11:47:54.314442+00:00,0.40532424999400973,5.533015583991073,54,128,,24.962500989743333,23.13385857259254,5.533179917023517,23.13317150707355,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,1,3,0,dc02daab-d1f2-49ac-8e29-01ee86148e0a,short,106,128,2026-08-31T11:47:59.847681+00:00,0.25306625000666827,5.405130041996017,20,128,,24.844412873734196,23.681206373479203,5.405301624967251,23.680454650072466,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,1,4,0,677f2a25-31e5-43e7-b8fd-5e607453b904,medium,252,128,2026-08-31T11:48:05.253048+00:00,0.42382929200539365,5.569426125031896,43,128,,24.875637200809187,22.982619236962577,5.569605667027645,22.981878368475286,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,warmup,4,0,0,7109b7b9-3690-49d6-8b20-969734c3dcad,short,106,128,2026-08-31T11:48:10.822944+00:00,1.1026008339831606,8.694552083965391,20,128,,16.859960738064487,14.721862467884838,8.695109791995492,58.883672805527404,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,warmup,4,0,1,b34a5a99-e633-4f3a-892d-67af2b425779,medium,252,128,2026-08-31T11:48:10.823149+00:00,1.1023455830290914,8.694302666990552,43,128,,16.859947782161328,14.722284799904022,8.695109791995492,58.883672805527404,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,warmup,4,0,2,1260b39f-a4b1-4d94-9406-70028f614ce0,code,267,128,2026-08-31T11:48:10.823017+00:00,1.0163157919887453,8.694556291971821,54,128,,16.670485901070972,14.721855342772315,8.695109791995492,58.883672805527404,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,warmup,4,0,3,b43d525a-50de-4e41-b00c-d1159d1ddfd4,short,106,128,2026-08-31T11:48:10.822845+00:00,1.1026693329913542,8.694680040993262,20,128,,16.85982869664412,14.721645810600473,8.695109791995492,58.883672805527404,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,warmup,4,1,0,1da340a9-b91e-4f9b-8217-2af7c2fbc030,medium,252,128,2026-08-31T11:48:19.518058+00:00,1.245261833013501,9.168966500030365,43,128,,16.154059922603068,13.960133892906699,9.169412165996619,55.83782152346415,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,warmup,4,1,1,23e076e6-7b50-4d08-a3ff-f79ddc53bafe,code,267,128,2026-08-31T11:48:19.518183+00:00,1.2451274580089375,9.168911040993407,54,128,,16.153899038190183,13.96021833211415,9.169412165996619,55.83782152346415,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,warmup,4,1,2,c777d58b-0520-4f3f-8537-6a163ab350f5,short,106,128,2026-08-31T11:48:19.518319+00:00,1.2449699160060845,9.168749333010055,20,128,,16.153907531211612,13.960464546583719,9.169412165996619,55.83782152346415,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,warmup,4,1,3,5609655c-e54a-499f-99c6-c77c19c1a3e0,medium,252,128,2026-08-31T11:48:19.518407+00:00,1.174761500034947,9.16857587499544,43,128,,16.012380822970083,13.960728661152478,9.169412165996619,55.83782152346415,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,0,0,d2615398-7368-4f9a-9424-7a2e76d8955d,short,106,128,2026-08-31T11:48:28.687706+00:00,1.0508670000126585,9.266459833015688,20,128,,15.580129468662143,13.813257954666332,9.267125833022874,55.24906095215814,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,0,1,b307dcc2-b325-4005-a63f-54123aa28e31,medium,252,128,2026-08-31T11:48:28.687614+00:00,1.0509756249957718,9.266665957984515,43,128,,15.579944570943383,13.812950696653772,9.267125833022874,55.24906095215814,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,0,2,09c7607f-96f5-4a6d-a9e1-3637c8afc257,code,267,128,2026-08-31T11:48:28.687778+00:00,1.0507545830332674,9.266332915984094,54,128,,15.580156966749492,13.813447149001583,9.267125833022874,55.24906095215814,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,0,3,0328d6c0-e3f4-4a86-917d-dbeb5b8deee7,short,106,128,2026-08-31T11:48:28.687501+00:00,0.9898037080420181,9.266721500025596,20,128,,15.464693889308833,13.81286790583341,9.267125833022874,55.24906095215814,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,1,0,74110182-284c-4250-9395-e3d0f2772eef,medium,252,128,2026-08-31T11:48:37.954714+00:00,1.1083811249700375,9.379890666983556,43,128,,15.474805336299134,13.646214496992963,9.380457791965455,54.58155788926826,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,1,1,fca02813-8ad6-4e99-bc76-9e922ea9a63a,code,267,128,2026-08-31T11:48:37.954992+00:00,1.1690869170124643,9.379730832995847,54,128,,15.589520299476966,13.646447033397155,9.380457791965455,54.58155788926826,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,1,2,5eb13719-9063-4ddd-92c8-73332709ecfd,short,106,128,2026-08-31T11:48:37.954812+00:00,1.1692542500095442,9.37985170801403,20,128,,15.589608509574806,13.646271176189105,9.380457791965455,54.58155788926826,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,1,3,9d9a5b23-6950-4d5c-bad2-72f896d38009,medium,252,128,2026-08-31T11:48:37.954917+00:00,1.1691360420081764,9.379842250025831,43,128,,15.58940202671112,13.646284936150979,9.380457791965455,54.58155788926826,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,2,0,caf79143-9d5b-4fb2-830e-3bd2d38cc5d3,code,267,128,2026-08-31T11:48:47.335341+00:00,1.172440624970477,9.591207291989122,54,128,,15.204127286417465,13.345556623190662,9.591755541972816,53.37917524685921,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,2,1,d77a5e81-922c-49ce-96c5-1fb443746984,short,106,128,2026-08-31T11:48:47.335566+00:00,1.2360592089826241,9.591093584022019,20,128,,15.320104532711328,13.345714842490702,9.591755541972816,53.37917524685921,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,2,2,9ccc94c9-d569-4785-9f48-e48e760b0e15,medium,252,128,2026-08-31T11:48:47.335447+00:00,1.236121624999214,9.591066916007549,43,128,,15.320267882275031,13.345751950324445,9.591755541972816,53.37917524685921,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,2,3,e7c5be37-35a8-47a5-869d-a3bee16750d4,code,267,128,2026-08-31T11:48:47.335653+00:00,1.2359632499865256,9.590960291970987,54,128,,15.32017298830757,13.345900316900948,9.591755541972816,53.37917524685921,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,3,0,49d569ca-4eef-46fd-88f9-58cf68e01898,short,106,128,2026-08-31T11:48:56.927355+00:00,1.067110000003595,9.073982625035569,20,124,,15.486695718417879,13.665443843575128,9.254564541974105,54.89183177619698,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,3,1,a34ede89-ca43-4a1c-a7c8-c916bfec24a5,medium,252,128,2026-08-31T11:48:56.927278+00:00,1.0672226250171661,9.253953957988415,43,128,,15.635055652124883,13.831925313341854,9.254564541974105,54.89183177619698,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,3,2,af0dfbe9-f1a5-434a-a714-98fad2af83a3,code,267,128,2026-08-31T11:48:56.927205+00:00,1.067305459000636,9.254084500018507,54,128,,15.634964539617723,13.831730194352993,9.254564541974105,54.89183177619698,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,3,3,80dc0643-6f40-4ab5-805d-ace4e9f149bb,short,106,128,2026-08-31T11:48:56.927082+00:00,1.0054587500053458,9.254091374983545,20,128,,15.517723460297553,13.831719918610334,9.254564541974105,54.89183177619698,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,4,0,12e42bf2-6a75-4f89-a598-ac14d5b67a78,medium,252,128,2026-08-31T11:49:06.181834+00:00,1.2329518750193529,9.273020750028081,43,128,,15.920261628338482,13.803484695061464,9.273481999989599,55.21119251652985,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,4,1,5f2cf183-3654-45be-81ed-4eebfb57bce1,code,267,128,2026-08-31T11:49:06.182126+00:00,1.1721314580063336,9.272628374979831,54,128,,15.801499748959015,13.804068795141205,9.273481999989599,55.21119251652985,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,4,2,746da721-ed00-4c80-bc92-ed115313463e,short,106,128,2026-08-31T11:49:06.182026+00:00,1.2329447079682723,9.272807125002146,20,128,,15.920670449385964,13.80380269690667,9.273481999989599,55.21119251652985,
20260831T114727Z,vllm-metal,qwen3.8-27b-nvfp4,measured,4,4,3,5c45d61d-9d4a-4fc4-931d-5193cdcaa598,medium,252,128,2026-08-31T11:49:06.181940+00:00,1.2330110420007259,9.272864667000249,43,128,,15.920687859538933,13.803717038545729,9.273481999989599,55.21119251652985,
1 run_id runtime model phase concurrency round slot request_id prompt_name prompt_chars max_tokens started_at ttft_s total_s prompt_tokens output_tokens prompt_tps decode_tps request_tps group_elapsed_s group_output_tps error
2 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 warmup 1 0 0 8b712536-91f2-4ded-b334-6db253435ba3 short 106 128 2026-08-31T11:47:32.567050+00:00 0.561656582984142 5.602756957989186 20 128 25.39128175956463 22.845895504619364 5.6035958750289865 22.842475234589944
3 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 warmup 1 1 0 be5a9e94-abd6-4cbc-8029-7a55e3ad129c medium 252 128 2026-08-31T11:47:38.170087+00:00 0.3621360409888439 5.426666249986738 43 128 25.273815086064428 23.587225398339694 5.426845958980266 23.586444311762243
4 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 1 0 0 e0ac985d-818e-45ec-a2c1-7e077fe0f1cd short 106 128 2026-08-31T11:47:43.597014+00:00 0.21451262495247647 5.262066499970388 20 128 25.358817987761604 24.325044162919706 5.262241083022673 24.324237141654443
5 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 1 1 0 428bad62-8fbf-41a1-9b36-07912506da11 medium 252 128 2026-08-31T11:47:48.859322+00:00 0.37227433401858434 5.454897999996319 43 128 25.18384370198632 23.465150035818517 5.455072959011886 23.464397444683396
6 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 1 2 0 b2de9565-6ff0-47a1-91e6-847b83994f6c code 267 128 2026-08-31T11:47:54.314442+00:00 0.40532424999400973 5.533015583991073 54 128 24.962500989743333 23.13385857259254 5.533179917023517 23.13317150707355
7 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 1 3 0 dc02daab-d1f2-49ac-8e29-01ee86148e0a short 106 128 2026-08-31T11:47:59.847681+00:00 0.25306625000666827 5.405130041996017 20 128 24.844412873734196 23.681206373479203 5.405301624967251 23.680454650072466
8 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 1 4 0 677f2a25-31e5-43e7-b8fd-5e607453b904 medium 252 128 2026-08-31T11:48:05.253048+00:00 0.42382929200539365 5.569426125031896 43 128 24.875637200809187 22.982619236962577 5.569605667027645 22.981878368475286
9 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 warmup 4 0 0 7109b7b9-3690-49d6-8b20-969734c3dcad short 106 128 2026-08-31T11:48:10.822944+00:00 1.1026008339831606 8.694552083965391 20 128 16.859960738064487 14.721862467884838 8.695109791995492 58.883672805527404
10 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 warmup 4 0 1 b34a5a99-e633-4f3a-892d-67af2b425779 medium 252 128 2026-08-31T11:48:10.823149+00:00 1.1023455830290914 8.694302666990552 43 128 16.859947782161328 14.722284799904022 8.695109791995492 58.883672805527404
11 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 warmup 4 0 2 1260b39f-a4b1-4d94-9406-70028f614ce0 code 267 128 2026-08-31T11:48:10.823017+00:00 1.0163157919887453 8.694556291971821 54 128 16.670485901070972 14.721855342772315 8.695109791995492 58.883672805527404
12 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 warmup 4 0 3 b43d525a-50de-4e41-b00c-d1159d1ddfd4 short 106 128 2026-08-31T11:48:10.822845+00:00 1.1026693329913542 8.694680040993262 20 128 16.85982869664412 14.721645810600473 8.695109791995492 58.883672805527404
13 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 warmup 4 1 0 1da340a9-b91e-4f9b-8217-2af7c2fbc030 medium 252 128 2026-08-31T11:48:19.518058+00:00 1.245261833013501 9.168966500030365 43 128 16.154059922603068 13.960133892906699 9.169412165996619 55.83782152346415
14 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 warmup 4 1 1 23e076e6-7b50-4d08-a3ff-f79ddc53bafe code 267 128 2026-08-31T11:48:19.518183+00:00 1.2451274580089375 9.168911040993407 54 128 16.153899038190183 13.96021833211415 9.169412165996619 55.83782152346415
15 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 warmup 4 1 2 c777d58b-0520-4f3f-8537-6a163ab350f5 short 106 128 2026-08-31T11:48:19.518319+00:00 1.2449699160060845 9.168749333010055 20 128 16.153907531211612 13.960464546583719 9.169412165996619 55.83782152346415
16 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 warmup 4 1 3 5609655c-e54a-499f-99c6-c77c19c1a3e0 medium 252 128 2026-08-31T11:48:19.518407+00:00 1.174761500034947 9.16857587499544 43 128 16.012380822970083 13.960728661152478 9.169412165996619 55.83782152346415
17 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 0 0 d2615398-7368-4f9a-9424-7a2e76d8955d short 106 128 2026-08-31T11:48:28.687706+00:00 1.0508670000126585 9.266459833015688 20 128 15.580129468662143 13.813257954666332 9.267125833022874 55.24906095215814
18 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 0 1 b307dcc2-b325-4005-a63f-54123aa28e31 medium 252 128 2026-08-31T11:48:28.687614+00:00 1.0509756249957718 9.266665957984515 43 128 15.579944570943383 13.812950696653772 9.267125833022874 55.24906095215814
19 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 0 2 09c7607f-96f5-4a6d-a9e1-3637c8afc257 code 267 128 2026-08-31T11:48:28.687778+00:00 1.0507545830332674 9.266332915984094 54 128 15.580156966749492 13.813447149001583 9.267125833022874 55.24906095215814
20 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 0 3 0328d6c0-e3f4-4a86-917d-dbeb5b8deee7 short 106 128 2026-08-31T11:48:28.687501+00:00 0.9898037080420181 9.266721500025596 20 128 15.464693889308833 13.81286790583341 9.267125833022874 55.24906095215814
21 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 1 0 74110182-284c-4250-9395-e3d0f2772eef medium 252 128 2026-08-31T11:48:37.954714+00:00 1.1083811249700375 9.379890666983556 43 128 15.474805336299134 13.646214496992963 9.380457791965455 54.58155788926826
22 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 1 1 fca02813-8ad6-4e99-bc76-9e922ea9a63a code 267 128 2026-08-31T11:48:37.954992+00:00 1.1690869170124643 9.379730832995847 54 128 15.589520299476966 13.646447033397155 9.380457791965455 54.58155788926826
23 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 1 2 5eb13719-9063-4ddd-92c8-73332709ecfd short 106 128 2026-08-31T11:48:37.954812+00:00 1.1692542500095442 9.37985170801403 20 128 15.589608509574806 13.646271176189105 9.380457791965455 54.58155788926826
24 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 1 3 9d9a5b23-6950-4d5c-bad2-72f896d38009 medium 252 128 2026-08-31T11:48:37.954917+00:00 1.1691360420081764 9.379842250025831 43 128 15.58940202671112 13.646284936150979 9.380457791965455 54.58155788926826
25 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 2 0 caf79143-9d5b-4fb2-830e-3bd2d38cc5d3 code 267 128 2026-08-31T11:48:47.335341+00:00 1.172440624970477 9.591207291989122 54 128 15.204127286417465 13.345556623190662 9.591755541972816 53.37917524685921
26 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 2 1 d77a5e81-922c-49ce-96c5-1fb443746984 short 106 128 2026-08-31T11:48:47.335566+00:00 1.2360592089826241 9.591093584022019 20 128 15.320104532711328 13.345714842490702 9.591755541972816 53.37917524685921
27 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 2 2 9ccc94c9-d569-4785-9f48-e48e760b0e15 medium 252 128 2026-08-31T11:48:47.335447+00:00 1.236121624999214 9.591066916007549 43 128 15.320267882275031 13.345751950324445 9.591755541972816 53.37917524685921
28 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 2 3 e7c5be37-35a8-47a5-869d-a3bee16750d4 code 267 128 2026-08-31T11:48:47.335653+00:00 1.2359632499865256 9.590960291970987 54 128 15.32017298830757 13.345900316900948 9.591755541972816 53.37917524685921
29 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 3 0 49d569ca-4eef-46fd-88f9-58cf68e01898 short 106 128 2026-08-31T11:48:56.927355+00:00 1.067110000003595 9.073982625035569 20 124 15.486695718417879 13.665443843575128 9.254564541974105 54.89183177619698
30 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 3 1 a34ede89-ca43-4a1c-a7c8-c916bfec24a5 medium 252 128 2026-08-31T11:48:56.927278+00:00 1.0672226250171661 9.253953957988415 43 128 15.635055652124883 13.831925313341854 9.254564541974105 54.89183177619698
31 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 3 2 af0dfbe9-f1a5-434a-a714-98fad2af83a3 code 267 128 2026-08-31T11:48:56.927205+00:00 1.067305459000636 9.254084500018507 54 128 15.634964539617723 13.831730194352993 9.254564541974105 54.89183177619698
32 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 3 3 80dc0643-6f40-4ab5-805d-ace4e9f149bb short 106 128 2026-08-31T11:48:56.927082+00:00 1.0054587500053458 9.254091374983545 20 128 15.517723460297553 13.831719918610334 9.254564541974105 54.89183177619698
33 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 4 0 12e42bf2-6a75-4f89-a598-ac14d5b67a78 medium 252 128 2026-08-31T11:49:06.181834+00:00 1.2329518750193529 9.273020750028081 43 128 15.920261628338482 13.803484695061464 9.273481999989599 55.21119251652985
34 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 4 1 5f2cf183-3654-45be-81ed-4eebfb57bce1 code 267 128 2026-08-31T11:49:06.182126+00:00 1.1721314580063336 9.272628374979831 54 128 15.801499748959015 13.804068795141205 9.273481999989599 55.21119251652985
35 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 4 2 746da721-ed00-4c80-bc92-ed115313463e short 106 128 2026-08-31T11:49:06.182026+00:00 1.2329447079682723 9.272807125002146 20 128 15.920670449385964 13.80380269690667 9.273481999989599 55.21119251652985
36 20260831T114727Z vllm-metal qwen3.8-27b-nvfp4 measured 4 4 3 5c45d61d-9d4a-4fc4-931d-5193cdcaa598 medium 252 128 2026-08-31T11:49:06.181940+00:00 1.2330110420007259 9.272864667000249 43 128 15.920687859538933 13.803717038545729 9.273481999989599 55.21119251652985

842
benchmarks/inference/results/20260831T114727Z/raw.json Normal file
View File

@@ -0,0 +1,842 @@
[
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "warmup",
"concurrency": 1,
"round": 0,
"slot": 0,
"request_id": "8b712536-91f2-4ded-b334-6db253435ba3",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:47:32.567050+00:00",
"ttft_s": 0.561656582984142,
"total_s": 5.602756957989186,
"prompt_tokens": 20,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 25.39128175956463,
"request_tps": 22.845895504619364,
"group_elapsed_s": 5.6035958750289865,
"group_output_tps": 22.842475234589944,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "warmup",
"concurrency": 1,
"round": 1,
"slot": 0,
"request_id": "be5a9e94-abd6-4cbc-8029-7a55e3ad129c",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:47:38.170087+00:00",
"ttft_s": 0.3621360409888439,
"total_s": 5.426666249986738,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 25.273815086064428,
"request_tps": 23.587225398339694,
"group_elapsed_s": 5.426845958980266,
"group_output_tps": 23.586444311762243,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 1,
"round": 0,
"slot": 0,
"request_id": "e0ac985d-818e-45ec-a2c1-7e077fe0f1cd",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:47:43.597014+00:00",
"ttft_s": 0.21451262495247647,
"total_s": 5.262066499970388,
"prompt_tokens": 20,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 25.358817987761604,
"request_tps": 24.325044162919706,
"group_elapsed_s": 5.262241083022673,
"group_output_tps": 24.324237141654443,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 1,
"round": 1,
"slot": 0,
"request_id": "428bad62-8fbf-41a1-9b36-07912506da11",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:47:48.859322+00:00",
"ttft_s": 0.37227433401858434,
"total_s": 5.454897999996319,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 25.18384370198632,
"request_tps": 23.465150035818517,
"group_elapsed_s": 5.455072959011886,
"group_output_tps": 23.464397444683396,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 1,
"round": 2,
"slot": 0,
"request_id": "b2de9565-6ff0-47a1-91e6-847b83994f6c",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:47:54.314442+00:00",
"ttft_s": 0.40532424999400973,
"total_s": 5.533015583991073,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 24.962500989743333,
"request_tps": 23.13385857259254,
"group_elapsed_s": 5.533179917023517,
"group_output_tps": 23.13317150707355,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 1,
"round": 3,
"slot": 0,
"request_id": "dc02daab-d1f2-49ac-8e29-01ee86148e0a",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:47:59.847681+00:00",
"ttft_s": 0.25306625000666827,
"total_s": 5.405130041996017,
"prompt_tokens": 20,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 24.844412873734196,
"request_tps": 23.681206373479203,
"group_elapsed_s": 5.405301624967251,
"group_output_tps": 23.680454650072466,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 1,
"round": 4,
"slot": 0,
"request_id": "677f2a25-31e5-43e7-b8fd-5e607453b904",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:05.253048+00:00",
"ttft_s": 0.42382929200539365,
"total_s": 5.569426125031896,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 24.875637200809187,
"request_tps": 22.982619236962577,
"group_elapsed_s": 5.569605667027645,
"group_output_tps": 22.981878368475286,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "warmup",
"concurrency": 4,
"round": 0,
"slot": 0,
"request_id": "7109b7b9-3690-49d6-8b20-969734c3dcad",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:10.822944+00:00",
"ttft_s": 1.1026008339831606,
"total_s": 8.694552083965391,
"prompt_tokens": 20,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 16.859960738064487,
"request_tps": 14.721862467884838,
"group_elapsed_s": 8.695109791995492,
"group_output_tps": 58.883672805527404,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "warmup",
"concurrency": 4,
"round": 0,
"slot": 1,
"request_id": "b34a5a99-e633-4f3a-892d-67af2b425779",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:10.823149+00:00",
"ttft_s": 1.1023455830290914,
"total_s": 8.694302666990552,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 16.859947782161328,
"request_tps": 14.722284799904022,
"group_elapsed_s": 8.695109791995492,
"group_output_tps": 58.883672805527404,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "warmup",
"concurrency": 4,
"round": 0,
"slot": 2,
"request_id": "1260b39f-a4b1-4d94-9406-70028f614ce0",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:10.823017+00:00",
"ttft_s": 1.0163157919887453,
"total_s": 8.694556291971821,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 16.670485901070972,
"request_tps": 14.721855342772315,
"group_elapsed_s": 8.695109791995492,
"group_output_tps": 58.883672805527404,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "warmup",
"concurrency": 4,
"round": 0,
"slot": 3,
"request_id": "b43d525a-50de-4e41-b00c-d1159d1ddfd4",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:10.822845+00:00",
"ttft_s": 1.1026693329913542,
"total_s": 8.694680040993262,
"prompt_tokens": 20,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 16.85982869664412,
"request_tps": 14.721645810600473,
"group_elapsed_s": 8.695109791995492,
"group_output_tps": 58.883672805527404,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "warmup",
"concurrency": 4,
"round": 1,
"slot": 0,
"request_id": "1da340a9-b91e-4f9b-8217-2af7c2fbc030",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:19.518058+00:00",
"ttft_s": 1.245261833013501,
"total_s": 9.168966500030365,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 16.154059922603068,
"request_tps": 13.960133892906699,
"group_elapsed_s": 9.169412165996619,
"group_output_tps": 55.83782152346415,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "warmup",
"concurrency": 4,
"round": 1,
"slot": 1,
"request_id": "23e076e6-7b50-4d08-a3ff-f79ddc53bafe",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:19.518183+00:00",
"ttft_s": 1.2451274580089375,
"total_s": 9.168911040993407,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 16.153899038190183,
"request_tps": 13.96021833211415,
"group_elapsed_s": 9.169412165996619,
"group_output_tps": 55.83782152346415,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "warmup",
"concurrency": 4,
"round": 1,
"slot": 2,
"request_id": "c777d58b-0520-4f3f-8537-6a163ab350f5",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:19.518319+00:00",
"ttft_s": 1.2449699160060845,
"total_s": 9.168749333010055,
"prompt_tokens": 20,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 16.153907531211612,
"request_tps": 13.960464546583719,
"group_elapsed_s": 9.169412165996619,
"group_output_tps": 55.83782152346415,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "warmup",
"concurrency": 4,
"round": 1,
"slot": 3,
"request_id": "5609655c-e54a-499f-99c6-c77c19c1a3e0",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:19.518407+00:00",
"ttft_s": 1.174761500034947,
"total_s": 9.16857587499544,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 16.012380822970083,
"request_tps": 13.960728661152478,
"group_elapsed_s": 9.169412165996619,
"group_output_tps": 55.83782152346415,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 0,
"slot": 0,
"request_id": "d2615398-7368-4f9a-9424-7a2e76d8955d",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:28.687706+00:00",
"ttft_s": 1.0508670000126585,
"total_s": 9.266459833015688,
"prompt_tokens": 20,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.580129468662143,
"request_tps": 13.813257954666332,
"group_elapsed_s": 9.267125833022874,
"group_output_tps": 55.24906095215814,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 0,
"slot": 1,
"request_id": "b307dcc2-b325-4005-a63f-54123aa28e31",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:28.687614+00:00",
"ttft_s": 1.0509756249957718,
"total_s": 9.266665957984515,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.579944570943383,
"request_tps": 13.812950696653772,
"group_elapsed_s": 9.267125833022874,
"group_output_tps": 55.24906095215814,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 0,
"slot": 2,
"request_id": "09c7607f-96f5-4a6d-a9e1-3637c8afc257",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:28.687778+00:00",
"ttft_s": 1.0507545830332674,
"total_s": 9.266332915984094,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.580156966749492,
"request_tps": 13.813447149001583,
"group_elapsed_s": 9.267125833022874,
"group_output_tps": 55.24906095215814,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 0,
"slot": 3,
"request_id": "0328d6c0-e3f4-4a86-917d-dbeb5b8deee7",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:28.687501+00:00",
"ttft_s": 0.9898037080420181,
"total_s": 9.266721500025596,
"prompt_tokens": 20,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.464693889308833,
"request_tps": 13.81286790583341,
"group_elapsed_s": 9.267125833022874,
"group_output_tps": 55.24906095215814,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 1,
"slot": 0,
"request_id": "74110182-284c-4250-9395-e3d0f2772eef",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:37.954714+00:00",
"ttft_s": 1.1083811249700375,
"total_s": 9.379890666983556,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.474805336299134,
"request_tps": 13.646214496992963,
"group_elapsed_s": 9.380457791965455,
"group_output_tps": 54.58155788926826,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 1,
"slot": 1,
"request_id": "fca02813-8ad6-4e99-bc76-9e922ea9a63a",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:37.954992+00:00",
"ttft_s": 1.1690869170124643,
"total_s": 9.379730832995847,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.589520299476966,
"request_tps": 13.646447033397155,
"group_elapsed_s": 9.380457791965455,
"group_output_tps": 54.58155788926826,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 1,
"slot": 2,
"request_id": "5eb13719-9063-4ddd-92c8-73332709ecfd",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:37.954812+00:00",
"ttft_s": 1.1692542500095442,
"total_s": 9.37985170801403,
"prompt_tokens": 20,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.589608509574806,
"request_tps": 13.646271176189105,
"group_elapsed_s": 9.380457791965455,
"group_output_tps": 54.58155788926826,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 1,
"slot": 3,
"request_id": "9d9a5b23-6950-4d5c-bad2-72f896d38009",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:37.954917+00:00",
"ttft_s": 1.1691360420081764,
"total_s": 9.379842250025831,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.58940202671112,
"request_tps": 13.646284936150979,
"group_elapsed_s": 9.380457791965455,
"group_output_tps": 54.58155788926826,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 2,
"slot": 0,
"request_id": "caf79143-9d5b-4fb2-830e-3bd2d38cc5d3",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:47.335341+00:00",
"ttft_s": 1.172440624970477,
"total_s": 9.591207291989122,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.204127286417465,
"request_tps": 13.345556623190662,
"group_elapsed_s": 9.591755541972816,
"group_output_tps": 53.37917524685921,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 2,
"slot": 1,
"request_id": "d77a5e81-922c-49ce-96c5-1fb443746984",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:47.335566+00:00",
"ttft_s": 1.2360592089826241,
"total_s": 9.591093584022019,
"prompt_tokens": 20,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.320104532711328,
"request_tps": 13.345714842490702,
"group_elapsed_s": 9.591755541972816,
"group_output_tps": 53.37917524685921,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 2,
"slot": 2,
"request_id": "9ccc94c9-d569-4785-9f48-e48e760b0e15",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:47.335447+00:00",
"ttft_s": 1.236121624999214,
"total_s": 9.591066916007549,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.320267882275031,
"request_tps": 13.345751950324445,
"group_elapsed_s": 9.591755541972816,
"group_output_tps": 53.37917524685921,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 2,
"slot": 3,
"request_id": "e7c5be37-35a8-47a5-869d-a3bee16750d4",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:47.335653+00:00",
"ttft_s": 1.2359632499865256,
"total_s": 9.590960291970987,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.32017298830757,
"request_tps": 13.345900316900948,
"group_elapsed_s": 9.591755541972816,
"group_output_tps": 53.37917524685921,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 3,
"slot": 0,
"request_id": "49d569ca-4eef-46fd-88f9-58cf68e01898",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:56.927355+00:00",
"ttft_s": 1.067110000003595,
"total_s": 9.073982625035569,
"prompt_tokens": 20,
"output_tokens": 124,
"prompt_tps": null,
"decode_tps": 15.486695718417879,
"request_tps": 13.665443843575128,
"group_elapsed_s": 9.254564541974105,
"group_output_tps": 54.89183177619698,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 3,
"slot": 1,
"request_id": "a34ede89-ca43-4a1c-a7c8-c916bfec24a5",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:56.927278+00:00",
"ttft_s": 1.0672226250171661,
"total_s": 9.253953957988415,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.635055652124883,
"request_tps": 13.831925313341854,
"group_elapsed_s": 9.254564541974105,
"group_output_tps": 54.89183177619698,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 3,
"slot": 2,
"request_id": "af0dfbe9-f1a5-434a-a714-98fad2af83a3",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:56.927205+00:00",
"ttft_s": 1.067305459000636,
"total_s": 9.254084500018507,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.634964539617723,
"request_tps": 13.831730194352993,
"group_elapsed_s": 9.254564541974105,
"group_output_tps": 54.89183177619698,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 3,
"slot": 3,
"request_id": "80dc0643-6f40-4ab5-805d-ace4e9f149bb",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:48:56.927082+00:00",
"ttft_s": 1.0054587500053458,
"total_s": 9.254091374983545,
"prompt_tokens": 20,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.517723460297553,
"request_tps": 13.831719918610334,
"group_elapsed_s": 9.254564541974105,
"group_output_tps": 54.89183177619698,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 4,
"slot": 0,
"request_id": "12e42bf2-6a75-4f89-a598-ac14d5b67a78",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:49:06.181834+00:00",
"ttft_s": 1.2329518750193529,
"total_s": 9.273020750028081,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.920261628338482,
"request_tps": 13.803484695061464,
"group_elapsed_s": 9.273481999989599,
"group_output_tps": 55.21119251652985,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 4,
"slot": 1,
"request_id": "5f2cf183-3654-45be-81ed-4eebfb57bce1",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:49:06.182126+00:00",
"ttft_s": 1.1721314580063336,
"total_s": 9.272628374979831,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.801499748959015,
"request_tps": 13.804068795141205,
"group_elapsed_s": 9.273481999989599,
"group_output_tps": 55.21119251652985,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 4,
"slot": 2,
"request_id": "746da721-ed00-4c80-bc92-ed115313463e",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:49:06.182026+00:00",
"ttft_s": 1.2329447079682723,
"total_s": 9.272807125002146,
"prompt_tokens": 20,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.920670449385964,
"request_tps": 13.80380269690667,
"group_elapsed_s": 9.273481999989599,
"group_output_tps": 55.21119251652985,
"error": null
},
{
"run_id": "20260831T114727Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"phase": "measured",
"concurrency": 4,
"round": 4,
"slot": 3,
"request_id": "5c45d61d-9d4a-4fc4-931d-5193cdcaa598",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:49:06.181940+00:00",
"ttft_s": 1.2330110420007259,
"total_s": 9.272864667000249,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.920687859538933,
"request_tps": 13.803717038545729,
"group_elapsed_s": 9.273481999989599,
"group_output_tps": 55.21119251652985,
"error": null
}
]

54
benchmarks/inference/results/20260831T114727Z/summary.json Normal file
View File

@@ -0,0 +1,54 @@
[
{
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"concurrency": 1,
"requests": 5,
"successes": 5,
"failures": 0,
"ttft_s_mean": 0.3338013501954265,
"ttft_s_p50": 0.37227433401858434,
"ttft_s_p95": 0.4201282836031169,
"total_s_mean": 5.444907250197138,
"total_s_p50": 5.454897999996319,
"total_s_p95": 5.562144016823732,
"prompt_tps_mean": null,
"prompt_tps_p50": null,
"prompt_tps_p95": null,
"decode_tps_mean": 25.04504255080693,
"decode_tps_p50": 24.962500989743333,
"decode_tps_p95": 25.32382313060655,
"request_tps_mean": 23.517575676354507,
"request_tps_p50": 23.465150035818517,
"request_tps_p95": 24.196276605031606,
"aggregate_output_tps_mean": 23.516827822391825,
"aggregate_output_tps_p50": 23.464397444683396,
"aggregate_output_tps_p95": 24.195480643338048
},
{
"runtime": "vllm-metal",
"model": "qwen3.8-27b-nvfp4",
"concurrency": 4,
"requests": 20,
"successes": 20,
"failures": 0,
"ttft_s_mean": 1.1358489938022103,
"ttft_s_p50": 1.1691114795103204,
"ttft_s_p95": 1.2360623297834537,
"total_s_mean": 9.343862856202758,
"total_s_p50": 9.272835896001197,
"total_s_p95": 9.591099269420374,
"prompt_tps_mean": null,
"prompt_tps_p50": null,
"prompt_tps_p95": null,
"decode_tps_mean": 15.572524640705884,
"decode_tps_p50": 15.580143217705817,
"decode_tps_p95": 15.920671319893613,
"request_tps_mean": 13.679827878866371,
"request_tps_p50": 13.803600866803595,
"request_tps_p95": 13.831739950302437,
"aggregate_output_tps_mean": 54.66256367620249,
"aggregate_output_tps_p50": 54.89183177619698,
"aggregate_output_tps_p95": 55.24148726503248
}
]

135
benchmarks/inference/results/20260831T114928Z/metadata.json Normal file
View File

File diff suppressed because one or more lines are too long

36
benchmarks/inference/results/20260831T114928Z/raw.csv Normal file
View File

@@ -0,0 +1,36 @@
run_id,runtime,model,phase,concurrency,round,slot,request_id,prompt_name,prompt_chars,max_tokens,started_at,ttft_s,total_s,prompt_tokens,output_tokens,prompt_tps,decode_tps,request_tps,group_elapsed_s,group_output_tps,error
20260831T114928Z,ollama,qwen3.8:27b-mlx,warmup,1,0,0,0bd09e14-c217-44ad-b0a1-98600e1bf40e,short,106,128,2026-08-31T11:49:33.251646+00:00,3.627692041976843,5.577126249962021,20,96,116.69505030650684,46.54721674973356,17.213166010121025,5.578097124991473,17.210170036282175,
20260831T114928Z,ollama,qwen3.8:27b-mlx,warmup,1,1,0,ada11e2f-0b97-42ec-a5be-5175f0bb29b1,medium,252,128,2026-08-31T11:49:38.829357+00:00,0.3621791250188835,2.7543381249997765,43,128,157.23847751774449,52.253166030964316,46.472144737135494,2.7547111669555306,46.46585149667936,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,1,0,0,a3bd0047-d992-4553-9b17-4472b6290acc,short,106,128,2026-08-31T11:49:41.584102+00:00,0.11936691601295024,2.1721785410190932,20,96,448.5805263612496,45.72630609321048,44.19526212378514,2.172526082955301,44.18819214792146,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,1,1,0,a8033a77-d60f-42b4-a2a9-8f6a97ef3496,medium,252,128,2026-08-31T11:49:43.757773+00:00,0.14179470797535032,2.78225216595456,43,128,816.5668113066018,47.377908815894095,46.00589463683088,2.7835853330325335,45.98386062788756,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,1,2,0,9cc954fc-95f5-474e-b219-8a89d7091934,code,267,128,2026-08-31T11:49:46.540349+00:00,0.45632574998307973,2.7544727920321748,54,128,146.71421293937848,54.2491762219149,46.46987269951035,2.754789875005372,46.46452390484062,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,1,3,0,e8ee5060-7c14-4c34-bd5b-76d7e51fef07,short,106,128,2026-08-31T11:49:49.295554+00:00,0.0983880830463022,2.5220755000482313,20,96,2758620.689655172,38.503422116977305,38.063888253212134,2.522709292010404,38.054325285929174,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,1,4,0,edd132c8-7aaa-4eb0-98e8-2b2d711f2cf9,medium,252,128,2026-08-31T11:49:51.817976+00:00,0.09851625002920628,3.6179742500535212,43,128,3245283.0188679243,35.649936286183205,35.3789140423281,3.6182677499600686,35.37604424145024,
20260831T114928Z,ollama,qwen3.8:27b-mlx,warmup,4,0,0,f0df217a-d72c-4b5c-b4a7-87c267716da0,short,106,128,2026-08-31T11:49:55.436979+00:00,0.11441224999725819,2.9100067079998553,20,96,4138216.428719222,33.47564745836445,32.989614675487815,12.524156374973245,35.77087243139417,
20260831T114928Z,ollama,qwen3.8:27b-mlx,warmup,4,0,1,9565d3c0-e753-4a7b-be43-115556beec5b,medium,252,128,2026-08-31T11:49:55.437104+00:00,6.277438874996733,9.582254499953706,43,128,3865168.539325843,37.881971614565146,13.358025504396528,12.524156374973245,35.77087243139417,
20260831T114928Z,ollama,qwen3.8:27b-mlx,warmup,4,0,2,42b7e290-9d92-4647-bbdb-3b3ba99f4c3e,code,267,128,2026-08-31T11:49:55.436744+00:00,3.0367828750167973,6.193101375014521,54,128,771.3321549092078,39.94659431201623,20.66815836672783,12.524156374973245,35.77087243139417,
20260831T114928Z,ollama,qwen3.8:27b-mlx,warmup,4,0,3,c22aaea0-7394-47e8-b6ab-25444fc113d1,short,106,128,2026-08-31T11:49:55.437235+00:00,9.661167457990814,12.522873707988765,20,96,3966679.888932963,32.71048444655768,7.665972063485584,12.524156374973245,35.77087243139417,
20260831T114928Z,ollama,qwen3.8:27b-mlx,warmup,4,1,0,84f401af-117d-4956-b939-fdbb1e1ceecf,medium,252,128,2026-08-31T11:50:07.961234+00:00,3.091476584027987,6.373854791978374,43,128,5375000.0,38.1244834091544,20.08203891953902,12.858689875050914,37.32884179214249,
20260831T114928Z,ollama,qwen3.8:27b-mlx,warmup,4,1,1,b3c9f50e-fcf6-4ad6-b00f-9de925c9092a,code,267,128,2026-08-31T11:50:07.961465+00:00,9.849772959016263,12.85736475000158,54,128,11076923.076923078,41.56311628897905,9.955383742223248,12.858689875050914,37.32884179214249,
20260831T114928Z,ollama,qwen3.8:27b-mlx,warmup,4,1,2,5f528a81-943d-445a-a2f2-9b986c01c96e,short,106,128,2026-08-31T11:50:07.961041+00:00,0.11284091597190127,3.010363540961407,20,96,1476995.7905619969,32.336156802028626,31.889836125686298,12.858689875050914,37.32884179214249,
20260831T114928Z,ollama,qwen3.8:27b-mlx,warmup,4,1,3,45991adb-aae4-4122-bb0d-21d34b549b46,medium,252,128,2026-08-31T11:50:07.960837+00:00,6.452098417037632,9.774627042002976,43,128,7319148.936170213,37.70880560635668,13.095128791100224,12.858689875050914,37.32884179214249,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,0,0,1cd3b269-0ed7-4470-9bf5-9aba6dbdda3b,short,106,128,2026-08-31T11:50:20.819745+00:00,9.799659290991258,12.57556495803874,20,96,2461538.461538462,33.69646737394496,7.633851864335801,12.576361375045963,35.781414558656905,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,0,1,b2258258-f465-4131-bb56-cdead87e67c5,medium,252,128,2026-08-31T11:50:20.820473+00:00,2.9737471669795923,6.33726870897226,43,128,4755585.04755585,37.255703188397476,20.19797579654126,12.576361375045963,35.781414558656905,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,0,2,9bb209a2-19a6-4172-a166-c29a75683a37,code,267,128,2026-08-31T11:50:20.819981+00:00,0.11236062500393018,2.89358854200691,54,128,3661016.9491525427,44.88156766233007,44.23572949014475,12.576361375045963,35.781414558656905,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,0,3,c74fbeef-16a3-40b2-ade6-0b24b3ee0147,short,106,128,2026-08-31T11:50:20.820727+00:00,6.417795290995855,9.71814520796761,20,98,2608582.235554976,29.056692998778793,10.084228821735737,12.576361375045963,35.781414558656905,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,1,0,844d6c49-9989-400d-bb5d-31f8490da904,medium,252,128,2026-08-31T11:50:33.396890+00:00,10.48226895800326,14.052842833043542,43,128,5083944.194845118,35.11947732750843,9.108477303896379,14.054668250028044,34.152353613828076,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,1,1,213efda7-5da6-47a7-9517-f63ad082e62e,code,267,128,2026-08-31T11:50:33.397263+00:00,0.11672520800493658,3.508996458025649,54,128,3630252.100840336,36.93011583511385,36.47766577456727,14.054668250028044,34.152353613828076,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,1,2,d03d55d9-a506-44a9-887e-e0ab77392c8d,short,106,128,2026-08-31T11:50:33.397083+00:00,3.606067124987021,6.893670374993235,20,96,3356267.830172848,28.527554720530254,13.925818145909567,14.054668250028044,34.152353613828076,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,1,3,55be3260-b3a0-422a-b918-a81261da6a84,medium,252,128,2026-08-31T11:50:33.396391+00:00,6.9750645000021905,10.400937125028577,43,128,6745098.039215687,36.57719818551596,12.30658338391295,14.054668250028044,34.152353613828076,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,2,0,930b484d-3177-4ee9-9d72-5442ee8cd7c9,code,267,128,2026-08-31T11:50:47.451672+00:00,0.11444733297685161,3.540263083006721,54,128,4248957.431741286,36.58630699945727,36.15550511327833,14.210740375041496,33.918007597024484,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,2,1,231f3cbb-61f2-4042-8a99-def6a8698a7d,short,106,128,2026-08-31T11:50:47.451387+00:00,10.67886250000447,14.209862457995769,20,98,2840102.243680773,27.132075592063398,6.896618478165229,14.210740375041496,33.918007597024484,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,2,2,32437705-bf67-438f-bad7-0bb10d0e2030,medium,252,128,2026-08-31T11:50:47.451856+00:00,3.622358249966055,7.2201729999505915,43,128,4777777.777777778,34.87549075162609,17.72810706902396,14.210740375041496,33.918007597024484,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,2,3,a16a724c-155d-4e2d-ba28-3a8683af4d40,code,267,128,2026-08-31T11:50:47.451136+00:00,7.304396207968239,10.594903749995865,54,128,17515407.071034707,38.04511611184116,12.081280115456448,14.210740375041496,33.918007597024484,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,3,0,f6ab47e7-5bdf-4b5c-be3f-0e639727a2bb,short,106,128,2026-08-31T11:51:01.661760+00:00,10.42098737496417,14.485269291966688,20,96,2105263.1578947366,23.198198961620538,6.627422525947802,14.485774166998453,30.926893850149572,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,3,1,8e015249-8b0c-4560-b93d-603afb968f5b,medium,252,128,2026-08-31T11:51:01.662046+00:00,0.11227079102536663,3.6567685829941183,43,128,3571131.9657835728,35.39152606628793,35.00358228717748,14.485774166998453,30.926893850149572,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,3,2,d2848890-f070-45de-8a2f-4bb2b8788a87,code,267,128,2026-08-31T11:51:01.661862+00:00,7.207546124991495,10.33818608301226,54,128,9458749.343142407,39.94848206301452,12.381282264819164,14.485774166998453,30.926893850149572,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,3,3,aba512eb-9825-4e49-9ab8-0c75678ff654,short,106,128,2026-08-31T11:51:01.661952+00:00,3.738764916022774,7.12777191598434,20,96,2652168.1474605487,27.731643965337334,13.468444435590849,14.485774166998453,30.926893850149572,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,4,0,4c24ae09-884b-4b94-ae8e-37346f1408f2,medium,252,128,2026-08-31T11:51:16.148220+00:00,10.924698917020578,15.198258958000224,43,128,7068880.486602006,29.434480556585335,8.422017308280036,15.199551291007083,31.579879616840746,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,4,1,c94ceb91-b83d-4d47-a36d-71741993f366,code,267,128,2026-08-31T11:51:16.148464+00:00,0.12565479200566188,3.500943167018704,54,128,4025044.7227191413,37.03108618648992,36.56157609350765,15.199551291007083,31.579879616840746,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,4,2,0672967f-d135-44b0-b89f-7995f59b2d8c,short,106,128,2026-08-31T11:51:16.148378+00:00,7.600669999956153,10.846751042001415,20,96,2666666.6666666665,28.898123213254255,8.850576511645125,15.199551291007083,31.579879616840746,
20260831T114928Z,ollama,qwen3.8:27b-mlx,measured,4,4,3,8185344b-c228-457a-8c3d-6a7b5f803594,medium,252,128,2026-08-31T11:51:16.147977+00:00,3.5925854169763625,7.520279875025153,43,128,4229785.559708833,31.98591420303284,17.02064313126004,15.199551291007083,31.579879616840746,
1 run_id runtime model phase concurrency round slot request_id prompt_name prompt_chars max_tokens started_at ttft_s total_s prompt_tokens output_tokens prompt_tps decode_tps request_tps group_elapsed_s group_output_tps error
2 20260831T114928Z ollama qwen3.8:27b-mlx warmup 1 0 0 0bd09e14-c217-44ad-b0a1-98600e1bf40e short 106 128 2026-08-31T11:49:33.251646+00:00 3.627692041976843 5.577126249962021 20 96 116.69505030650684 46.54721674973356 17.213166010121025 5.578097124991473 17.210170036282175
3 20260831T114928Z ollama qwen3.8:27b-mlx warmup 1 1 0 ada11e2f-0b97-42ec-a5be-5175f0bb29b1 medium 252 128 2026-08-31T11:49:38.829357+00:00 0.3621791250188835 2.7543381249997765 43 128 157.23847751774449 52.253166030964316 46.472144737135494 2.7547111669555306 46.46585149667936
4 20260831T114928Z ollama qwen3.8:27b-mlx measured 1 0 0 a3bd0047-d992-4553-9b17-4472b6290acc short 106 128 2026-08-31T11:49:41.584102+00:00 0.11936691601295024 2.1721785410190932 20 96 448.5805263612496 45.72630609321048 44.19526212378514 2.172526082955301 44.18819214792146
5 20260831T114928Z ollama qwen3.8:27b-mlx measured 1 1 0 a8033a77-d60f-42b4-a2a9-8f6a97ef3496 medium 252 128 2026-08-31T11:49:43.757773+00:00 0.14179470797535032 2.78225216595456 43 128 816.5668113066018 47.377908815894095 46.00589463683088 2.7835853330325335 45.98386062788756
6 20260831T114928Z ollama qwen3.8:27b-mlx measured 1 2 0 9cc954fc-95f5-474e-b219-8a89d7091934 code 267 128 2026-08-31T11:49:46.540349+00:00 0.45632574998307973 2.7544727920321748 54 128 146.71421293937848 54.2491762219149 46.46987269951035 2.754789875005372 46.46452390484062
7 20260831T114928Z ollama qwen3.8:27b-mlx measured 1 3 0 e8ee5060-7c14-4c34-bd5b-76d7e51fef07 short 106 128 2026-08-31T11:49:49.295554+00:00 0.0983880830463022 2.5220755000482313 20 96 2758620.689655172 38.503422116977305 38.063888253212134 2.522709292010404 38.054325285929174
8 20260831T114928Z ollama qwen3.8:27b-mlx measured 1 4 0 edd132c8-7aaa-4eb0-98e8-2b2d711f2cf9 medium 252 128 2026-08-31T11:49:51.817976+00:00 0.09851625002920628 3.6179742500535212 43 128 3245283.0188679243 35.649936286183205 35.3789140423281 3.6182677499600686 35.37604424145024
9 20260831T114928Z ollama qwen3.8:27b-mlx warmup 4 0 0 f0df217a-d72c-4b5c-b4a7-87c267716da0 short 106 128 2026-08-31T11:49:55.436979+00:00 0.11441224999725819 2.9100067079998553 20 96 4138216.428719222 33.47564745836445 32.989614675487815 12.524156374973245 35.77087243139417
10 20260831T114928Z ollama qwen3.8:27b-mlx warmup 4 0 1 9565d3c0-e753-4a7b-be43-115556beec5b medium 252 128 2026-08-31T11:49:55.437104+00:00 6.277438874996733 9.582254499953706 43 128 3865168.539325843 37.881971614565146 13.358025504396528 12.524156374973245 35.77087243139417
11 20260831T114928Z ollama qwen3.8:27b-mlx warmup 4 0 2 42b7e290-9d92-4647-bbdb-3b3ba99f4c3e code 267 128 2026-08-31T11:49:55.436744+00:00 3.0367828750167973 6.193101375014521 54 128 771.3321549092078 39.94659431201623 20.66815836672783 12.524156374973245 35.77087243139417
12 20260831T114928Z ollama qwen3.8:27b-mlx warmup 4 0 3 c22aaea0-7394-47e8-b6ab-25444fc113d1 short 106 128 2026-08-31T11:49:55.437235+00:00 9.661167457990814 12.522873707988765 20 96 3966679.888932963 32.71048444655768 7.665972063485584 12.524156374973245 35.77087243139417
13 20260831T114928Z ollama qwen3.8:27b-mlx warmup 4 1 0 84f401af-117d-4956-b939-fdbb1e1ceecf medium 252 128 2026-08-31T11:50:07.961234+00:00 3.091476584027987 6.373854791978374 43 128 5375000.0 38.1244834091544 20.08203891953902 12.858689875050914 37.32884179214249
14 20260831T114928Z ollama qwen3.8:27b-mlx warmup 4 1 1 b3c9f50e-fcf6-4ad6-b00f-9de925c9092a code 267 128 2026-08-31T11:50:07.961465+00:00 9.849772959016263 12.85736475000158 54 128 11076923.076923078 41.56311628897905 9.955383742223248 12.858689875050914 37.32884179214249
15 20260831T114928Z ollama qwen3.8:27b-mlx warmup 4 1 2 5f528a81-943d-445a-a2f2-9b986c01c96e short 106 128 2026-08-31T11:50:07.961041+00:00 0.11284091597190127 3.010363540961407 20 96 1476995.7905619969 32.336156802028626 31.889836125686298 12.858689875050914 37.32884179214249
16 20260831T114928Z ollama qwen3.8:27b-mlx warmup 4 1 3 45991adb-aae4-4122-bb0d-21d34b549b46 medium 252 128 2026-08-31T11:50:07.960837+00:00 6.452098417037632 9.774627042002976 43 128 7319148.936170213 37.70880560635668 13.095128791100224 12.858689875050914 37.32884179214249
17 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 0 0 1cd3b269-0ed7-4470-9bf5-9aba6dbdda3b short 106 128 2026-08-31T11:50:20.819745+00:00 9.799659290991258 12.57556495803874 20 96 2461538.461538462 33.69646737394496 7.633851864335801 12.576361375045963 35.781414558656905
18 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 0 1 b2258258-f465-4131-bb56-cdead87e67c5 medium 252 128 2026-08-31T11:50:20.820473+00:00 2.9737471669795923 6.33726870897226 43 128 4755585.04755585 37.255703188397476 20.19797579654126 12.576361375045963 35.781414558656905
19 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 0 2 9bb209a2-19a6-4172-a166-c29a75683a37 code 267 128 2026-08-31T11:50:20.819981+00:00 0.11236062500393018 2.89358854200691 54 128 3661016.9491525427 44.88156766233007 44.23572949014475 12.576361375045963 35.781414558656905
20 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 0 3 c74fbeef-16a3-40b2-ade6-0b24b3ee0147 short 106 128 2026-08-31T11:50:20.820727+00:00 6.417795290995855 9.71814520796761 20 98 2608582.235554976 29.056692998778793 10.084228821735737 12.576361375045963 35.781414558656905
21 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 1 0 844d6c49-9989-400d-bb5d-31f8490da904 medium 252 128 2026-08-31T11:50:33.396890+00:00 10.48226895800326 14.052842833043542 43 128 5083944.194845118 35.11947732750843 9.108477303896379 14.054668250028044 34.152353613828076
22 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 1 1 213efda7-5da6-47a7-9517-f63ad082e62e code 267 128 2026-08-31T11:50:33.397263+00:00 0.11672520800493658 3.508996458025649 54 128 3630252.100840336 36.93011583511385 36.47766577456727 14.054668250028044 34.152353613828076
23 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 1 2 d03d55d9-a506-44a9-887e-e0ab77392c8d short 106 128 2026-08-31T11:50:33.397083+00:00 3.606067124987021 6.893670374993235 20 96 3356267.830172848 28.527554720530254 13.925818145909567 14.054668250028044 34.152353613828076
24 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 1 3 55be3260-b3a0-422a-b918-a81261da6a84 medium 252 128 2026-08-31T11:50:33.396391+00:00 6.9750645000021905 10.400937125028577 43 128 6745098.039215687 36.57719818551596 12.30658338391295 14.054668250028044 34.152353613828076
25 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 2 0 930b484d-3177-4ee9-9d72-5442ee8cd7c9 code 267 128 2026-08-31T11:50:47.451672+00:00 0.11444733297685161 3.540263083006721 54 128 4248957.431741286 36.58630699945727 36.15550511327833 14.210740375041496 33.918007597024484
26 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 2 1 231f3cbb-61f2-4042-8a99-def6a8698a7d short 106 128 2026-08-31T11:50:47.451387+00:00 10.67886250000447 14.209862457995769 20 98 2840102.243680773 27.132075592063398 6.896618478165229 14.210740375041496 33.918007597024484
27 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 2 2 32437705-bf67-438f-bad7-0bb10d0e2030 medium 252 128 2026-08-31T11:50:47.451856+00:00 3.622358249966055 7.2201729999505915 43 128 4777777.777777778 34.87549075162609 17.72810706902396 14.210740375041496 33.918007597024484
28 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 2 3 a16a724c-155d-4e2d-ba28-3a8683af4d40 code 267 128 2026-08-31T11:50:47.451136+00:00 7.304396207968239 10.594903749995865 54 128 17515407.071034707 38.04511611184116 12.081280115456448 14.210740375041496 33.918007597024484
29 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 3 0 f6ab47e7-5bdf-4b5c-be3f-0e639727a2bb short 106 128 2026-08-31T11:51:01.661760+00:00 10.42098737496417 14.485269291966688 20 96 2105263.1578947366 23.198198961620538 6.627422525947802 14.485774166998453 30.926893850149572
30 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 3 1 8e015249-8b0c-4560-b93d-603afb968f5b medium 252 128 2026-08-31T11:51:01.662046+00:00 0.11227079102536663 3.6567685829941183 43 128 3571131.9657835728 35.39152606628793 35.00358228717748 14.485774166998453 30.926893850149572
31 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 3 2 d2848890-f070-45de-8a2f-4bb2b8788a87 code 267 128 2026-08-31T11:51:01.661862+00:00 7.207546124991495 10.33818608301226 54 128 9458749.343142407 39.94848206301452 12.381282264819164 14.485774166998453 30.926893850149572
32 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 3 3 aba512eb-9825-4e49-9ab8-0c75678ff654 short 106 128 2026-08-31T11:51:01.661952+00:00 3.738764916022774 7.12777191598434 20 96 2652168.1474605487 27.731643965337334 13.468444435590849 14.485774166998453 30.926893850149572
33 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 4 0 4c24ae09-884b-4b94-ae8e-37346f1408f2 medium 252 128 2026-08-31T11:51:16.148220+00:00 10.924698917020578 15.198258958000224 43 128 7068880.486602006 29.434480556585335 8.422017308280036 15.199551291007083 31.579879616840746
34 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 4 1 c94ceb91-b83d-4d47-a36d-71741993f366 code 267 128 2026-08-31T11:51:16.148464+00:00 0.12565479200566188 3.500943167018704 54 128 4025044.7227191413 37.03108618648992 36.56157609350765 15.199551291007083 31.579879616840746
35 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 4 2 0672967f-d135-44b0-b89f-7995f59b2d8c short 106 128 2026-08-31T11:51:16.148378+00:00 7.600669999956153 10.846751042001415 20 96 2666666.6666666665 28.898123213254255 8.850576511645125 15.199551291007083 31.579879616840746
36 20260831T114928Z ollama qwen3.8:27b-mlx measured 4 4 3 8185344b-c228-457a-8c3d-6a7b5f803594 medium 252 128 2026-08-31T11:51:16.147977+00:00 3.5925854169763625 7.520279875025153 43 128 4229785.559708833 31.98591420303284 17.02064313126004 15.199551291007083 31.579879616840746

842
benchmarks/inference/results/20260831T114928Z/raw.json Normal file
View File

@@ -0,0 +1,842 @@
[
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "warmup",
"concurrency": 1,
"round": 0,
"slot": 0,
"request_id": "0bd09e14-c217-44ad-b0a1-98600e1bf40e",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:49:33.251646+00:00",
"ttft_s": 3.627692041976843,
"total_s": 5.577126249962021,
"prompt_tokens": 20,
"output_tokens": 96,
"prompt_tps": 116.69505030650684,
"decode_tps": 46.54721674973356,
"request_tps": 17.213166010121025,
"group_elapsed_s": 5.578097124991473,
"group_output_tps": 17.210170036282175,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "warmup",
"concurrency": 1,
"round": 1,
"slot": 0,
"request_id": "ada11e2f-0b97-42ec-a5be-5175f0bb29b1",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:49:38.829357+00:00",
"ttft_s": 0.3621791250188835,
"total_s": 2.7543381249997765,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 157.23847751774449,
"decode_tps": 52.253166030964316,
"request_tps": 46.472144737135494,
"group_elapsed_s": 2.7547111669555306,
"group_output_tps": 46.46585149667936,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 1,
"round": 0,
"slot": 0,
"request_id": "a3bd0047-d992-4553-9b17-4472b6290acc",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:49:41.584102+00:00",
"ttft_s": 0.11936691601295024,
"total_s": 2.1721785410190932,
"prompt_tokens": 20,
"output_tokens": 96,
"prompt_tps": 448.5805263612496,
"decode_tps": 45.72630609321048,
"request_tps": 44.19526212378514,
"group_elapsed_s": 2.172526082955301,
"group_output_tps": 44.18819214792146,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 1,
"round": 1,
"slot": 0,
"request_id": "a8033a77-d60f-42b4-a2a9-8f6a97ef3496",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:49:43.757773+00:00",
"ttft_s": 0.14179470797535032,
"total_s": 2.78225216595456,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 816.5668113066018,
"decode_tps": 47.377908815894095,
"request_tps": 46.00589463683088,
"group_elapsed_s": 2.7835853330325335,
"group_output_tps": 45.98386062788756,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 1,
"round": 2,
"slot": 0,
"request_id": "9cc954fc-95f5-474e-b219-8a89d7091934",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:49:46.540349+00:00",
"ttft_s": 0.45632574998307973,
"total_s": 2.7544727920321748,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 146.71421293937848,
"decode_tps": 54.2491762219149,
"request_tps": 46.46987269951035,
"group_elapsed_s": 2.754789875005372,
"group_output_tps": 46.46452390484062,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 1,
"round": 3,
"slot": 0,
"request_id": "e8ee5060-7c14-4c34-bd5b-76d7e51fef07",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:49:49.295554+00:00",
"ttft_s": 0.0983880830463022,
"total_s": 2.5220755000482313,
"prompt_tokens": 20,
"output_tokens": 96,
"prompt_tps": 2758620.689655172,
"decode_tps": 38.503422116977305,
"request_tps": 38.063888253212134,
"group_elapsed_s": 2.522709292010404,
"group_output_tps": 38.054325285929174,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 1,
"round": 4,
"slot": 0,
"request_id": "edd132c8-7aaa-4eb0-98e8-2b2d711f2cf9",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:49:51.817976+00:00",
"ttft_s": 0.09851625002920628,
"total_s": 3.6179742500535212,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 3245283.0188679243,
"decode_tps": 35.649936286183205,
"request_tps": 35.3789140423281,
"group_elapsed_s": 3.6182677499600686,
"group_output_tps": 35.37604424145024,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "warmup",
"concurrency": 4,
"round": 0,
"slot": 0,
"request_id": "f0df217a-d72c-4b5c-b4a7-87c267716da0",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:49:55.436979+00:00",
"ttft_s": 0.11441224999725819,
"total_s": 2.9100067079998553,
"prompt_tokens": 20,
"output_tokens": 96,
"prompt_tps": 4138216.428719222,
"decode_tps": 33.47564745836445,
"request_tps": 32.989614675487815,
"group_elapsed_s": 12.524156374973245,
"group_output_tps": 35.77087243139417,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "warmup",
"concurrency": 4,
"round": 0,
"slot": 1,
"request_id": "9565d3c0-e753-4a7b-be43-115556beec5b",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:49:55.437104+00:00",
"ttft_s": 6.277438874996733,
"total_s": 9.582254499953706,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 3865168.539325843,
"decode_tps": 37.881971614565146,
"request_tps": 13.358025504396528,
"group_elapsed_s": 12.524156374973245,
"group_output_tps": 35.77087243139417,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "warmup",
"concurrency": 4,
"round": 0,
"slot": 2,
"request_id": "42b7e290-9d92-4647-bbdb-3b3ba99f4c3e",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:49:55.436744+00:00",
"ttft_s": 3.0367828750167973,
"total_s": 6.193101375014521,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 771.3321549092078,
"decode_tps": 39.94659431201623,
"request_tps": 20.66815836672783,
"group_elapsed_s": 12.524156374973245,
"group_output_tps": 35.77087243139417,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "warmup",
"concurrency": 4,
"round": 0,
"slot": 3,
"request_id": "c22aaea0-7394-47e8-b6ab-25444fc113d1",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:49:55.437235+00:00",
"ttft_s": 9.661167457990814,
"total_s": 12.522873707988765,
"prompt_tokens": 20,
"output_tokens": 96,
"prompt_tps": 3966679.888932963,
"decode_tps": 32.71048444655768,
"request_tps": 7.665972063485584,
"group_elapsed_s": 12.524156374973245,
"group_output_tps": 35.77087243139417,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "warmup",
"concurrency": 4,
"round": 1,
"slot": 0,
"request_id": "84f401af-117d-4956-b939-fdbb1e1ceecf",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:50:07.961234+00:00",
"ttft_s": 3.091476584027987,
"total_s": 6.373854791978374,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 5375000.0,
"decode_tps": 38.1244834091544,
"request_tps": 20.08203891953902,
"group_elapsed_s": 12.858689875050914,
"group_output_tps": 37.32884179214249,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "warmup",
"concurrency": 4,
"round": 1,
"slot": 1,
"request_id": "b3c9f50e-fcf6-4ad6-b00f-9de925c9092a",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:50:07.961465+00:00",
"ttft_s": 9.849772959016263,
"total_s": 12.85736475000158,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 11076923.076923078,
"decode_tps": 41.56311628897905,
"request_tps": 9.955383742223248,
"group_elapsed_s": 12.858689875050914,
"group_output_tps": 37.32884179214249,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "warmup",
"concurrency": 4,
"round": 1,
"slot": 2,
"request_id": "5f528a81-943d-445a-a2f2-9b986c01c96e",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:50:07.961041+00:00",
"ttft_s": 0.11284091597190127,
"total_s": 3.010363540961407,
"prompt_tokens": 20,
"output_tokens": 96,
"prompt_tps": 1476995.7905619969,
"decode_tps": 32.336156802028626,
"request_tps": 31.889836125686298,
"group_elapsed_s": 12.858689875050914,
"group_output_tps": 37.32884179214249,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "warmup",
"concurrency": 4,
"round": 1,
"slot": 3,
"request_id": "45991adb-aae4-4122-bb0d-21d34b549b46",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:50:07.960837+00:00",
"ttft_s": 6.452098417037632,
"total_s": 9.774627042002976,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 7319148.936170213,
"decode_tps": 37.70880560635668,
"request_tps": 13.095128791100224,
"group_elapsed_s": 12.858689875050914,
"group_output_tps": 37.32884179214249,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 0,
"slot": 0,
"request_id": "1cd3b269-0ed7-4470-9bf5-9aba6dbdda3b",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:50:20.819745+00:00",
"ttft_s": 9.799659290991258,
"total_s": 12.57556495803874,
"prompt_tokens": 20,
"output_tokens": 96,
"prompt_tps": 2461538.461538462,
"decode_tps": 33.69646737394496,
"request_tps": 7.633851864335801,
"group_elapsed_s": 12.576361375045963,
"group_output_tps": 35.781414558656905,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 0,
"slot": 1,
"request_id": "b2258258-f465-4131-bb56-cdead87e67c5",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:50:20.820473+00:00",
"ttft_s": 2.9737471669795923,
"total_s": 6.33726870897226,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 4755585.04755585,
"decode_tps": 37.255703188397476,
"request_tps": 20.19797579654126,
"group_elapsed_s": 12.576361375045963,
"group_output_tps": 35.781414558656905,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 0,
"slot": 2,
"request_id": "9bb209a2-19a6-4172-a166-c29a75683a37",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:50:20.819981+00:00",
"ttft_s": 0.11236062500393018,
"total_s": 2.89358854200691,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 3661016.9491525427,
"decode_tps": 44.88156766233007,
"request_tps": 44.23572949014475,
"group_elapsed_s": 12.576361375045963,
"group_output_tps": 35.781414558656905,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 0,
"slot": 3,
"request_id": "c74fbeef-16a3-40b2-ade6-0b24b3ee0147",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:50:20.820727+00:00",
"ttft_s": 6.417795290995855,
"total_s": 9.71814520796761,
"prompt_tokens": 20,
"output_tokens": 98,
"prompt_tps": 2608582.235554976,
"decode_tps": 29.056692998778793,
"request_tps": 10.084228821735737,
"group_elapsed_s": 12.576361375045963,
"group_output_tps": 35.781414558656905,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 1,
"slot": 0,
"request_id": "844d6c49-9989-400d-bb5d-31f8490da904",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:50:33.396890+00:00",
"ttft_s": 10.48226895800326,
"total_s": 14.052842833043542,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 5083944.194845118,
"decode_tps": 35.11947732750843,
"request_tps": 9.108477303896379,
"group_elapsed_s": 14.054668250028044,
"group_output_tps": 34.152353613828076,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 1,
"slot": 1,
"request_id": "213efda7-5da6-47a7-9517-f63ad082e62e",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:50:33.397263+00:00",
"ttft_s": 0.11672520800493658,
"total_s": 3.508996458025649,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 3630252.100840336,
"decode_tps": 36.93011583511385,
"request_tps": 36.47766577456727,
"group_elapsed_s": 14.054668250028044,
"group_output_tps": 34.152353613828076,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 1,
"slot": 2,
"request_id": "d03d55d9-a506-44a9-887e-e0ab77392c8d",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:50:33.397083+00:00",
"ttft_s": 3.606067124987021,
"total_s": 6.893670374993235,
"prompt_tokens": 20,
"output_tokens": 96,
"prompt_tps": 3356267.830172848,
"decode_tps": 28.527554720530254,
"request_tps": 13.925818145909567,
"group_elapsed_s": 14.054668250028044,
"group_output_tps": 34.152353613828076,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 1,
"slot": 3,
"request_id": "55be3260-b3a0-422a-b918-a81261da6a84",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:50:33.396391+00:00",
"ttft_s": 6.9750645000021905,
"total_s": 10.400937125028577,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 6745098.039215687,
"decode_tps": 36.57719818551596,
"request_tps": 12.30658338391295,
"group_elapsed_s": 14.054668250028044,
"group_output_tps": 34.152353613828076,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 2,
"slot": 0,
"request_id": "930b484d-3177-4ee9-9d72-5442ee8cd7c9",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:50:47.451672+00:00",
"ttft_s": 0.11444733297685161,
"total_s": 3.540263083006721,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 4248957.431741286,
"decode_tps": 36.58630699945727,
"request_tps": 36.15550511327833,
"group_elapsed_s": 14.210740375041496,
"group_output_tps": 33.918007597024484,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 2,
"slot": 1,
"request_id": "231f3cbb-61f2-4042-8a99-def6a8698a7d",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:50:47.451387+00:00",
"ttft_s": 10.67886250000447,
"total_s": 14.209862457995769,
"prompt_tokens": 20,
"output_tokens": 98,
"prompt_tps": 2840102.243680773,
"decode_tps": 27.132075592063398,
"request_tps": 6.896618478165229,
"group_elapsed_s": 14.210740375041496,
"group_output_tps": 33.918007597024484,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 2,
"slot": 2,
"request_id": "32437705-bf67-438f-bad7-0bb10d0e2030",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:50:47.451856+00:00",
"ttft_s": 3.622358249966055,
"total_s": 7.2201729999505915,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 4777777.777777778,
"decode_tps": 34.87549075162609,
"request_tps": 17.72810706902396,
"group_elapsed_s": 14.210740375041496,
"group_output_tps": 33.918007597024484,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 2,
"slot": 3,
"request_id": "a16a724c-155d-4e2d-ba28-3a8683af4d40",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:50:47.451136+00:00",
"ttft_s": 7.304396207968239,
"total_s": 10.594903749995865,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 17515407.071034707,
"decode_tps": 38.04511611184116,
"request_tps": 12.081280115456448,
"group_elapsed_s": 14.210740375041496,
"group_output_tps": 33.918007597024484,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 3,
"slot": 0,
"request_id": "f6ab47e7-5bdf-4b5c-be3f-0e639727a2bb",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:51:01.661760+00:00",
"ttft_s": 10.42098737496417,
"total_s": 14.485269291966688,
"prompt_tokens": 20,
"output_tokens": 96,
"prompt_tps": 2105263.1578947366,
"decode_tps": 23.198198961620538,
"request_tps": 6.627422525947802,
"group_elapsed_s": 14.485774166998453,
"group_output_tps": 30.926893850149572,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 3,
"slot": 1,
"request_id": "8e015249-8b0c-4560-b93d-603afb968f5b",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:51:01.662046+00:00",
"ttft_s": 0.11227079102536663,
"total_s": 3.6567685829941183,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 3571131.9657835728,
"decode_tps": 35.39152606628793,
"request_tps": 35.00358228717748,
"group_elapsed_s": 14.485774166998453,
"group_output_tps": 30.926893850149572,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 3,
"slot": 2,
"request_id": "d2848890-f070-45de-8a2f-4bb2b8788a87",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:51:01.661862+00:00",
"ttft_s": 7.207546124991495,
"total_s": 10.33818608301226,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 9458749.343142407,
"decode_tps": 39.94848206301452,
"request_tps": 12.381282264819164,
"group_elapsed_s": 14.485774166998453,
"group_output_tps": 30.926893850149572,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 3,
"slot": 3,
"request_id": "aba512eb-9825-4e49-9ab8-0c75678ff654",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:51:01.661952+00:00",
"ttft_s": 3.738764916022774,
"total_s": 7.12777191598434,
"prompt_tokens": 20,
"output_tokens": 96,
"prompt_tps": 2652168.1474605487,
"decode_tps": 27.731643965337334,
"request_tps": 13.468444435590849,
"group_elapsed_s": 14.485774166998453,
"group_output_tps": 30.926893850149572,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 4,
"slot": 0,
"request_id": "4c24ae09-884b-4b94-ae8e-37346f1408f2",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:51:16.148220+00:00",
"ttft_s": 10.924698917020578,
"total_s": 15.198258958000224,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 7068880.486602006,
"decode_tps": 29.434480556585335,
"request_tps": 8.422017308280036,
"group_elapsed_s": 15.199551291007083,
"group_output_tps": 31.579879616840746,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 4,
"slot": 1,
"request_id": "c94ceb91-b83d-4d47-a36d-71741993f366",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T11:51:16.148464+00:00",
"ttft_s": 0.12565479200566188,
"total_s": 3.500943167018704,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 4025044.7227191413,
"decode_tps": 37.03108618648992,
"request_tps": 36.56157609350765,
"group_elapsed_s": 15.199551291007083,
"group_output_tps": 31.579879616840746,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 4,
"slot": 2,
"request_id": "0672967f-d135-44b0-b89f-7995f59b2d8c",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T11:51:16.148378+00:00",
"ttft_s": 7.600669999956153,
"total_s": 10.846751042001415,
"prompt_tokens": 20,
"output_tokens": 96,
"prompt_tps": 2666666.6666666665,
"decode_tps": 28.898123213254255,
"request_tps": 8.850576511645125,
"group_elapsed_s": 15.199551291007083,
"group_output_tps": 31.579879616840746,
"error": null
},
{
"run_id": "20260831T114928Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 4,
"round": 4,
"slot": 3,
"request_id": "8185344b-c228-457a-8c3d-6a7b5f803594",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T11:51:16.147977+00:00",
"ttft_s": 3.5925854169763625,
"total_s": 7.520279875025153,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 4229785.559708833,
"decode_tps": 31.98591420303284,
"request_tps": 17.02064313126004,
"group_elapsed_s": 15.199551291007083,
"group_output_tps": 31.579879616840746,
"error": null
}
]

54
benchmarks/inference/results/20260831T114928Z/summary.json Normal file
View File

@@ -0,0 +1,54 @@
[
{
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"concurrency": 1,
"requests": 5,
"successes": 5,
"failures": 0,
"ttft_s_mean": 0.18287834140937775,
"ttft_s_p50": 0.11936691601295024,
"ttft_s_p95": 0.3934195415815338,
"total_s_mean": 2.769790649821516,
"total_s_p50": 2.7544727920321748,
"total_s_p95": 3.450829833233729,
"prompt_tps_mean": 1201063.1140147406,
"prompt_tps_p50": 816.5668113066018,
"prompt_tps_p95": 3147950.5530253737,
"decode_tps_mean": 44.301349906835995,
"decode_tps_p50": 45.72630609321048,
"decode_tps_p95": 52.87492274071074,
"request_tps_mean": 42.02276635113332,
"request_tps_p50": 44.19526212378514,
"request_tps_p95": 46.37707708697445,
"aggregate_output_tps_mean": 42.01338924160581,
"aggregate_output_tps_p50": 44.18819214792146,
"aggregate_output_tps_p95": 46.368391249450006
},
{
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"concurrency": 4,
"requests": 20,
"successes": 20,
"failures": 0,
"ttft_s_mean": 5.296346539442311,
"ttft_s_p50": 5.078280103509314,
"ttft_s_p95": 10.691154320855276,
"total_s_mean": 8.731022270751419,
"total_s_p50": 8.619212541496381,
"total_s_p95": 14.520918775268365,
"prompt_tps_mean": 4873110.971654413,
"prompt_tps_p50": 3843030.8359358422,
"prompt_tps_p95": 9861582.229537027,
"decode_tps_mean": 33.61516109813652,
"decode_tps_p50": 34.997484039567254,
"decode_tps_p95": 40.1951363429803,
"request_tps_mean": 18.258369295759792,
"request_tps_p50": 12.924863350205007,
"request_tps_p95": 36.94528376333951,
"aggregate_output_tps_mean": 33.27170984729996,
"aggregate_output_tps_p50": 33.918007597024484,
"aggregate_output_tps_p95": 35.45560236969114
}
]

135
benchmarks/inference/results/20260831T121855Z/metadata.json Normal file
View File

File diff suppressed because one or more lines are too long

36
benchmarks/inference/results/20260831T121855Z/raw.csv Normal file
View File

@@ -0,0 +1,36 @@
run_id,runtime,model,phase,concurrency,round,slot,request_id,prompt_name,prompt_chars,max_tokens,started_at,ttft_s,total_s,prompt_tokens,output_tokens,prompt_tps,decode_tps,request_tps,group_elapsed_s,group_output_tps,error
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,warmup,1,0,0,556d36df-74f6-4dba-866f-134c99f385ac,short,106,128,2026-08-31T12:19:01.706610+00:00,8.133006000018213,13.098198042018339,20,95,80.39981477169073,18.60021913626931,7.252906063509266,13.099143458006438,7.252382593148428,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,warmup,1,1,0,456b9efb-0afa-48e4-8093-041758c70c7d,medium,252,128,2026-08-31T12:19:14.805394+00:00,0.6389443330117501,6.043248832982499,43,128,91.99219534791703,23.084069916067204,21.180660194134138,6.043762709014118,21.178859290602404,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,1,0,0,78330ae3-a01a-4bf4-b1b6-e4ae861145a8,short,106,128,2026-08-31T12:19:20.848831+00:00,0.25939199997810647,5.077765540976543,20,95,194.8949698826367,19.20226982827757,18.709016639970706,5.077990833029617,18.708186588694836,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,1,1,0,f8373947-d690-42a4-b613-33f71746b2f9,medium,252,128,2026-08-31T12:19:25.927037+00:00,0.25913220801157877,5.760092166019604,43,128,423.79232394227944,22.737809958094928,22.22186664913243,5.761121416988317,22.217896609947385,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,1,2,0,9e12f0f4-50d6-444c-8752-4565aa55084b,code,267,128,2026-08-31T12:19:31.688098+00:00,0.6637360000167973,5.271547083975747,54,128,115.32507388679936,26.793290081008884,24.281296924974765,5.271936749981251,24.27950221528269,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,1,3,0,583bec1f-0983-4208-8635-439a9a0c5424,short,106,128,2026-08-31T12:19:36.960047+00:00,0.15107908402569592,5.119054667011369,20,95,4102564.1025641025,18.66803886577614,18.558113983858536,5.11973733396735,18.555639440663118,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,1,4,0,641937dd-533c-493e-b7b3-3fa2cc6e8879,medium,252,128,2026-08-31T12:19:42.079662+00:00,0.1495445409673266,5.59909695800161,43,128,3794229.2420365307,22.9734392268289,22.86082933732315,5.599817292008083,22.85788862838049,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,warmup,4,0,0,3dbdfe4c-7d43-4575-a1e6-6b74bd5a10a7,short,106,128,2026-08-31T12:19:47.680109+00:00,0.16311579098692164,4.8760897910106,20,95,1791151.7105498835,19.64848684540538,19.482824162741817,22.031325709016528,20.24390206429885,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,warmup,4,0,1,1ea04fa9-7f60-4936-ba53-6829df8e5714,medium,252,128,2026-08-31T12:19:47.680573+00:00,10.314829125010874,16.516400250024162,43,128,6370370.37037037,20.216323480613685,7.749872736331438,22.031325709016528,20.24390206429885,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,warmup,4,0,2,046681e1-e7be-4c58-8284-88f821e4a8a0,code,267,128,2026-08-31T12:19:47.680452+00:00,5.110771542007569,10.15410141699249,54,128,514.3377603686088,24.759396813702583,12.605743703307615,22.031325709016528,20.24390206429885,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,warmup,4,0,3,f496e50f-a2db-4fad-a2e1-2a7965fd862d,short,106,128,2026-08-31T12:19:47.680317+00:00,16.645131999976,22.03015966597013,20,95,6666666.666666666,17.25091779989875,4.312270153300159,22.031325709016528,20.24390206429885,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,warmup,4,1,0,cb3d0e9e-0a6f-4f36-aec9-4b20cbc8f87d,medium,252,128,2026-08-31T12:20:09.710819+00:00,17.595250790996943,23.510470708017237,43,128,19843101.06137517,21.163606353137514,5.444382700357892,23.51121658302145,20.37325454038428,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,warmup,4,1,1,fa3bb29e-7370-4255-877e-376076d7aa76,code,267,128,2026-08-31T12:20:09.711021+00:00,5.7327396660111845,11.353798499971163,54,128,9000000.0,22.18879858004337,11.273760054868431,23.51121658302145,20.37325454038428,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,warmup,4,1,2,93ec5b28-1a29-426a-96ff-b3b525fb3c71,short,106,128,2026-08-31T12:20:09.710927+00:00,0.1655573329771869,5.57563720800681,20,95,1563477.1732332706,17.170644798585553,17.038411298277563,23.51121658302145,20.37325454038428,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,warmup,4,1,3,dcac6e9a-3580-4bb1-8787-9cef1c64bfd5,medium,252,128,2026-08-31T12:20:09.711149+00:00,11.527663582994137,17.459117082995363,43,128,6331909.880724489,21.03978753266432,7.331413117371669,23.51121658302145,20.37325454038428,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,0,0,d42caf5b-fb6a-4e96-8e42-fd0b8fc1945b,short,106,128,2026-08-31T12:20:33.222727+00:00,6.285906166012865,10.864910541044082,20,95,8421052.631578946,20.156958877874334,8.743744335594945,21.625605207984336,20.623700271534293,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,0,1,87483860-d4c6-4f13-928c-48de78b6ef70,medium,252,128,2026-08-31T12:20:33.222336+00:00,0.17068937502335757,6.147177167003974,43,128,4669345.205776958,20.965937740191077,20.822565630133763,21.625605207984336,20.623700271534293,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,0,2,ee57b9cf-06bc-468b-bd1e-f936abf292df,code,267,128,2026-08-31T12:20:33.222603+00:00,16.19913299998734,21.62435016600648,54,128,11675675.675675677,22.963769927247082,5.919253018812849,21.625605207984336,20.623700271534293,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,0,3,13164aa4-836c-4761-8f99-95bfe18d33a1,short,106,128,2026-08-31T12:20:33.222066+00:00,11.00553745898651,16.043328458967153,20,95,10214504.596527068,18.355407643172423,5.9214645042626,21.625605207984336,20.623700271534293,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,1,0,51389c06-2a1c-4dcb-a847-5f0c914e298c,medium,252,128,2026-08-31T12:20:54.847154+00:00,0.16574266698444262,6.310333041998092,43,128,5402688.779997488,20.423563700405246,20.284190889466956,23.574590582982637,20.318486478648204,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,1,1,1952ea53-9c68-4536-8e9d-089790a8bdee,code,267,128,2026-08-31T12:20:54.847267+00:00,18.0774574590032,23.57407404197147,54,128,6513086.47931492,22.77481444475703,5.4296936444717945,23.574590582982637,20.318486478648204,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,1,2,d71abf03-e17c-42b0-b6bb-c57bf6961bc2,short,106,128,2026-08-31T12:20:54.847373+00:00,12.635340875014663,17.94654095900478,20,95,9229349.330872172,17.481471270830138,5.293499188339869,23.574590582982637,20.318486478648204,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,1,3,4e10d58d-5d6d-46e7-8254-46dd35cfeb9f,medium,252,128,2026-08-31T12:20:54.847461+00:00,6.461538125004154,12.505505541979801,43,128,3739130.434782609,20.730070992395465,10.235491845596812,23.574590582982637,20.318486478648204,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,2,0,63b32437-7995-44f4-b5ca-f0f52da2bda7,code,267,128,2026-08-31T12:21:18.421714+00:00,17.908610125014093,23.736398082983214,54,128,15071169.411108011,21.428265908373906,5.392562070812423,23.736987709009554,20.179477104341757,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,2,1,3b8fa85d-5574-4ac2-b33c-7208ef99a34f,short,106,128,2026-08-31T12:21:18.421631+00:00,6.036650207999628,11.081229583011009,20,95,4571428.571428572,18.37088402458498,8.573055840810985,23.736987709009554,20.179477104341757,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,2,2,76178335-d531-49f7-83d5-84a414d73ce5,medium,252,128,2026-08-31T12:21:18.421520+00:00,11.268355667009018,17.757357084017713,43,128,7533286.615276804,19.182821184166418,7.208279891786645,23.736987709009554,20.179477104341757,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,2,3,728d2a73-194b-41bb-9dda-216960415fcf,code,267,128,2026-08-31T12:21:18.421818+00:00,0.17636983300326392,5.888466249976773,54,128,7200000.0,21.89716979543998,21.737409125934093,23.736987709009554,20.179477104341757,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,3,0,dfcbb66f-3170-41e1-a4e1-a9b7449517ce,short,106,128,2026-08-31T12:21:42.158434+00:00,6.034018917009234,11.539304499980062,20,95,3200000.0,16.88203353475361,8.232731877399036,23.861715458042454,18.69102834556173,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,3,1,04f9942b-bc73-4665-9830-40b7a50beb96,medium,252,128,2026-08-31T12:21:42.158640+00:00,11.68165541702183,18.32316145801451,43,128,9921550.530687585,18.896968681699633,6.985694051394885,23.861715458042454,18.69102834556173,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,3,2,e2e252a5-26bf-4277-a744-cf7456cdfac7,code,267,128,2026-08-31T12:21:42.158549+00:00,0.17531787499319762,5.8762462079757825,54,128,5355548.943766736,21.95473700787165,21.782613503543573,23.861715458042454,18.69102834556173,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,3,3,61a1f98f-fb9a-49a8-8f00-04bb52b494cc,short,106,128,2026-08-31T12:21:42.158328+00:00,18.46465341700241,23.86128708301112,20,95,5782017.924255566,17.187946685051212,3.981344328556299,23.861715458042454,18.69102834556173,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,4,0,ae555d62-2572-4a62-8cd6-7ae714aace26,medium,252,128,2026-08-31T12:22:06.019994+00:00,20.83534020796651,28.275580540997908,43,128,7937973.047812442,16.887931275751107,4.526874340012493,28.276262458995916,16.940004029691313,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,4,1,c0147505-dd4a-4ad8-bb9d-0955114a8ee0,code,267,128,2026-08-31T12:22:06.020061+00:00,13.962619374971837,20.66803166596219,54,128,10125632.852053255,18.57306514141657,6.193139340443381,28.276262458995916,16.940004029691313,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,4,2,3a3d625f-5d64-42d0-9145-3056ea9864db,short,106,128,2026-08-31T12:22:06.019916+00:00,7.783273542008828,13.761600458994508,20,95,1927710.843373494,15.539155324201145,6.903266831722942,28.276262458995916,16.940004029691313,
20260831T121855Z,ollama,qwen3.8:27b-mxfp8,measured,4,4,3,8ea78898-3c59-49f2-9734-f01bb668d810,medium,252,128,2026-08-31T12:22:06.019812+00:00,0.17432458401890472,7.616022333968431,43,128,2866666.6666666665,16.90294641795698,16.806673403399,28.276262458995916,16.940004029691313,
1 run_id runtime model phase concurrency round slot request_id prompt_name prompt_chars max_tokens started_at ttft_s total_s prompt_tokens output_tokens prompt_tps decode_tps request_tps group_elapsed_s group_output_tps error
2 20260831T121855Z ollama qwen3.8:27b-mxfp8 warmup 1 0 0 556d36df-74f6-4dba-866f-134c99f385ac short 106 128 2026-08-31T12:19:01.706610+00:00 8.133006000018213 13.098198042018339 20 95 80.39981477169073 18.60021913626931 7.252906063509266 13.099143458006438 7.252382593148428
3 20260831T121855Z ollama qwen3.8:27b-mxfp8 warmup 1 1 0 456b9efb-0afa-48e4-8093-041758c70c7d medium 252 128 2026-08-31T12:19:14.805394+00:00 0.6389443330117501 6.043248832982499 43 128 91.99219534791703 23.084069916067204 21.180660194134138 6.043762709014118 21.178859290602404
4 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 1 0 0 78330ae3-a01a-4bf4-b1b6-e4ae861145a8 short 106 128 2026-08-31T12:19:20.848831+00:00 0.25939199997810647 5.077765540976543 20 95 194.8949698826367 19.20226982827757 18.709016639970706 5.077990833029617 18.708186588694836
5 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 1 1 0 f8373947-d690-42a4-b613-33f71746b2f9 medium 252 128 2026-08-31T12:19:25.927037+00:00 0.25913220801157877 5.760092166019604 43 128 423.79232394227944 22.737809958094928 22.22186664913243 5.761121416988317 22.217896609947385
6 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 1 2 0 9e12f0f4-50d6-444c-8752-4565aa55084b code 267 128 2026-08-31T12:19:31.688098+00:00 0.6637360000167973 5.271547083975747 54 128 115.32507388679936 26.793290081008884 24.281296924974765 5.271936749981251 24.27950221528269
7 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 1 3 0 583bec1f-0983-4208-8635-439a9a0c5424 short 106 128 2026-08-31T12:19:36.960047+00:00 0.15107908402569592 5.119054667011369 20 95 4102564.1025641025 18.66803886577614 18.558113983858536 5.11973733396735 18.555639440663118
8 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 1 4 0 641937dd-533c-493e-b7b3-3fa2cc6e8879 medium 252 128 2026-08-31T12:19:42.079662+00:00 0.1495445409673266 5.59909695800161 43 128 3794229.2420365307 22.9734392268289 22.86082933732315 5.599817292008083 22.85788862838049
9 20260831T121855Z ollama qwen3.8:27b-mxfp8 warmup 4 0 0 3dbdfe4c-7d43-4575-a1e6-6b74bd5a10a7 short 106 128 2026-08-31T12:19:47.680109+00:00 0.16311579098692164 4.8760897910106 20 95 1791151.7105498835 19.64848684540538 19.482824162741817 22.031325709016528 20.24390206429885
10 20260831T121855Z ollama qwen3.8:27b-mxfp8 warmup 4 0 1 1ea04fa9-7f60-4936-ba53-6829df8e5714 medium 252 128 2026-08-31T12:19:47.680573+00:00 10.314829125010874 16.516400250024162 43 128 6370370.37037037 20.216323480613685 7.749872736331438 22.031325709016528 20.24390206429885
11 20260831T121855Z ollama qwen3.8:27b-mxfp8 warmup 4 0 2 046681e1-e7be-4c58-8284-88f821e4a8a0 code 267 128 2026-08-31T12:19:47.680452+00:00 5.110771542007569 10.15410141699249 54 128 514.3377603686088 24.759396813702583 12.605743703307615 22.031325709016528 20.24390206429885
12 20260831T121855Z ollama qwen3.8:27b-mxfp8 warmup 4 0 3 f496e50f-a2db-4fad-a2e1-2a7965fd862d short 106 128 2026-08-31T12:19:47.680317+00:00 16.645131999976 22.03015966597013 20 95 6666666.666666666 17.25091779989875 4.312270153300159 22.031325709016528 20.24390206429885
13 20260831T121855Z ollama qwen3.8:27b-mxfp8 warmup 4 1 0 cb3d0e9e-0a6f-4f36-aec9-4b20cbc8f87d medium 252 128 2026-08-31T12:20:09.710819+00:00 17.595250790996943 23.510470708017237 43 128 19843101.06137517 21.163606353137514 5.444382700357892 23.51121658302145 20.37325454038428
14 20260831T121855Z ollama qwen3.8:27b-mxfp8 warmup 4 1 1 fa3bb29e-7370-4255-877e-376076d7aa76 code 267 128 2026-08-31T12:20:09.711021+00:00 5.7327396660111845 11.353798499971163 54 128 9000000.0 22.18879858004337 11.273760054868431 23.51121658302145 20.37325454038428
15 20260831T121855Z ollama qwen3.8:27b-mxfp8 warmup 4 1 2 93ec5b28-1a29-426a-96ff-b3b525fb3c71 short 106 128 2026-08-31T12:20:09.710927+00:00 0.1655573329771869 5.57563720800681 20 95 1563477.1732332706 17.170644798585553 17.038411298277563 23.51121658302145 20.37325454038428
16 20260831T121855Z ollama qwen3.8:27b-mxfp8 warmup 4 1 3 dcac6e9a-3580-4bb1-8787-9cef1c64bfd5 medium 252 128 2026-08-31T12:20:09.711149+00:00 11.527663582994137 17.459117082995363 43 128 6331909.880724489 21.03978753266432 7.331413117371669 23.51121658302145 20.37325454038428
17 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 0 0 d42caf5b-fb6a-4e96-8e42-fd0b8fc1945b short 106 128 2026-08-31T12:20:33.222727+00:00 6.285906166012865 10.864910541044082 20 95 8421052.631578946 20.156958877874334 8.743744335594945 21.625605207984336 20.623700271534293
18 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 0 1 87483860-d4c6-4f13-928c-48de78b6ef70 medium 252 128 2026-08-31T12:20:33.222336+00:00 0.17068937502335757 6.147177167003974 43 128 4669345.205776958 20.965937740191077 20.822565630133763 21.625605207984336 20.623700271534293
19 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 0 2 ee57b9cf-06bc-468b-bd1e-f936abf292df code 267 128 2026-08-31T12:20:33.222603+00:00 16.19913299998734 21.62435016600648 54 128 11675675.675675677 22.963769927247082 5.919253018812849 21.625605207984336 20.623700271534293
20 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 0 3 13164aa4-836c-4761-8f99-95bfe18d33a1 short 106 128 2026-08-31T12:20:33.222066+00:00 11.00553745898651 16.043328458967153 20 95 10214504.596527068 18.355407643172423 5.9214645042626 21.625605207984336 20.623700271534293
21 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 1 0 51389c06-2a1c-4dcb-a847-5f0c914e298c medium 252 128 2026-08-31T12:20:54.847154+00:00 0.16574266698444262 6.310333041998092 43 128 5402688.779997488 20.423563700405246 20.284190889466956 23.574590582982637 20.318486478648204
22 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 1 1 1952ea53-9c68-4536-8e9d-089790a8bdee code 267 128 2026-08-31T12:20:54.847267+00:00 18.0774574590032 23.57407404197147 54 128 6513086.47931492 22.77481444475703 5.4296936444717945 23.574590582982637 20.318486478648204
23 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 1 2 d71abf03-e17c-42b0-b6bb-c57bf6961bc2 short 106 128 2026-08-31T12:20:54.847373+00:00 12.635340875014663 17.94654095900478 20 95 9229349.330872172 17.481471270830138 5.293499188339869 23.574590582982637 20.318486478648204
24 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 1 3 4e10d58d-5d6d-46e7-8254-46dd35cfeb9f medium 252 128 2026-08-31T12:20:54.847461+00:00 6.461538125004154 12.505505541979801 43 128 3739130.434782609 20.730070992395465 10.235491845596812 23.574590582982637 20.318486478648204
25 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 2 0 63b32437-7995-44f4-b5ca-f0f52da2bda7 code 267 128 2026-08-31T12:21:18.421714+00:00 17.908610125014093 23.736398082983214 54 128 15071169.411108011 21.428265908373906 5.392562070812423 23.736987709009554 20.179477104341757
26 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 2 1 3b8fa85d-5574-4ac2-b33c-7208ef99a34f short 106 128 2026-08-31T12:21:18.421631+00:00 6.036650207999628 11.081229583011009 20 95 4571428.571428572 18.37088402458498 8.573055840810985 23.736987709009554 20.179477104341757
27 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 2 2 76178335-d531-49f7-83d5-84a414d73ce5 medium 252 128 2026-08-31T12:21:18.421520+00:00 11.268355667009018 17.757357084017713 43 128 7533286.615276804 19.182821184166418 7.208279891786645 23.736987709009554 20.179477104341757
28 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 2 3 728d2a73-194b-41bb-9dda-216960415fcf code 267 128 2026-08-31T12:21:18.421818+00:00 0.17636983300326392 5.888466249976773 54 128 7200000.0 21.89716979543998 21.737409125934093 23.736987709009554 20.179477104341757
29 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 3 0 dfcbb66f-3170-41e1-a4e1-a9b7449517ce short 106 128 2026-08-31T12:21:42.158434+00:00 6.034018917009234 11.539304499980062 20 95 3200000.0 16.88203353475361 8.232731877399036 23.861715458042454 18.69102834556173
30 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 3 1 04f9942b-bc73-4665-9830-40b7a50beb96 medium 252 128 2026-08-31T12:21:42.158640+00:00 11.68165541702183 18.32316145801451 43 128 9921550.530687585 18.896968681699633 6.985694051394885 23.861715458042454 18.69102834556173
31 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 3 2 e2e252a5-26bf-4277-a744-cf7456cdfac7 code 267 128 2026-08-31T12:21:42.158549+00:00 0.17531787499319762 5.8762462079757825 54 128 5355548.943766736 21.95473700787165 21.782613503543573 23.861715458042454 18.69102834556173
32 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 3 3 61a1f98f-fb9a-49a8-8f00-04bb52b494cc short 106 128 2026-08-31T12:21:42.158328+00:00 18.46465341700241 23.86128708301112 20 95 5782017.924255566 17.187946685051212 3.981344328556299 23.861715458042454 18.69102834556173
33 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 4 0 ae555d62-2572-4a62-8cd6-7ae714aace26 medium 252 128 2026-08-31T12:22:06.019994+00:00 20.83534020796651 28.275580540997908 43 128 7937973.047812442 16.887931275751107 4.526874340012493 28.276262458995916 16.940004029691313
34 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 4 1 c0147505-dd4a-4ad8-bb9d-0955114a8ee0 code 267 128 2026-08-31T12:22:06.020061+00:00 13.962619374971837 20.66803166596219 54 128 10125632.852053255 18.57306514141657 6.193139340443381 28.276262458995916 16.940004029691313
35 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 4 2 3a3d625f-5d64-42d0-9145-3056ea9864db short 106 128 2026-08-31T12:22:06.019916+00:00 7.783273542008828 13.761600458994508 20 95 1927710.843373494 15.539155324201145 6.903266831722942 28.276262458995916 16.940004029691313
36 20260831T121855Z ollama qwen3.8:27b-mxfp8 measured 4 4 3 8ea78898-3c59-49f2-9734-f01bb668d810 medium 252 128 2026-08-31T12:22:06.019812+00:00 0.17432458401890472 7.616022333968431 43 128 2866666.6666666665 16.90294641795698 16.806673403399 28.276262458995916 16.940004029691313

842
benchmarks/inference/results/20260831T121855Z/raw.json Normal file
View File

@@ -0,0 +1,842 @@
[
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "warmup",
"concurrency": 1,
"round": 0,
"slot": 0,
"request_id": "556d36df-74f6-4dba-866f-134c99f385ac",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T12:19:01.706610+00:00",
"ttft_s": 8.133006000018213,
"total_s": 13.098198042018339,
"prompt_tokens": 20,
"output_tokens": 95,
"prompt_tps": 80.39981477169073,
"decode_tps": 18.60021913626931,
"request_tps": 7.252906063509266,
"group_elapsed_s": 13.099143458006438,
"group_output_tps": 7.252382593148428,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "warmup",
"concurrency": 1,
"round": 1,
"slot": 0,
"request_id": "456b9efb-0afa-48e4-8093-041758c70c7d",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T12:19:14.805394+00:00",
"ttft_s": 0.6389443330117501,
"total_s": 6.043248832982499,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 91.99219534791703,
"decode_tps": 23.084069916067204,
"request_tps": 21.180660194134138,
"group_elapsed_s": 6.043762709014118,
"group_output_tps": 21.178859290602404,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 1,
"round": 0,
"slot": 0,
"request_id": "78330ae3-a01a-4bf4-b1b6-e4ae861145a8",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T12:19:20.848831+00:00",
"ttft_s": 0.25939199997810647,
"total_s": 5.077765540976543,
"prompt_tokens": 20,
"output_tokens": 95,
"prompt_tps": 194.8949698826367,
"decode_tps": 19.20226982827757,
"request_tps": 18.709016639970706,
"group_elapsed_s": 5.077990833029617,
"group_output_tps": 18.708186588694836,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 1,
"round": 1,
"slot": 0,
"request_id": "f8373947-d690-42a4-b613-33f71746b2f9",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T12:19:25.927037+00:00",
"ttft_s": 0.25913220801157877,
"total_s": 5.760092166019604,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 423.79232394227944,
"decode_tps": 22.737809958094928,
"request_tps": 22.22186664913243,
"group_elapsed_s": 5.761121416988317,
"group_output_tps": 22.217896609947385,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 1,
"round": 2,
"slot": 0,
"request_id": "9e12f0f4-50d6-444c-8752-4565aa55084b",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T12:19:31.688098+00:00",
"ttft_s": 0.6637360000167973,
"total_s": 5.271547083975747,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 115.32507388679936,
"decode_tps": 26.793290081008884,
"request_tps": 24.281296924974765,
"group_elapsed_s": 5.271936749981251,
"group_output_tps": 24.27950221528269,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 1,
"round": 3,
"slot": 0,
"request_id": "583bec1f-0983-4208-8635-439a9a0c5424",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T12:19:36.960047+00:00",
"ttft_s": 0.15107908402569592,
"total_s": 5.119054667011369,
"prompt_tokens": 20,
"output_tokens": 95,
"prompt_tps": 4102564.1025641025,
"decode_tps": 18.66803886577614,
"request_tps": 18.558113983858536,
"group_elapsed_s": 5.11973733396735,
"group_output_tps": 18.555639440663118,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 1,
"round": 4,
"slot": 0,
"request_id": "641937dd-533c-493e-b7b3-3fa2cc6e8879",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T12:19:42.079662+00:00",
"ttft_s": 0.1495445409673266,
"total_s": 5.59909695800161,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 3794229.2420365307,
"decode_tps": 22.9734392268289,
"request_tps": 22.86082933732315,
"group_elapsed_s": 5.599817292008083,
"group_output_tps": 22.85788862838049,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "warmup",
"concurrency": 4,
"round": 0,
"slot": 0,
"request_id": "3dbdfe4c-7d43-4575-a1e6-6b74bd5a10a7",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T12:19:47.680109+00:00",
"ttft_s": 0.16311579098692164,
"total_s": 4.8760897910106,
"prompt_tokens": 20,
"output_tokens": 95,
"prompt_tps": 1791151.7105498835,
"decode_tps": 19.64848684540538,
"request_tps": 19.482824162741817,
"group_elapsed_s": 22.031325709016528,
"group_output_tps": 20.24390206429885,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "warmup",
"concurrency": 4,
"round": 0,
"slot": 1,
"request_id": "1ea04fa9-7f60-4936-ba53-6829df8e5714",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T12:19:47.680573+00:00",
"ttft_s": 10.314829125010874,
"total_s": 16.516400250024162,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 6370370.37037037,
"decode_tps": 20.216323480613685,
"request_tps": 7.749872736331438,
"group_elapsed_s": 22.031325709016528,
"group_output_tps": 20.24390206429885,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "warmup",
"concurrency": 4,
"round": 0,
"slot": 2,
"request_id": "046681e1-e7be-4c58-8284-88f821e4a8a0",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T12:19:47.680452+00:00",
"ttft_s": 5.110771542007569,
"total_s": 10.15410141699249,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 514.3377603686088,
"decode_tps": 24.759396813702583,
"request_tps": 12.605743703307615,
"group_elapsed_s": 22.031325709016528,
"group_output_tps": 20.24390206429885,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "warmup",
"concurrency": 4,
"round": 0,
"slot": 3,
"request_id": "f496e50f-a2db-4fad-a2e1-2a7965fd862d",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T12:19:47.680317+00:00",
"ttft_s": 16.645131999976,
"total_s": 22.03015966597013,
"prompt_tokens": 20,
"output_tokens": 95,
"prompt_tps": 6666666.666666666,
"decode_tps": 17.25091779989875,
"request_tps": 4.312270153300159,
"group_elapsed_s": 22.031325709016528,
"group_output_tps": 20.24390206429885,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "warmup",
"concurrency": 4,
"round": 1,
"slot": 0,
"request_id": "cb3d0e9e-0a6f-4f36-aec9-4b20cbc8f87d",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T12:20:09.710819+00:00",
"ttft_s": 17.595250790996943,
"total_s": 23.510470708017237,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 19843101.06137517,
"decode_tps": 21.163606353137514,
"request_tps": 5.444382700357892,
"group_elapsed_s": 23.51121658302145,
"group_output_tps": 20.37325454038428,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "warmup",
"concurrency": 4,
"round": 1,
"slot": 1,
"request_id": "fa3bb29e-7370-4255-877e-376076d7aa76",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T12:20:09.711021+00:00",
"ttft_s": 5.7327396660111845,
"total_s": 11.353798499971163,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 9000000.0,
"decode_tps": 22.18879858004337,
"request_tps": 11.273760054868431,
"group_elapsed_s": 23.51121658302145,
"group_output_tps": 20.37325454038428,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "warmup",
"concurrency": 4,
"round": 1,
"slot": 2,
"request_id": "93ec5b28-1a29-426a-96ff-b3b525fb3c71",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T12:20:09.710927+00:00",
"ttft_s": 0.1655573329771869,
"total_s": 5.57563720800681,
"prompt_tokens": 20,
"output_tokens": 95,
"prompt_tps": 1563477.1732332706,
"decode_tps": 17.170644798585553,
"request_tps": 17.038411298277563,
"group_elapsed_s": 23.51121658302145,
"group_output_tps": 20.37325454038428,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "warmup",
"concurrency": 4,
"round": 1,
"slot": 3,
"request_id": "dcac6e9a-3580-4bb1-8787-9cef1c64bfd5",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T12:20:09.711149+00:00",
"ttft_s": 11.527663582994137,
"total_s": 17.459117082995363,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 6331909.880724489,
"decode_tps": 21.03978753266432,
"request_tps": 7.331413117371669,
"group_elapsed_s": 23.51121658302145,
"group_output_tps": 20.37325454038428,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 0,
"slot": 0,
"request_id": "d42caf5b-fb6a-4e96-8e42-fd0b8fc1945b",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T12:20:33.222727+00:00",
"ttft_s": 6.285906166012865,
"total_s": 10.864910541044082,
"prompt_tokens": 20,
"output_tokens": 95,
"prompt_tps": 8421052.631578946,
"decode_tps": 20.156958877874334,
"request_tps": 8.743744335594945,
"group_elapsed_s": 21.625605207984336,
"group_output_tps": 20.623700271534293,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 0,
"slot": 1,
"request_id": "87483860-d4c6-4f13-928c-48de78b6ef70",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T12:20:33.222336+00:00",
"ttft_s": 0.17068937502335757,
"total_s": 6.147177167003974,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 4669345.205776958,
"decode_tps": 20.965937740191077,
"request_tps": 20.822565630133763,
"group_elapsed_s": 21.625605207984336,
"group_output_tps": 20.623700271534293,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 0,
"slot": 2,
"request_id": "ee57b9cf-06bc-468b-bd1e-f936abf292df",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T12:20:33.222603+00:00",
"ttft_s": 16.19913299998734,
"total_s": 21.62435016600648,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 11675675.675675677,
"decode_tps": 22.963769927247082,
"request_tps": 5.919253018812849,
"group_elapsed_s": 21.625605207984336,
"group_output_tps": 20.623700271534293,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 0,
"slot": 3,
"request_id": "13164aa4-836c-4761-8f99-95bfe18d33a1",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T12:20:33.222066+00:00",
"ttft_s": 11.00553745898651,
"total_s": 16.043328458967153,
"prompt_tokens": 20,
"output_tokens": 95,
"prompt_tps": 10214504.596527068,
"decode_tps": 18.355407643172423,
"request_tps": 5.9214645042626,
"group_elapsed_s": 21.625605207984336,
"group_output_tps": 20.623700271534293,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 1,
"slot": 0,
"request_id": "51389c06-2a1c-4dcb-a847-5f0c914e298c",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T12:20:54.847154+00:00",
"ttft_s": 0.16574266698444262,
"total_s": 6.310333041998092,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 5402688.779997488,
"decode_tps": 20.423563700405246,
"request_tps": 20.284190889466956,
"group_elapsed_s": 23.574590582982637,
"group_output_tps": 20.318486478648204,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 1,
"slot": 1,
"request_id": "1952ea53-9c68-4536-8e9d-089790a8bdee",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T12:20:54.847267+00:00",
"ttft_s": 18.0774574590032,
"total_s": 23.57407404197147,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 6513086.47931492,
"decode_tps": 22.77481444475703,
"request_tps": 5.4296936444717945,
"group_elapsed_s": 23.574590582982637,
"group_output_tps": 20.318486478648204,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 1,
"slot": 2,
"request_id": "d71abf03-e17c-42b0-b6bb-c57bf6961bc2",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T12:20:54.847373+00:00",
"ttft_s": 12.635340875014663,
"total_s": 17.94654095900478,
"prompt_tokens": 20,
"output_tokens": 95,
"prompt_tps": 9229349.330872172,
"decode_tps": 17.481471270830138,
"request_tps": 5.293499188339869,
"group_elapsed_s": 23.574590582982637,
"group_output_tps": 20.318486478648204,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 1,
"slot": 3,
"request_id": "4e10d58d-5d6d-46e7-8254-46dd35cfeb9f",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T12:20:54.847461+00:00",
"ttft_s": 6.461538125004154,
"total_s": 12.505505541979801,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 3739130.434782609,
"decode_tps": 20.730070992395465,
"request_tps": 10.235491845596812,
"group_elapsed_s": 23.574590582982637,
"group_output_tps": 20.318486478648204,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 2,
"slot": 0,
"request_id": "63b32437-7995-44f4-b5ca-f0f52da2bda7",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T12:21:18.421714+00:00",
"ttft_s": 17.908610125014093,
"total_s": 23.736398082983214,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 15071169.411108011,
"decode_tps": 21.428265908373906,
"request_tps": 5.392562070812423,
"group_elapsed_s": 23.736987709009554,
"group_output_tps": 20.179477104341757,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 2,
"slot": 1,
"request_id": "3b8fa85d-5574-4ac2-b33c-7208ef99a34f",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T12:21:18.421631+00:00",
"ttft_s": 6.036650207999628,
"total_s": 11.081229583011009,
"prompt_tokens": 20,
"output_tokens": 95,
"prompt_tps": 4571428.571428572,
"decode_tps": 18.37088402458498,
"request_tps": 8.573055840810985,
"group_elapsed_s": 23.736987709009554,
"group_output_tps": 20.179477104341757,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 2,
"slot": 2,
"request_id": "76178335-d531-49f7-83d5-84a414d73ce5",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T12:21:18.421520+00:00",
"ttft_s": 11.268355667009018,
"total_s": 17.757357084017713,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 7533286.615276804,
"decode_tps": 19.182821184166418,
"request_tps": 7.208279891786645,
"group_elapsed_s": 23.736987709009554,
"group_output_tps": 20.179477104341757,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 2,
"slot": 3,
"request_id": "728d2a73-194b-41bb-9dda-216960415fcf",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T12:21:18.421818+00:00",
"ttft_s": 0.17636983300326392,
"total_s": 5.888466249976773,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 7200000.0,
"decode_tps": 21.89716979543998,
"request_tps": 21.737409125934093,
"group_elapsed_s": 23.736987709009554,
"group_output_tps": 20.179477104341757,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 3,
"slot": 0,
"request_id": "dfcbb66f-3170-41e1-a4e1-a9b7449517ce",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T12:21:42.158434+00:00",
"ttft_s": 6.034018917009234,
"total_s": 11.539304499980062,
"prompt_tokens": 20,
"output_tokens": 95,
"prompt_tps": 3200000.0,
"decode_tps": 16.88203353475361,
"request_tps": 8.232731877399036,
"group_elapsed_s": 23.861715458042454,
"group_output_tps": 18.69102834556173,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 3,
"slot": 1,
"request_id": "04f9942b-bc73-4665-9830-40b7a50beb96",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T12:21:42.158640+00:00",
"ttft_s": 11.68165541702183,
"total_s": 18.32316145801451,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 9921550.530687585,
"decode_tps": 18.896968681699633,
"request_tps": 6.985694051394885,
"group_elapsed_s": 23.861715458042454,
"group_output_tps": 18.69102834556173,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 3,
"slot": 2,
"request_id": "e2e252a5-26bf-4277-a744-cf7456cdfac7",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T12:21:42.158549+00:00",
"ttft_s": 0.17531787499319762,
"total_s": 5.8762462079757825,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 5355548.943766736,
"decode_tps": 21.95473700787165,
"request_tps": 21.782613503543573,
"group_elapsed_s": 23.861715458042454,
"group_output_tps": 18.69102834556173,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 3,
"slot": 3,
"request_id": "61a1f98f-fb9a-49a8-8f00-04bb52b494cc",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T12:21:42.158328+00:00",
"ttft_s": 18.46465341700241,
"total_s": 23.86128708301112,
"prompt_tokens": 20,
"output_tokens": 95,
"prompt_tps": 5782017.924255566,
"decode_tps": 17.187946685051212,
"request_tps": 3.981344328556299,
"group_elapsed_s": 23.861715458042454,
"group_output_tps": 18.69102834556173,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 4,
"slot": 0,
"request_id": "ae555d62-2572-4a62-8cd6-7ae714aace26",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T12:22:06.019994+00:00",
"ttft_s": 20.83534020796651,
"total_s": 28.275580540997908,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 7937973.047812442,
"decode_tps": 16.887931275751107,
"request_tps": 4.526874340012493,
"group_elapsed_s": 28.276262458995916,
"group_output_tps": 16.940004029691313,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 4,
"slot": 1,
"request_id": "c0147505-dd4a-4ad8-bb9d-0955114a8ee0",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T12:22:06.020061+00:00",
"ttft_s": 13.962619374971837,
"total_s": 20.66803166596219,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 10125632.852053255,
"decode_tps": 18.57306514141657,
"request_tps": 6.193139340443381,
"group_elapsed_s": 28.276262458995916,
"group_output_tps": 16.940004029691313,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 4,
"slot": 2,
"request_id": "3a3d625f-5d64-42d0-9145-3056ea9864db",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T12:22:06.019916+00:00",
"ttft_s": 7.783273542008828,
"total_s": 13.761600458994508,
"prompt_tokens": 20,
"output_tokens": 95,
"prompt_tps": 1927710.843373494,
"decode_tps": 15.539155324201145,
"request_tps": 6.903266831722942,
"group_elapsed_s": 28.276262458995916,
"group_output_tps": 16.940004029691313,
"error": null
},
{
"run_id": "20260831T121855Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 4,
"slot": 3,
"request_id": "8ea78898-3c59-49f2-9734-f01bb668d810",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T12:22:06.019812+00:00",
"ttft_s": 0.17432458401890472,
"total_s": 7.616022333968431,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 2866666.6666666665,
"decode_tps": 16.90294641795698,
"request_tps": 16.806673403399,
"group_elapsed_s": 28.276262458995916,
"group_output_tps": 16.940004029691313,
"error": null
}
]

54
benchmarks/inference/results/20260831T121855Z/summary.json Normal file
View File

@@ -0,0 +1,54 @@
[
{
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"concurrency": 1,
"requests": 5,
"successes": 5,
"failures": 0,
"ttft_s_mean": 0.296576766599901,
"ttft_s_p50": 0.25913220801157877,
"ttft_s_p95": 0.5828672000090591,
"total_s_mean": 5.365511283196975,
"total_s_p50": 5.271547083975747,
"total_s_p95": 5.7278931244160045,
"prompt_tps_mean": 1579505.471393669,
"prompt_tps_p50": 423.79232394227944,
"prompt_tps_p95": 4040897.1304585882,
"decode_tps_mean": 22.074969591997284,
"decode_tps_p50": 22.737809958094928,
"decode_tps_p95": 26.029319910172887,
"request_tps_mean": 21.326224707051917,
"request_tps_p50": 22.22186664913243,
"request_tps_p95": 23.99720340744444,
"aggregate_output_tps_mean": 21.3238226965937,
"aggregate_output_tps_p50": 22.217896609947385,
"aggregate_output_tps_p95": 23.99517949790225
},
{
"runtime": "ollama",
"model": "qwen3.8:27b-mxfp8",
"concurrency": 4,
"requests": 20,
"successes": 20,
"failures": 0,
"ttft_s_mean": 9.275126714701765,
"ttft_s_p50": 9.394405500497669,
"ttft_s_p95": 18.583187756550615,
"total_s_mean": 15.170045258343453,
"total_s_p50": 14.90246445898083,
"total_s_p95": 24.08200175591046,
"prompt_tps_mean": 7067890.927047749,
"prompt_tps_p50": 6856543.23965746,
"prompt_tps_p95": 11845450.362447295,
"decode_tps_mean": 19.377795978907,
"decode_tps_p50": 19.039894932933024,
"decode_tps_p95": 22.784262218881533,
"request_tps_mean": 10.048677383124767,
"request_tps_p50": 7.0969869715907645,
"request_tps_p95": 21.739669344814565,
"aggregate_output_tps_mean": 19.35053924595546,
"aggregate_output_tps_p50": 20.179477104341757,
"aggregate_output_tps_p95": 20.562657512957074
}
]

136
benchmarks/inference/results/20260831T132015Z/metadata.json Normal file
View File

@@ -0,0 +1,136 @@
{
"run_id": "20260831T132015Z",
"created_at": "2026-08-31T13:20:15.702115+00:00",
"host": {
"platform": "macOS-15.6-arm64-arm-64bit",
"machine": "arm64",
"python": "3.12.8"
},
"arguments": {
"output_dir": "/Users/christianmeyer/Documents/git/lai_automatisierung/benchmarks/inference/results",
"runtime": "vllm",
"ollama_url": "http://127.0.0.1:11434",
"ollama_model": "qwen3.8:27b-mlx",
"vllm_url": "http://127.0.0.1:8001",
"vllm_model": "qwen3.8-27b-mxfp8",
"warmups": 2,
"rounds": 5,
"concurrency": [
1,
4
],
"max_tokens": 128,
"seed": 42,
"timeout": 600
},
"runtimes": [
{
"name": "vllm-metal",
"kind": "openai",
"base_url": "http://127.0.0.1:8001",
"model": "qwen3.8-27b-mxfp8"
}
],
"versions": {
"ollama": "ollama version is 0.32.14",
"vllm": "INFO 08-31 15:20:16 [__init__.py:52] Available plugins for group vllm.platform_plugins:\nINFO 08-31 15:20:16 [__init__.py:54] - metal -> vllm_metal:register\nINFO 08-31 15:20:16 [__init__.py:57] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.\nINFO 08-31 15:20:18 [__init__.py:254] Platform plugin metal is activated\nINFO 08-31 15:20:19 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.\n0.28.0+cpu"
},
"model_metadata": {
"vllm-metal": {
"object": "list",
"data": [
{
"id": "qwen3.8-27b-mxfp8",
"object": "model",
"created": 1788182421,
"owned_by": "vllm",
"root": "mlx-community/Qwen3.8-27B-mxfp8",
"parent": null,
"max_model_len": 8192,
"permission": [
{
"id": "modelperm-a3c90f9b6d588379",
"object": "model_permission",
"created": 1788182421,
"allow_create_engine": false,
"allow_sampling": true,
"allow_logprobs": true,
"allow_search_indices": false,
"allow_view": true,
"allow_fine_tuning": false,
"organization": "*",
"group": null,
"is_blocking": false
}
]
}
]
}
},
"prompts": {
"short": "In exactly four concise bullet points, explain why database transactions are useful. Do not use a heading.",
"medium": "You are reviewing a Laravel service that transfers money between two accounts. Explain the failure modes involving concurrency, partial writes, retries, and audit logs. Then provide a robust implementation plan. Keep the response technical and concise.",
"code": "Write a Python 3.12 function named merge_intervals that accepts a list of integer pairs, validates each pair, merges overlapping or adjacent intervals, and returns sorted tuples. Include type hints, a short docstring, and five assert-based examples. Output only code."
},
"measurement_notes": {
"ttft": "Client wall time from request start until first non-empty generated text chunk.",
"total": "Client wall time from request start until stream completion.",
"decode_tps": "Ollama server eval metric; otherwise output tokens / (total - TTFT).",
"aggregate_output_tps": "Sum of successful output tokens / concurrent group wall time."
},
"completed_at": "2026-08-31T13:22:48.218681+00:00",
"summary": [
{
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"concurrency": 1,
"requests": 5,
"successes": 5,
"failures": 0,
"ttft_s_mean": 0.37155269999057056,
"ttft_s_p50": 0.4171057919738814,
"ttft_s_p95": 0.4531440335791558,
"total_s_mean": 7.930883799993898,
"total_s_p50": 8.838838333962485,
"total_s_p95": 8.998963149765041,
"prompt_tps_mean": null,
"prompt_tps_p50": null,
"prompt_tps_p95": null,
"decode_tps_mean": 15.079897408512505,
"decode_tps_p50": 15.126147490283795,
"decode_tps_p95": 15.196162811046243,
"request_tps_mean": 14.380896425252896,
"request_tps_p50": 14.386982882152838,
"request_tps_p95": 14.53502856074605,
"aggregate_output_tps_mean": 14.380548747737752,
"aggregate_output_tps_p50": 14.386712701914696,
"aggregate_output_tps_p95": 14.534638049200744
},
{
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"concurrency": 4,
"requests": 20,
"successes": 20,
"failures": 0,
"ttft_s_mean": 1.255672066603438,
"ttft_s_p50": 1.3144360205042176,
"ttft_s_p95": 1.419064954231726,
"total_s_mean": 12.251293779097614,
"total_s_p50": 12.702340687508695,
"total_s_p95": 13.641910566360457,
"prompt_tps_mean": null,
"prompt_tps_p50": null,
"prompt_tps_p95": null,
"decode_tps_mean": 10.500220268362742,
"decode_tps_p50": 10.485677148644122,
"decode_tps_p95": 11.071162546140778,
"request_tps_mean": 9.417789343799708,
"request_tps_p50": 9.430282582501674,
"request_tps_p95": 10.077752113288362,
"aggregate_output_tps_mean": 35.09087333775396,
"aggregate_output_tps_p50": 34.96408475660248,
"aggregate_output_tps_p95": 35.52678209465127
}
]
}

36
benchmarks/inference/results/20260831T132015Z/raw.csv Normal file
View File

@@ -0,0 +1,36 @@
run_id,runtime,model,phase,concurrency,round,slot,request_id,prompt_name,prompt_chars,max_tokens,started_at,ttft_s,total_s,prompt_tokens,output_tokens,prompt_tps,decode_tps,request_tps,group_elapsed_s,group_output_tps,error
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,warmup,1,0,0,4c059f2d-caa2-44ae-ab1d-539d608c86d8,short,106,128,2026-08-31T13:20:21.117908+00:00,0.3135264579905197,6.390581874991767,20,93,,15.303464197450301,14.552665440988472,6.39150691695977,14.550559235604652,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,warmup,1,1,0,1bf28507-8f2f-4127-ba2f-2f5637e722bf,medium,252,128,2026-08-31T13:20:27.508796+00:00,0.3822946660220623,8.7317384579801,43,128,,15.330362499509992,14.659165596401756,8.731932666967623,14.658839558419446,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,1,0,0,81b82bc5-db99-4c10-9ec1-1f684485db59,short,106,128,2026-08-31T13:20:36.240817+00:00,0.2682818329776637,6.39245533302892,20,93,,15.185722612075187,14.548400443172758,6.392639332974795,14.547981695179217,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,1,1,0,1de32693-20bd-450c-8341-17773d6c57b2,medium,252,128,2026-08-31T13:20:42.633525+00:00,0.4171057919738814,8.838838333962485,43,128,,15.198772860789006,14.481541031039212,8.839007750037126,14.481263465286855,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,1,2,0,e821a6d4-8b00-477b-a66f-3579d33c583b,code,267,128,2026-08-31T13:20:51.472629+00:00,0.4347635000012815,8.896931416995358,54,128,,15.126147490283795,14.386982882152838,8.897098499990534,14.386712701914696,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,1,3,0,6415410d-9d1c-4521-b2e2-04c40ce2fd72,short,106,128,2026-08-31T13:21:00.369851+00:00,0.2798732080264017,6.501722833025269,20,93,,14.947323642528074,14.303901041060966,6.501933082996402,14.303438502498576,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,1,4,0,44eda129-4834-4a55-a14a-33a4a510922d,medium,252,128,2026-08-31T13:21:06.871859+00:00,0.4577391669736244,9.024471082957461,43,128,,14.94152043688646,14.183656728838715,9.024667916994076,14.183347373809413,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,warmup,4,0,0,a696baea-c3a4-422a-9d29-c4fff0e6d70e,short,106,128,2026-08-31T13:21:15.897040+00:00,1.1126187500194646,9.980102250003256,20,93,,10.48775562989996,9.318541801510065,12.400747582956683,35.64301241059653,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,warmup,4,0,1,2a75391e-8c1c-4244-8cea-f8f97ce3c40d,medium,252,128,2026-08-31T13:21:15.896746+00:00,1.1128792079980485,12.400287791970186,43,128,,11.340069693388886,10.322341073639151,12.400747582956683,35.64301241059653,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,warmup,4,0,2,21360267-b5fe-4bd2-8b87-d5bbdf4a9c49,code,267,128,2026-08-31T13:21:15.896944+00:00,1.1127408749889582,12.400139749981463,54,128,,11.340079447673899,10.322464309339042,12.400747582956683,35.64301241059653,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,warmup,4,0,3,106f9109-b093-475b-85c2-dc0847a3f3e3,short,106,128,2026-08-31T13:21:15.896863+00:00,1.0121074999915436,9.893255082948599,20,93,,10.471619701317342,9.400343892910335,12.400747582956683,35.64301241059653,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,warmup,4,1,0,fd6e44d9-a975-4760-853a-c85d6547b8f2,medium,252,128,2026-08-31T13:21:28.297798+00:00,1.4682920000050217,13.96469266602071,43,128,,10.242949423676817,9.165973291446168,13.965293916000519,36.232678169433896,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,warmup,4,1,1,a5541448-562a-4c84-aeff-3e2e5b8aba34,code,267,128,2026-08-31T13:21:28.297877+00:00,1.4681764579727314,13.9645455830032,54,128,,10.242975277003744,9.166069833005798,13.965293916000519,36.232678169433896,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,warmup,4,1,2,bb897f87-b4a0-4483-9946-4f6483212386,short,106,128,2026-08-31T13:21:28.297713+00:00,1.3568928329623304,13.431885249970946,20,122,,10.103526013660515,9.082864968658356,13.965293916000519,36.232678169433896,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,warmup,4,1,3,576c3ce6-5566-4a17-b311-3c2f8f3a5a17,medium,252,128,2026-08-31T13:21:28.297622+00:00,1.4684889169875532,13.964902083971538,43,128,,10.242939176993687,9.16583583832745,13.965293916000519,36.232678169433896,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,0,0,75886439-869f-49d6-881d-4bc7cf650b04,short,106,128,2026-08-31T13:21:42.263079+00:00,1.1142227500095032,10.27476887498051,20,93,,10.152233145411333,9.051298489687577,12.701848333003,34.79808516147676,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,0,1,2ecba00e-62dd-4ea7-a1e1-c7b0c12aae0e,medium,252,128,2026-08-31T13:21:42.263219+00:00,1.1140572919975966,12.70125433296198,43,128,,11.046675011003936,10.077744815157159,12.701848333003,34.79808516147676,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,0,2,c1df76bd-2c64-4e34-a5be-10d74b64ce17,code,267,128,2026-08-31T13:21:42.263452+00:00,1.1138802919886075,12.701070374983829,54,128,,11.046681644400257,10.077890777781237,12.701848333003,34.79808516147676,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,0,3,1bf72731-e2e3-4f8d-bf42-627b9fce292a,short,106,128,2026-08-31T13:21:42.263323+00:00,1.0123424580087885,10.185485332971439,20,93,,10.138291888359873,9.130640019572722,12.701848333003,34.79808516147676,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,1,0,a04682c5-d585-4466-8fad-17ed2b9ba80b,medium,252,128,2026-08-31T13:21:54.962444+00:00,1.4147373330197297,13.505381957977079,43,128,,10.586697729563905,9.477703066694506,13.505815375014208,35.31811939932565,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,1,1,bf6552cc-7c18-4617-8260-b6a63ff65b3c,code,267,128,2026-08-31T13:21:54.962650+00:00,1.4145112080150284,13.505133582977578,54,128,,10.586717211933143,9.477877372596776,13.505815375014208,35.31811939932565,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,1,2,36abe2cb-3ddf-4311-853a-4151aef2b328,short,106,128,2026-08-31T13:21:54.962724+00:00,1.4144840000080876,10.538147750019561,20,93,,10.193273508121454,8.825080289828673,13.505815375014208,35.31811939932565,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,1,3,110efa2a-5fb4-48e9-835b-3092cd0df4fb,medium,252,128,2026-08-31T13:21:54.962552+00:00,1.3138721249997616,13.505327666993253,43,128,,10.49915652475652,9.477741166756687,13.505815375014208,35.31811939932565,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,2,0,4d78c4f5-e05f-4ffe-afb6-748b8db712f6,code,267,128,2026-08-31T13:22:08.467061+00:00,1.2189033330068924,13.40631824999582,54,128,,10.502637423262865,9.547736941127733,13.406804582977202,35.57894776848267,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,2,1,9beb5379-ecf9-45d3-8725-13dd100b245d,short,106,128,2026-08-31T13:22:08.467363+00:00,1.3166250829817727,10.428931292030029,20,93,,10.205978362278222,8.917500498931489,13.406804582977202,35.57894776848267,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,2,2,e863870c-69a1-44bd-9d57-7d28f9a30c17,medium,252,128,2026-08-31T13:22:08.467157+00:00,1.3168415420223027,13.406341000052635,43,128,,10.58770054495327,9.547720738976986,13.406804582977202,35.57894776848267,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,2,3,21ffaa25-b50a-4b5f-b562-4a7810b56506,code,267,128,2026-08-31T13:22:08.467234+00:00,1.316743416013196,13.406194041017443,54,128,,10.587743311946818,9.547825401331103,13.406804582977202,35.57894776848267,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,3,0,00728b55-60a4-46d3-8000-146e38d0b3a6,short,106,128,2026-08-31T13:22:21.873446+00:00,1.0375588749884628,10.190524249977898,20,93,,10.160641517788692,9.126125184403708,12.702927250007633,34.795129602882234,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,3,1,531d5ceb-d88b-49ab-ad8d-7bea678c3bb8,medium,252,128,2026-08-31T13:22:21.873549+00:00,1.140847125032451,12.702382000046782,43,128,,11.07119438584415,10.076850152949941,12.702927250007633,34.795129602882234,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,3,2,a4b2f7bb-ab5c-405b-9514-d64bf34153ea,code,267,128,2026-08-31T13:22:21.873705+00:00,1.1407294999808073,12.702299374970607,54,128,,11.071160870366917,10.076915700177802,12.702927250007633,34.795129602882234,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,3,3,1b77ace8-b3ef-47f2-a2cd-06b6f9400a6e,short,106,128,2026-08-31T13:22:21.873633+00:00,1.1407265829620883,10.277835665969178,20,93,,10.178274020276117,9.048597683647658,12.702927250007633,34.795129602882234,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,4,0,1143b997-b26b-4ca6-a4ad-dd1014020849,medium,252,128,2026-08-31T13:22:34.576247+00:00,1.3149999160086736,13.642209500016179,43,128,,10.383534012925246,9.382644358294614,13.642570749972947,34.96408475660248,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,4,1,907e97b8-fec4-47ff-b971-45716a15b6fb,code,267,128,2026-08-31T13:22:34.576465+00:00,1.419054625032004,13.641894833010156,54,128,,10.472197772531725,9.38286078047386,13.642570749972947,34.96408475660248,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,4,2,d5875acf-e267-4393-84c1-a01dac48c4a5,short,106,128,2026-08-31T13:22:34.576349+00:00,1.4192612090264447,10.662482584011741,20,93,,10.061427312742246,8.722171339295066,13.642570749972947,34.96408475660248,
20260831T132015Z,vllm-metal,qwen3.8-27b-mxfp8,measured,4,4,3,d2828df9-67d3-44b0-9809-bdb8c75f5282,medium,252,128,2026-08-31T13:22:34.576542+00:00,1.4190426669665612,13.641892916988581,43,128,,10.47218916878814,9.382862098308841,13.642570749972947,34.96408475660248,
1 run_id runtime model phase concurrency round slot request_id prompt_name prompt_chars max_tokens started_at ttft_s total_s prompt_tokens output_tokens prompt_tps decode_tps request_tps group_elapsed_s group_output_tps error
2 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 warmup 1 0 0 4c059f2d-caa2-44ae-ab1d-539d608c86d8 short 106 128 2026-08-31T13:20:21.117908+00:00 0.3135264579905197 6.390581874991767 20 93 15.303464197450301 14.552665440988472 6.39150691695977 14.550559235604652
3 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 warmup 1 1 0 1bf28507-8f2f-4127-ba2f-2f5637e722bf medium 252 128 2026-08-31T13:20:27.508796+00:00 0.3822946660220623 8.7317384579801 43 128 15.330362499509992 14.659165596401756 8.731932666967623 14.658839558419446
4 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 1 0 0 81b82bc5-db99-4c10-9ec1-1f684485db59 short 106 128 2026-08-31T13:20:36.240817+00:00 0.2682818329776637 6.39245533302892 20 93 15.185722612075187 14.548400443172758 6.392639332974795 14.547981695179217
5 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 1 1 0 1de32693-20bd-450c-8341-17773d6c57b2 medium 252 128 2026-08-31T13:20:42.633525+00:00 0.4171057919738814 8.838838333962485 43 128 15.198772860789006 14.481541031039212 8.839007750037126 14.481263465286855
6 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 1 2 0 e821a6d4-8b00-477b-a66f-3579d33c583b code 267 128 2026-08-31T13:20:51.472629+00:00 0.4347635000012815 8.896931416995358 54 128 15.126147490283795 14.386982882152838 8.897098499990534 14.386712701914696
7 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 1 3 0 6415410d-9d1c-4521-b2e2-04c40ce2fd72 short 106 128 2026-08-31T13:21:00.369851+00:00 0.2798732080264017 6.501722833025269 20 93 14.947323642528074 14.303901041060966 6.501933082996402 14.303438502498576
8 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 1 4 0 44eda129-4834-4a55-a14a-33a4a510922d medium 252 128 2026-08-31T13:21:06.871859+00:00 0.4577391669736244 9.024471082957461 43 128 14.94152043688646 14.183656728838715 9.024667916994076 14.183347373809413
9 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 warmup 4 0 0 a696baea-c3a4-422a-9d29-c4fff0e6d70e short 106 128 2026-08-31T13:21:15.897040+00:00 1.1126187500194646 9.980102250003256 20 93 10.48775562989996 9.318541801510065 12.400747582956683 35.64301241059653
10 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 warmup 4 0 1 2a75391e-8c1c-4244-8cea-f8f97ce3c40d medium 252 128 2026-08-31T13:21:15.896746+00:00 1.1128792079980485 12.400287791970186 43 128 11.340069693388886 10.322341073639151 12.400747582956683 35.64301241059653
11 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 warmup 4 0 2 21360267-b5fe-4bd2-8b87-d5bbdf4a9c49 code 267 128 2026-08-31T13:21:15.896944+00:00 1.1127408749889582 12.400139749981463 54 128 11.340079447673899 10.322464309339042 12.400747582956683 35.64301241059653
12 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 warmup 4 0 3 106f9109-b093-475b-85c2-dc0847a3f3e3 short 106 128 2026-08-31T13:21:15.896863+00:00 1.0121074999915436 9.893255082948599 20 93 10.471619701317342 9.400343892910335 12.400747582956683 35.64301241059653
13 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 warmup 4 1 0 fd6e44d9-a975-4760-853a-c85d6547b8f2 medium 252 128 2026-08-31T13:21:28.297798+00:00 1.4682920000050217 13.96469266602071 43 128 10.242949423676817 9.165973291446168 13.965293916000519 36.232678169433896
14 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 warmup 4 1 1 a5541448-562a-4c84-aeff-3e2e5b8aba34 code 267 128 2026-08-31T13:21:28.297877+00:00 1.4681764579727314 13.9645455830032 54 128 10.242975277003744 9.166069833005798 13.965293916000519 36.232678169433896
15 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 warmup 4 1 2 bb897f87-b4a0-4483-9946-4f6483212386 short 106 128 2026-08-31T13:21:28.297713+00:00 1.3568928329623304 13.431885249970946 20 122 10.103526013660515 9.082864968658356 13.965293916000519 36.232678169433896
16 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 warmup 4 1 3 576c3ce6-5566-4a17-b311-3c2f8f3a5a17 medium 252 128 2026-08-31T13:21:28.297622+00:00 1.4684889169875532 13.964902083971538 43 128 10.242939176993687 9.16583583832745 13.965293916000519 36.232678169433896
17 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 0 0 75886439-869f-49d6-881d-4bc7cf650b04 short 106 128 2026-08-31T13:21:42.263079+00:00 1.1142227500095032 10.27476887498051 20 93 10.152233145411333 9.051298489687577 12.701848333003 34.79808516147676
18 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 0 1 2ecba00e-62dd-4ea7-a1e1-c7b0c12aae0e medium 252 128 2026-08-31T13:21:42.263219+00:00 1.1140572919975966 12.70125433296198 43 128 11.046675011003936 10.077744815157159 12.701848333003 34.79808516147676
19 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 0 2 c1df76bd-2c64-4e34-a5be-10d74b64ce17 code 267 128 2026-08-31T13:21:42.263452+00:00 1.1138802919886075 12.701070374983829 54 128 11.046681644400257 10.077890777781237 12.701848333003 34.79808516147676
20 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 0 3 1bf72731-e2e3-4f8d-bf42-627b9fce292a short 106 128 2026-08-31T13:21:42.263323+00:00 1.0123424580087885 10.185485332971439 20 93 10.138291888359873 9.130640019572722 12.701848333003 34.79808516147676
21 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 1 0 a04682c5-d585-4466-8fad-17ed2b9ba80b medium 252 128 2026-08-31T13:21:54.962444+00:00 1.4147373330197297 13.505381957977079 43 128 10.586697729563905 9.477703066694506 13.505815375014208 35.31811939932565
22 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 1 1 bf6552cc-7c18-4617-8260-b6a63ff65b3c code 267 128 2026-08-31T13:21:54.962650+00:00 1.4145112080150284 13.505133582977578 54 128 10.586717211933143 9.477877372596776 13.505815375014208 35.31811939932565
23 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 1 2 36abe2cb-3ddf-4311-853a-4151aef2b328 short 106 128 2026-08-31T13:21:54.962724+00:00 1.4144840000080876 10.538147750019561 20 93 10.193273508121454 8.825080289828673 13.505815375014208 35.31811939932565
24 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 1 3 110efa2a-5fb4-48e9-835b-3092cd0df4fb medium 252 128 2026-08-31T13:21:54.962552+00:00 1.3138721249997616 13.505327666993253 43 128 10.49915652475652 9.477741166756687 13.505815375014208 35.31811939932565
25 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 2 0 4d78c4f5-e05f-4ffe-afb6-748b8db712f6 code 267 128 2026-08-31T13:22:08.467061+00:00 1.2189033330068924 13.40631824999582 54 128 10.502637423262865 9.547736941127733 13.406804582977202 35.57894776848267
26 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 2 1 9beb5379-ecf9-45d3-8725-13dd100b245d short 106 128 2026-08-31T13:22:08.467363+00:00 1.3166250829817727 10.428931292030029 20 93 10.205978362278222 8.917500498931489 13.406804582977202 35.57894776848267
27 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 2 2 e863870c-69a1-44bd-9d57-7d28f9a30c17 medium 252 128 2026-08-31T13:22:08.467157+00:00 1.3168415420223027 13.406341000052635 43 128 10.58770054495327 9.547720738976986 13.406804582977202 35.57894776848267
28 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 2 3 21ffaa25-b50a-4b5f-b562-4a7810b56506 code 267 128 2026-08-31T13:22:08.467234+00:00 1.316743416013196 13.406194041017443 54 128 10.587743311946818 9.547825401331103 13.406804582977202 35.57894776848267
29 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 3 0 00728b55-60a4-46d3-8000-146e38d0b3a6 short 106 128 2026-08-31T13:22:21.873446+00:00 1.0375588749884628 10.190524249977898 20 93 10.160641517788692 9.126125184403708 12.702927250007633 34.795129602882234
30 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 3 1 531d5ceb-d88b-49ab-ad8d-7bea678c3bb8 medium 252 128 2026-08-31T13:22:21.873549+00:00 1.140847125032451 12.702382000046782 43 128 11.07119438584415 10.076850152949941 12.702927250007633 34.795129602882234
31 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 3 2 a4b2f7bb-ab5c-405b-9514-d64bf34153ea code 267 128 2026-08-31T13:22:21.873705+00:00 1.1407294999808073 12.702299374970607 54 128 11.071160870366917 10.076915700177802 12.702927250007633 34.795129602882234
32 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 3 3 1b77ace8-b3ef-47f2-a2cd-06b6f9400a6e short 106 128 2026-08-31T13:22:21.873633+00:00 1.1407265829620883 10.277835665969178 20 93 10.178274020276117 9.048597683647658 12.702927250007633 34.795129602882234
33 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 4 0 1143b997-b26b-4ca6-a4ad-dd1014020849 medium 252 128 2026-08-31T13:22:34.576247+00:00 1.3149999160086736 13.642209500016179 43 128 10.383534012925246 9.382644358294614 13.642570749972947 34.96408475660248
34 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 4 1 907e97b8-fec4-47ff-b971-45716a15b6fb code 267 128 2026-08-31T13:22:34.576465+00:00 1.419054625032004 13.641894833010156 54 128 10.472197772531725 9.38286078047386 13.642570749972947 34.96408475660248
35 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 4 2 d5875acf-e267-4393-84c1-a01dac48c4a5 short 106 128 2026-08-31T13:22:34.576349+00:00 1.4192612090264447 10.662482584011741 20 93 10.061427312742246 8.722171339295066 13.642570749972947 34.96408475660248
36 20260831T132015Z vllm-metal qwen3.8-27b-mxfp8 measured 4 4 3 d2828df9-67d3-44b0-9809-bdb8c75f5282 medium 252 128 2026-08-31T13:22:34.576542+00:00 1.4190426669665612 13.641892916988581 43 128 10.47218916878814 9.382862098308841 13.642570749972947 34.96408475660248

842
benchmarks/inference/results/20260831T132015Z/raw.json Normal file
View File

@@ -0,0 +1,842 @@
[
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "warmup",
"concurrency": 1,
"round": 0,
"slot": 0,
"request_id": "4c059f2d-caa2-44ae-ab1d-539d608c86d8",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T13:20:21.117908+00:00",
"ttft_s": 0.3135264579905197,
"total_s": 6.390581874991767,
"prompt_tokens": 20,
"output_tokens": 93,
"prompt_tps": null,
"decode_tps": 15.303464197450301,
"request_tps": 14.552665440988472,
"group_elapsed_s": 6.39150691695977,
"group_output_tps": 14.550559235604652,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "warmup",
"concurrency": 1,
"round": 1,
"slot": 0,
"request_id": "1bf28507-8f2f-4127-ba2f-2f5637e722bf",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T13:20:27.508796+00:00",
"ttft_s": 0.3822946660220623,
"total_s": 8.7317384579801,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.330362499509992,
"request_tps": 14.659165596401756,
"group_elapsed_s": 8.731932666967623,
"group_output_tps": 14.658839558419446,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 1,
"round": 0,
"slot": 0,
"request_id": "81b82bc5-db99-4c10-9ec1-1f684485db59",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T13:20:36.240817+00:00",
"ttft_s": 0.2682818329776637,
"total_s": 6.39245533302892,
"prompt_tokens": 20,
"output_tokens": 93,
"prompt_tps": null,
"decode_tps": 15.185722612075187,
"request_tps": 14.548400443172758,
"group_elapsed_s": 6.392639332974795,
"group_output_tps": 14.547981695179217,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 1,
"round": 1,
"slot": 0,
"request_id": "1de32693-20bd-450c-8341-17773d6c57b2",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T13:20:42.633525+00:00",
"ttft_s": 0.4171057919738814,
"total_s": 8.838838333962485,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.198772860789006,
"request_tps": 14.481541031039212,
"group_elapsed_s": 8.839007750037126,
"group_output_tps": 14.481263465286855,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 1,
"round": 2,
"slot": 0,
"request_id": "e821a6d4-8b00-477b-a66f-3579d33c583b",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T13:20:51.472629+00:00",
"ttft_s": 0.4347635000012815,
"total_s": 8.896931416995358,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 15.126147490283795,
"request_tps": 14.386982882152838,
"group_elapsed_s": 8.897098499990534,
"group_output_tps": 14.386712701914696,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 1,
"round": 3,
"slot": 0,
"request_id": "6415410d-9d1c-4521-b2e2-04c40ce2fd72",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:00.369851+00:00",
"ttft_s": 0.2798732080264017,
"total_s": 6.501722833025269,
"prompt_tokens": 20,
"output_tokens": 93,
"prompt_tps": null,
"decode_tps": 14.947323642528074,
"request_tps": 14.303901041060966,
"group_elapsed_s": 6.501933082996402,
"group_output_tps": 14.303438502498576,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 1,
"round": 4,
"slot": 0,
"request_id": "44eda129-4834-4a55-a14a-33a4a510922d",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:06.871859+00:00",
"ttft_s": 0.4577391669736244,
"total_s": 9.024471082957461,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 14.94152043688646,
"request_tps": 14.183656728838715,
"group_elapsed_s": 9.024667916994076,
"group_output_tps": 14.183347373809413,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "warmup",
"concurrency": 4,
"round": 0,
"slot": 0,
"request_id": "a696baea-c3a4-422a-9d29-c4fff0e6d70e",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:15.897040+00:00",
"ttft_s": 1.1126187500194646,
"total_s": 9.980102250003256,
"prompt_tokens": 20,
"output_tokens": 93,
"prompt_tps": null,
"decode_tps": 10.48775562989996,
"request_tps": 9.318541801510065,
"group_elapsed_s": 12.400747582956683,
"group_output_tps": 35.64301241059653,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "warmup",
"concurrency": 4,
"round": 0,
"slot": 1,
"request_id": "2a75391e-8c1c-4244-8cea-f8f97ce3c40d",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:15.896746+00:00",
"ttft_s": 1.1128792079980485,
"total_s": 12.400287791970186,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 11.340069693388886,
"request_tps": 10.322341073639151,
"group_elapsed_s": 12.400747582956683,
"group_output_tps": 35.64301241059653,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "warmup",
"concurrency": 4,
"round": 0,
"slot": 2,
"request_id": "21360267-b5fe-4bd2-8b87-d5bbdf4a9c49",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:15.896944+00:00",
"ttft_s": 1.1127408749889582,
"total_s": 12.400139749981463,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 11.340079447673899,
"request_tps": 10.322464309339042,
"group_elapsed_s": 12.400747582956683,
"group_output_tps": 35.64301241059653,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "warmup",
"concurrency": 4,
"round": 0,
"slot": 3,
"request_id": "106f9109-b093-475b-85c2-dc0847a3f3e3",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:15.896863+00:00",
"ttft_s": 1.0121074999915436,
"total_s": 9.893255082948599,
"prompt_tokens": 20,
"output_tokens": 93,
"prompt_tps": null,
"decode_tps": 10.471619701317342,
"request_tps": 9.400343892910335,
"group_elapsed_s": 12.400747582956683,
"group_output_tps": 35.64301241059653,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "warmup",
"concurrency": 4,
"round": 1,
"slot": 0,
"request_id": "fd6e44d9-a975-4760-853a-c85d6547b8f2",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:28.297798+00:00",
"ttft_s": 1.4682920000050217,
"total_s": 13.96469266602071,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 10.242949423676817,
"request_tps": 9.165973291446168,
"group_elapsed_s": 13.965293916000519,
"group_output_tps": 36.232678169433896,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "warmup",
"concurrency": 4,
"round": 1,
"slot": 1,
"request_id": "a5541448-562a-4c84-aeff-3e2e5b8aba34",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:28.297877+00:00",
"ttft_s": 1.4681764579727314,
"total_s": 13.9645455830032,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 10.242975277003744,
"request_tps": 9.166069833005798,
"group_elapsed_s": 13.965293916000519,
"group_output_tps": 36.232678169433896,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "warmup",
"concurrency": 4,
"round": 1,
"slot": 2,
"request_id": "bb897f87-b4a0-4483-9946-4f6483212386",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:28.297713+00:00",
"ttft_s": 1.3568928329623304,
"total_s": 13.431885249970946,
"prompt_tokens": 20,
"output_tokens": 122,
"prompt_tps": null,
"decode_tps": 10.103526013660515,
"request_tps": 9.082864968658356,
"group_elapsed_s": 13.965293916000519,
"group_output_tps": 36.232678169433896,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "warmup",
"concurrency": 4,
"round": 1,
"slot": 3,
"request_id": "576c3ce6-5566-4a17-b311-3c2f8f3a5a17",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:28.297622+00:00",
"ttft_s": 1.4684889169875532,
"total_s": 13.964902083971538,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 10.242939176993687,
"request_tps": 9.16583583832745,
"group_elapsed_s": 13.965293916000519,
"group_output_tps": 36.232678169433896,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 0,
"slot": 0,
"request_id": "75886439-869f-49d6-881d-4bc7cf650b04",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:42.263079+00:00",
"ttft_s": 1.1142227500095032,
"total_s": 10.27476887498051,
"prompt_tokens": 20,
"output_tokens": 93,
"prompt_tps": null,
"decode_tps": 10.152233145411333,
"request_tps": 9.051298489687577,
"group_elapsed_s": 12.701848333003,
"group_output_tps": 34.79808516147676,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 0,
"slot": 1,
"request_id": "2ecba00e-62dd-4ea7-a1e1-c7b0c12aae0e",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:42.263219+00:00",
"ttft_s": 1.1140572919975966,
"total_s": 12.70125433296198,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 11.046675011003936,
"request_tps": 10.077744815157159,
"group_elapsed_s": 12.701848333003,
"group_output_tps": 34.79808516147676,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 0,
"slot": 2,
"request_id": "c1df76bd-2c64-4e34-a5be-10d74b64ce17",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:42.263452+00:00",
"ttft_s": 1.1138802919886075,
"total_s": 12.701070374983829,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 11.046681644400257,
"request_tps": 10.077890777781237,
"group_elapsed_s": 12.701848333003,
"group_output_tps": 34.79808516147676,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 0,
"slot": 3,
"request_id": "1bf72731-e2e3-4f8d-bf42-627b9fce292a",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:42.263323+00:00",
"ttft_s": 1.0123424580087885,
"total_s": 10.185485332971439,
"prompt_tokens": 20,
"output_tokens": 93,
"prompt_tps": null,
"decode_tps": 10.138291888359873,
"request_tps": 9.130640019572722,
"group_elapsed_s": 12.701848333003,
"group_output_tps": 34.79808516147676,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 1,
"slot": 0,
"request_id": "a04682c5-d585-4466-8fad-17ed2b9ba80b",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:54.962444+00:00",
"ttft_s": 1.4147373330197297,
"total_s": 13.505381957977079,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 10.586697729563905,
"request_tps": 9.477703066694506,
"group_elapsed_s": 13.505815375014208,
"group_output_tps": 35.31811939932565,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 1,
"slot": 1,
"request_id": "bf6552cc-7c18-4617-8260-b6a63ff65b3c",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:54.962650+00:00",
"ttft_s": 1.4145112080150284,
"total_s": 13.505133582977578,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 10.586717211933143,
"request_tps": 9.477877372596776,
"group_elapsed_s": 13.505815375014208,
"group_output_tps": 35.31811939932565,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 1,
"slot": 2,
"request_id": "36abe2cb-3ddf-4311-853a-4151aef2b328",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:54.962724+00:00",
"ttft_s": 1.4144840000080876,
"total_s": 10.538147750019561,
"prompt_tokens": 20,
"output_tokens": 93,
"prompt_tps": null,
"decode_tps": 10.193273508121454,
"request_tps": 8.825080289828673,
"group_elapsed_s": 13.505815375014208,
"group_output_tps": 35.31811939932565,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 1,
"slot": 3,
"request_id": "110efa2a-5fb4-48e9-835b-3092cd0df4fb",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T13:21:54.962552+00:00",
"ttft_s": 1.3138721249997616,
"total_s": 13.505327666993253,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 10.49915652475652,
"request_tps": 9.477741166756687,
"group_elapsed_s": 13.505815375014208,
"group_output_tps": 35.31811939932565,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 2,
"slot": 0,
"request_id": "4d78c4f5-e05f-4ffe-afb6-748b8db712f6",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T13:22:08.467061+00:00",
"ttft_s": 1.2189033330068924,
"total_s": 13.40631824999582,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 10.502637423262865,
"request_tps": 9.547736941127733,
"group_elapsed_s": 13.406804582977202,
"group_output_tps": 35.57894776848267,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 2,
"slot": 1,
"request_id": "9beb5379-ecf9-45d3-8725-13dd100b245d",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T13:22:08.467363+00:00",
"ttft_s": 1.3166250829817727,
"total_s": 10.428931292030029,
"prompt_tokens": 20,
"output_tokens": 93,
"prompt_tps": null,
"decode_tps": 10.205978362278222,
"request_tps": 8.917500498931489,
"group_elapsed_s": 13.406804582977202,
"group_output_tps": 35.57894776848267,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 2,
"slot": 2,
"request_id": "e863870c-69a1-44bd-9d57-7d28f9a30c17",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T13:22:08.467157+00:00",
"ttft_s": 1.3168415420223027,
"total_s": 13.406341000052635,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 10.58770054495327,
"request_tps": 9.547720738976986,
"group_elapsed_s": 13.406804582977202,
"group_output_tps": 35.57894776848267,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 2,
"slot": 3,
"request_id": "21ffaa25-b50a-4b5f-b562-4a7810b56506",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T13:22:08.467234+00:00",
"ttft_s": 1.316743416013196,
"total_s": 13.406194041017443,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 10.587743311946818,
"request_tps": 9.547825401331103,
"group_elapsed_s": 13.406804582977202,
"group_output_tps": 35.57894776848267,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 3,
"slot": 0,
"request_id": "00728b55-60a4-46d3-8000-146e38d0b3a6",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T13:22:21.873446+00:00",
"ttft_s": 1.0375588749884628,
"total_s": 10.190524249977898,
"prompt_tokens": 20,
"output_tokens": 93,
"prompt_tps": null,
"decode_tps": 10.160641517788692,
"request_tps": 9.126125184403708,
"group_elapsed_s": 12.702927250007633,
"group_output_tps": 34.795129602882234,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 3,
"slot": 1,
"request_id": "531d5ceb-d88b-49ab-ad8d-7bea678c3bb8",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T13:22:21.873549+00:00",
"ttft_s": 1.140847125032451,
"total_s": 12.702382000046782,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 11.07119438584415,
"request_tps": 10.076850152949941,
"group_elapsed_s": 12.702927250007633,
"group_output_tps": 34.795129602882234,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 3,
"slot": 2,
"request_id": "a4b2f7bb-ab5c-405b-9514-d64bf34153ea",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T13:22:21.873705+00:00",
"ttft_s": 1.1407294999808073,
"total_s": 12.702299374970607,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 11.071160870366917,
"request_tps": 10.076915700177802,
"group_elapsed_s": 12.702927250007633,
"group_output_tps": 34.795129602882234,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 3,
"slot": 3,
"request_id": "1b77ace8-b3ef-47f2-a2cd-06b6f9400a6e",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T13:22:21.873633+00:00",
"ttft_s": 1.1407265829620883,
"total_s": 10.277835665969178,
"prompt_tokens": 20,
"output_tokens": 93,
"prompt_tps": null,
"decode_tps": 10.178274020276117,
"request_tps": 9.048597683647658,
"group_elapsed_s": 12.702927250007633,
"group_output_tps": 34.795129602882234,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 4,
"slot": 0,
"request_id": "1143b997-b26b-4ca6-a4ad-dd1014020849",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T13:22:34.576247+00:00",
"ttft_s": 1.3149999160086736,
"total_s": 13.642209500016179,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 10.383534012925246,
"request_tps": 9.382644358294614,
"group_elapsed_s": 13.642570749972947,
"group_output_tps": 34.96408475660248,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 4,
"slot": 1,
"request_id": "907e97b8-fec4-47ff-b971-45716a15b6fb",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T13:22:34.576465+00:00",
"ttft_s": 1.419054625032004,
"total_s": 13.641894833010156,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 10.472197772531725,
"request_tps": 9.38286078047386,
"group_elapsed_s": 13.642570749972947,
"group_output_tps": 34.96408475660248,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 4,
"slot": 2,
"request_id": "d5875acf-e267-4393-84c1-a01dac48c4a5",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T13:22:34.576349+00:00",
"ttft_s": 1.4192612090264447,
"total_s": 10.662482584011741,
"prompt_tokens": 20,
"output_tokens": 93,
"prompt_tps": null,
"decode_tps": 10.061427312742246,
"request_tps": 8.722171339295066,
"group_elapsed_s": 13.642570749972947,
"group_output_tps": 34.96408475660248,
"error": null
},
{
"run_id": "20260831T132015Z",
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"phase": "measured",
"concurrency": 4,
"round": 4,
"slot": 3,
"request_id": "d2828df9-67d3-44b0-9809-bdb8c75f5282",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T13:22:34.576542+00:00",
"ttft_s": 1.4190426669665612,
"total_s": 13.641892916988581,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": null,
"decode_tps": 10.47218916878814,
"request_tps": 9.382862098308841,
"group_elapsed_s": 13.642570749972947,
"group_output_tps": 34.96408475660248,
"error": null
}
]

54
benchmarks/inference/results/20260831T132015Z/summary.json Normal file
View File

@@ -0,0 +1,54 @@
[
{
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"concurrency": 1,
"requests": 5,
"successes": 5,
"failures": 0,
"ttft_s_mean": 0.37155269999057056,
"ttft_s_p50": 0.4171057919738814,
"ttft_s_p95": 0.4531440335791558,
"total_s_mean": 7.930883799993898,
"total_s_p50": 8.838838333962485,
"total_s_p95": 8.998963149765041,
"prompt_tps_mean": null,
"prompt_tps_p50": null,
"prompt_tps_p95": null,
"decode_tps_mean": 15.079897408512505,
"decode_tps_p50": 15.126147490283795,
"decode_tps_p95": 15.196162811046243,
"request_tps_mean": 14.380896425252896,
"request_tps_p50": 14.386982882152838,
"request_tps_p95": 14.53502856074605,
"aggregate_output_tps_mean": 14.380548747737752,
"aggregate_output_tps_p50": 14.386712701914696,
"aggregate_output_tps_p95": 14.534638049200744
},
{
"runtime": "vllm-metal",
"model": "qwen3.8-27b-mxfp8",
"concurrency": 4,
"requests": 20,
"successes": 20,
"failures": 0,
"ttft_s_mean": 1.255672066603438,
"ttft_s_p50": 1.3144360205042176,
"ttft_s_p95": 1.419064954231726,
"total_s_mean": 12.251293779097614,
"total_s_p50": 12.702340687508695,
"total_s_p95": 13.641910566360457,
"prompt_tps_mean": null,
"prompt_tps_p50": null,
"prompt_tps_p95": null,
"decode_tps_mean": 10.500220268362742,
"decode_tps_p50": 10.485677148644122,
"decode_tps_p95": 11.071162546140778,
"request_tps_mean": 9.417789343799708,
"request_tps_p50": 9.430282582501674,
"request_tps_p95": 10.077752113288362,
"aggregate_output_tps_mean": 35.09087333775396,
"aggregate_output_tps_p50": 34.96408475660248,
"aggregate_output_tps_p95": 35.52678209465127
}
]

108
benchmarks/inference/results/20260831T134334Z/metadata.json Normal file
View File

File diff suppressed because one or more lines are too long

8
benchmarks/inference/results/20260831T134334Z/raw.csv Normal file
View File

@@ -0,0 +1,8 @@
run_id,runtime,model,phase,concurrency,round,slot,request_id,prompt_name,prompt_chars,max_tokens,started_at,ttft_s,total_s,prompt_tokens,output_tokens,prompt_tps,decode_tps,request_tps,group_elapsed_s,group_output_tps,error
20260831T134334Z,ollama,qwen3.8:27b-mlx,warmup,1,0,0,c3b04207-976b-4a0d-9d3c-80ef8dc4225f,short,106,128,2026-08-31T13:43:39.472249+00:00,3.4916902080294676,5.4140321660088375,20,96,114.72541286980045,48.19996255746575,17.731701079044402,5.4149448340176605,17.728712469407014,
20260831T134334Z,ollama,qwen3.8:27b-mlx,warmup,1,1,0,47d529a2-f167-46a0-af05-a6ca0e1929a4,medium,252,128,2026-08-31T13:43:44.886979+00:00,0.35504300001775846,2.7301927499938756,43,128,157.93211765391837,52.61516193511847,46.88313673101913,2.730699084000662,46.87444352618714,
20260831T134334Z,ollama,qwen3.8:27b-mlx,measured,1,0,0,5b638c8b-a185-48d7-8625-f6ebacc06853,short,106,128,2026-08-31T13:43:47.617556+00:00,0.11980204097926617,2.1940376659622416,20,96,441.616869764425,45.28260592046214,43.7549461840698,2.1943347500055097,43.749022340260055,
20260831T134334Z,ollama,qwen3.8:27b-mlx,measured,1,1,0,fd1c62de-2412-4243-85a0-0851e2364f7c,medium,252,128,2026-08-31T13:43:49.811979+00:00,0.12685995898209512,2.771603333996609,43,128,831.312776076798,47.52723693214773,46.18265479404875,2.771931625029538,46.17718519613052,
20260831T134334Z,ollama,qwen3.8:27b-mlx,measured,1,2,0,50050b20-0fd0-4e15-a4bb-ddfc74ada7e1,code,267,128,2026-08-31T13:43:52.584034+00:00,0.42160862497985363,2.646599249972496,54,128,159.3484419263456,56.09763282504741,48.363952344250905,2.6469858330092393,48.35688895791427,
20260831T134334Z,ollama,qwen3.8:27b-mlx,measured,1,3,0,22721212-cd1e-460f-9aa3-f2a1bb6f2f96,short,106,128,2026-08-31T13:43:55.231045+00:00,0.09599920798791572,2.2750009580049664,20,96,2500000.0,42.69653289142422,42.19778442826943,2.2753357499605045,42.19157546382611,
20260831T134334Z,ollama,qwen3.8:27b-mlx,measured,1,4,0,d95f9110-8465-4bf1-8b6d-511699a58906,medium,252,128,2026-08-31T13:43:57.506343+00:00,0.09756654204102233,2.8720681670238264,43,128,3766313.3923097136,45.01625903574264,44.56718732154595,2.8723847499932162,44.56227530114212,
1 run_id runtime model phase concurrency round slot request_id prompt_name prompt_chars max_tokens started_at ttft_s total_s prompt_tokens output_tokens prompt_tps decode_tps request_tps group_elapsed_s group_output_tps error
2 20260831T134334Z ollama qwen3.8:27b-mlx warmup 1 0 0 c3b04207-976b-4a0d-9d3c-80ef8dc4225f short 106 128 2026-08-31T13:43:39.472249+00:00 3.4916902080294676 5.4140321660088375 20 96 114.72541286980045 48.19996255746575 17.731701079044402 5.4149448340176605 17.728712469407014
3 20260831T134334Z ollama qwen3.8:27b-mlx warmup 1 1 0 47d529a2-f167-46a0-af05-a6ca0e1929a4 medium 252 128 2026-08-31T13:43:44.886979+00:00 0.35504300001775846 2.7301927499938756 43 128 157.93211765391837 52.61516193511847 46.88313673101913 2.730699084000662 46.87444352618714
4 20260831T134334Z ollama qwen3.8:27b-mlx measured 1 0 0 5b638c8b-a185-48d7-8625-f6ebacc06853 short 106 128 2026-08-31T13:43:47.617556+00:00 0.11980204097926617 2.1940376659622416 20 96 441.616869764425 45.28260592046214 43.7549461840698 2.1943347500055097 43.749022340260055
5 20260831T134334Z ollama qwen3.8:27b-mlx measured 1 1 0 fd1c62de-2412-4243-85a0-0851e2364f7c medium 252 128 2026-08-31T13:43:49.811979+00:00 0.12685995898209512 2.771603333996609 43 128 831.312776076798 47.52723693214773 46.18265479404875 2.771931625029538 46.17718519613052
6 20260831T134334Z ollama qwen3.8:27b-mlx measured 1 2 0 50050b20-0fd0-4e15-a4bb-ddfc74ada7e1 code 267 128 2026-08-31T13:43:52.584034+00:00 0.42160862497985363 2.646599249972496 54 128 159.3484419263456 56.09763282504741 48.363952344250905 2.6469858330092393 48.35688895791427
7 20260831T134334Z ollama qwen3.8:27b-mlx measured 1 3 0 22721212-cd1e-460f-9aa3-f2a1bb6f2f96 short 106 128 2026-08-31T13:43:55.231045+00:00 0.09599920798791572 2.2750009580049664 20 96 2500000.0 42.69653289142422 42.19778442826943 2.2753357499605045 42.19157546382611
8 20260831T134334Z ollama qwen3.8:27b-mlx measured 1 4 0 d95f9110-8465-4bf1-8b6d-511699a58906 medium 252 128 2026-08-31T13:43:57.506343+00:00 0.09756654204102233 2.8720681670238264 43 128 3766313.3923097136 45.01625903574264 44.56718732154595 2.8723847499932162 44.56227530114212

170
benchmarks/inference/results/20260831T134334Z/raw.json Normal file
View File

@@ -0,0 +1,170 @@
[
{
"run_id": "20260831T134334Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "warmup",
"concurrency": 1,
"round": 0,
"slot": 0,
"request_id": "c3b04207-976b-4a0d-9d3c-80ef8dc4225f",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T13:43:39.472249+00:00",
"ttft_s": 3.4916902080294676,
"total_s": 5.4140321660088375,
"prompt_tokens": 20,
"output_tokens": 96,
"prompt_tps": 114.72541286980045,
"decode_tps": 48.19996255746575,
"request_tps": 17.731701079044402,
"group_elapsed_s": 5.4149448340176605,
"group_output_tps": 17.728712469407014,
"error": null
},
{
"run_id": "20260831T134334Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "warmup",
"concurrency": 1,
"round": 1,
"slot": 0,
"request_id": "47d529a2-f167-46a0-af05-a6ca0e1929a4",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T13:43:44.886979+00:00",
"ttft_s": 0.35504300001775846,
"total_s": 2.7301927499938756,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 157.93211765391837,
"decode_tps": 52.61516193511847,
"request_tps": 46.88313673101913,
"group_elapsed_s": 2.730699084000662,
"group_output_tps": 46.87444352618714,
"error": null
},
{
"run_id": "20260831T134334Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 1,
"round": 0,
"slot": 0,
"request_id": "5b638c8b-a185-48d7-8625-f6ebacc06853",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T13:43:47.617556+00:00",
"ttft_s": 0.11980204097926617,
"total_s": 2.1940376659622416,
"prompt_tokens": 20,
"output_tokens": 96,
"prompt_tps": 441.616869764425,
"decode_tps": 45.28260592046214,
"request_tps": 43.7549461840698,
"group_elapsed_s": 2.1943347500055097,
"group_output_tps": 43.749022340260055,
"error": null
},
{
"run_id": "20260831T134334Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 1,
"round": 1,
"slot": 0,
"request_id": "fd1c62de-2412-4243-85a0-0851e2364f7c",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T13:43:49.811979+00:00",
"ttft_s": 0.12685995898209512,
"total_s": 2.771603333996609,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 831.312776076798,
"decode_tps": 47.52723693214773,
"request_tps": 46.18265479404875,
"group_elapsed_s": 2.771931625029538,
"group_output_tps": 46.17718519613052,
"error": null
},
{
"run_id": "20260831T134334Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 1,
"round": 2,
"slot": 0,
"request_id": "50050b20-0fd0-4e15-a4bb-ddfc74ada7e1",
"prompt_name": "code",
"prompt_chars": 267,
"max_tokens": 128,
"started_at": "2026-08-31T13:43:52.584034+00:00",
"ttft_s": 0.42160862497985363,
"total_s": 2.646599249972496,
"prompt_tokens": 54,
"output_tokens": 128,
"prompt_tps": 159.3484419263456,
"decode_tps": 56.09763282504741,
"request_tps": 48.363952344250905,
"group_elapsed_s": 2.6469858330092393,
"group_output_tps": 48.35688895791427,
"error": null
},
{
"run_id": "20260831T134334Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 1,
"round": 3,
"slot": 0,
"request_id": "22721212-cd1e-460f-9aa3-f2a1bb6f2f96",
"prompt_name": "short",
"prompt_chars": 106,
"max_tokens": 128,
"started_at": "2026-08-31T13:43:55.231045+00:00",
"ttft_s": 0.09599920798791572,
"total_s": 2.2750009580049664,
"prompt_tokens": 20,
"output_tokens": 96,
"prompt_tps": 2500000.0,
"decode_tps": 42.69653289142422,
"request_tps": 42.19778442826943,
"group_elapsed_s": 2.2753357499605045,
"group_output_tps": 42.19157546382611,
"error": null
},
{
"run_id": "20260831T134334Z",
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"phase": "measured",
"concurrency": 1,
"round": 4,
"slot": 0,
"request_id": "d95f9110-8465-4bf1-8b6d-511699a58906",
"prompt_name": "medium",
"prompt_chars": 252,
"max_tokens": 128,
"started_at": "2026-08-31T13:43:57.506343+00:00",
"ttft_s": 0.09756654204102233,
"total_s": 2.8720681670238264,
"prompt_tokens": 43,
"output_tokens": 128,
"prompt_tps": 3766313.3923097136,
"decode_tps": 45.01625903574264,
"request_tps": 44.56718732154595,
"group_elapsed_s": 2.8723847499932162,
"group_output_tps": 44.56227530114212,
"error": null
}
]

28
benchmarks/inference/results/20260831T134334Z/summary.json Normal file
View File

@@ -0,0 +1,28 @@
[
{
"runtime": "ollama",
"model": "qwen3.8:27b-mlx",
"concurrency": 1,
"requests": 5,
"successes": 5,
"failures": 0,
"ttft_s_mean": 0.1723672749940306,
"ttft_s_p50": 0.11980204097926617,
"ttft_s_p95": 0.36265889178030186,
"total_s_mean": 2.551861874992028,
"total_s_p50": 2.646599249972496,
"total_s_p95": 2.8519752004183827,
"prompt_tps_mean": 1253549.1340794961,
"prompt_tps_p50": 831.312776076798,
"prompt_tps_p95": 3513050.713847771,
"decode_tps_mean": 47.324053520964824,
"decode_tps_p50": 45.28260592046214,
"decode_tps_p95": 54.383553646467476,
"request_tps_mean": 45.013305014436966,
"request_tps_p50": 44.56718732154595,
"request_tps_p95": 47.927692834210475,
"aggregate_output_tps_mean": 45.00738945185462,
"aggregate_output_tps_p50": 44.56227530114212,
"aggregate_output_tps_p95": 47.92094820555752
}
]

37
benchmarks/inference/results/backend_verification.json Normal file
View File

@@ -0,0 +1,37 @@
{
"verified_at": "2026-08-31",
"host": "Apple M4 Max, arm64, macOS 15.6",
"ollama": {
"version": "0.32.14",
"mlx_version": "0.32.0-213-gadf21de",
"device": "gpu",
"metal_library": true,
"nvfp4_model": {
"name": "qwen3.8:27b-mlx",
"id": "5642e97495e1",
"format": "safetensors",
"quantization": "nvfp4",
"parameter_size": "27.8B",
"runner_log": "starting mlx runner subprocess"
},
"mxfp8_model": {
"name": "qwen3.8:27b-mxfp8",
"id": "464021588235",
"format": "safetensors",
"quantization": "mxfp8",
"parameter_size": "27.8B",
"runner_log": "starting mlx runner subprocess"
}
},
"vllm_metal": {
"vllm_version": "0.28.0+cpu",
"plugin_version": "0.4.0.dev20260831112704",
"mlx_version": "0.32.0",
"mlx_lm_version": "0.31.3",
"mlx_vlm_version": "0.6.4",
"platform_plugin": "metal",
"compute_log": "MLX device set to: Device(gpu, 0)",
"pytorch_device_log": "PyTorch device set to: mps",
"note": "The +cpu suffix identifies the portable vLLM core wheel; the Metal plugin supplies MLX GPU execution."
}
}

27
benchmarks/inference/results/cold_start.json Normal file
View File

@@ -0,0 +1,27 @@
{
"measured_at": "2026-08-31",
"notes": [
"Cold starts were measured separately from warm inference.",
"The vLLM model was already present in the Hugging Face cache; zero bytes were downloaded.",
"vLLM launch-to-ready is derived from the recorded epoch and second-resolution API startup log.",
"Ollama's daemon was already running, but the model was fully unloaded before the request.",
"The Ollama request used one output token and keep_alive=0 so generation time was negligible and the model was unloaded afterward."
],
"vllm_metal": {
"process_start_epoch": 1788177244.465068,
"api_ready_local_time": "2026-08-31T13:54:33+02:00",
"approx_launch_to_api_ready_s": 28.535,
"model_load_s": 3.25,
"engine_profile_cache_warmup_s": 9.71,
"download_bytes": 0
},
"ollama": {
"client_total_s": 3.455159667,
"server_total_s": 3.448873125,
"model_load_s": 3.244982292,
"prompt_eval_s": 0.13364925,
"decode_s": 0.069530292,
"prompt_tokens": 7,
"output_tokens": 1
}
}