Jason WittyandClaude Opus 5.5
698005c602
Standard benchmark, two fresh-restart runs: prompt -51% at 4K, -37% at 16K, decode -40% vs IQ3_S. 12/12 sanity checks. Kept active for husky-bench. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
106 lines
6.3 KiB
Markdown
106 lines
6.3 KiB
Markdown
# Standard benchmark runs
|
||
|
||
Strata's `standard-v1` suite (`strata.py bench run`, protocol `strata-qwen38-thinking-v1-draft.1`): 3×
|
||
short / 4K / 16K prompts with a 512-token cap at the Qwen model-card sampling (temperature 1.0, thinking
|
||
on, effort xhigh), then 12 sanity cases (instruction, retrieval, tool calls). Each folder is the runner's
|
||
own output: `README.md` (its report), `results.json`, `config.json`, `outputs.jsonl`, `provenance/`.
|
||
|
||
Engine 0.1.51 at `4b9e05a2`, gpu-unlock, idle server, run from the guest over loopback. Tuning arms and
|
||
the power-cap runs use IQ3_S ([`strata-iq3_s.json`](../../strata/strata-iq3_s.json)); the quant comparison
|
||
at the end says which config each run used.
|
||
|
||
## Tuning arms
|
||
|
||
Each arm = the setup config ([`strata/strata-iq3_s.json`](../../strata/strata-iq3_s.json)) plus one
|
||
change, a fresh engine restart, then the suite. Median of 3; decode and prompt are the engine's timings.
|
||
|
||
| Arm | Folder | Prompt 4K tok/s | Prompt 16K tok/s | TTFT 16K | Decode short | Decode 4K | Decode 16K |
|
||
|---|---|---:|---:|---:|---:|---:|---:|
|
||
| Warm baseline (15 h uptime, + concurrency stage) | `0f07e24d` | 1,776 | 2,938 | 5.61 s | 130.7 | 123.2 | 124.5 |
|
||
| **Restart baseline** | `87dfe6da` | **1,712** | **2,881** | **5.71 s** | **122.5** | **110.1** | **116.6** |
|
||
| `--adapt-async 1` | `1ecae1bc` | 1,712 | 2,870 | 5.73 s | 125.1 | 115.0 | 108.6 |
|
||
| `--pipeline-windows 2` | `5f934909` | 1,705 | 2,878 | 5.71 s | 110.0 | 104.2 | 102.7 |
|
||
| `--prefill auto:32768` | `92972582` | 1,702 | **2,239** | **7.35 s** | 122.7 | 111.6 | 105.9 |
|
||
| all three | `53bbc5d8` | 1,699 | 2,243 | 7.34 s | 121.1 | 121.7 | 113.4 |
|
||
|
||
All 12 sanity checks passed in every arm. The arms report "nonconforming" only because they used
|
||
`--skip-model-hash` and `--track custom`. GPU temps peaked at 56 / 64°C (GPU1 ~230 W).
|
||
|
||
Four requests at once (warm baseline, server capacity 1): 115.5 / 126.5 / 126.2 tok/s total per round,
|
||
so concurrent clients just queue.
|
||
|
||
## Conclusions
|
||
|
||
- **Keep the setup defaults.** None of the three flags helped on this machine.
|
||
- `--prefill auto:32768` is clearly worse at 16K: −22% prompt speed, +1.6 s to first token. Strata's
|
||
docs say chunks above 8192 aren't bit-exact on CUDA anyway. The gains reported elsewhere were at 74K
|
||
prompts on AMD; very long prompts were not tested here.
|
||
- `--pipeline-windows 2` left prompt speed unchanged and cost decode speed (−10% short, −12% at 16K). The
|
||
community 2×3090 report's +17% prefill did not reproduce (their cards were x16 + x4; ours are x8 + x8).
|
||
- `--adapt-async 1` is within noise.
|
||
- **Noise:** decode varies 94–135 tok/s *within* an arm (sampling at temperature 1.0 changes how many
|
||
draft tokens are accepted), so decode differences under ~10% between single runs aren't meaningful.
|
||
Prompt speed is tight (±2%) and is the reliable comparison.
|
||
- Prompt speed is already above the community 2×3090 report (1,798 tok/s at 19.9K). The 915 tok/s in
|
||
the earlier notes was a first cold prompt right after load, not a steady-state number.
|
||
|
||
## Power cap: 250 W vs 275 W
|
||
|
||
2026-10-27, setup defaults, **case closed**, caps set with `nvidia-smi -pl` at runtime. Runs alternated
|
||
250 / 275 / 250 / 275, each after a fresh restart. GPU temp, power, SM clock and throttle reasons were
|
||
sampled every 2 s.
|
||
|
||
| Cap | Folder | Prompt 4K tok/s | Prompt 16K tok/s | TTFT 16K | Decode short | Decode 4K | Decode 16K |
|
||
|---|---|---:|---:|---:|---:|---:|---:|
|
||
| 250 W | `6a05efdc` | 1,719 | 2,897 | 5.69 s | 122.9 | 112.9 | 108.2 |
|
||
| 275 W | `5c64d992` | 1,742 | 3,030 | 5.44 s | 125.3 | 115.3 | 116.1 |
|
||
| 250 W | `ae1396b5` | 1,712 | 2,887 | 5.71 s | 116.3 | 122.1 | 106.8 |
|
||
| 275 W | `16ea71d0` | 1,743 | 3,016 | 5.46 s | 114.2 | 111.6 | 111.0 |
|
||
|
||
While busy (samples above 150 W):
|
||
|
||
| Cap | GPU | Peak temp | Mean / peak W | Mean SM MHz | Power-throttled |
|
||
|---|---|---:|---:|---:|---:|
|
||
| 250 W | 0 | 51°C | 220 / 250 | 1,858 | 62 of 80 samples |
|
||
| 250 W | 1 | 59°C | 223 / 249 | 1,807 | 82 of 86 |
|
||
| 275 W | 0 | 53°C | 228 / 273 | 1,867 | 45 of 78 |
|
||
| 275 W | 1 | 63°C | 239 / 272 | 1,839 | 80 of 87 |
|
||
|
||
- **275 W: +4.6% prompt speed at 16K** (2,892 → 3,023 tok/s, −0.25 s to first token), +1.6% at 4K. Both
|
||
repeats agree to within 0.5%, so this is real and not noise.
|
||
- Decode is unchanged within noise; decode isn't power-bound.
|
||
- Cost: +2–4°C peak and +8–16 W mean per card while busy. All 12 sanity checks passed in every run.
|
||
- GPU1 still hits its power limit most of the time at 275 W, so a higher cap would probably buy a
|
||
little more prompt speed.
|
||
|
||
## Quant: IQ3_S vs UD-Q4_K_XL
|
||
|
||
2026-10-27, 275 W, case closed, setup defaults for each quant
|
||
([`strata-iq3_s.json`](../../strata/strata-iq3_s.json),
|
||
[`strata-unsloth-ud-q4_k_xl.json`](../../strata/strata-unsloth-ud-q4_k_xl.json)), fresh restart before each
|
||
run. IQ3_S rows are the two 275 W runs from the power-cap test.
|
||
|
||
| Quant | Folder | Prompt 4K tok/s | Prompt 16K tok/s | TTFT 16K | Decode short | Decode 4K | Decode 16K |
|
||
|---|---|---:|---:|---:|---:|---:|---:|
|
||
| IQ3_S | `5c64d992` | 1,742 | 3,030 | 5.44 s | 125.3 | 115.3 | 116.1 |
|
||
| IQ3_S | `16ea71d0` | 1,743 | 3,016 | 5.46 s | 114.2 | 111.6 | 111.0 |
|
||
| UD-Q4_K_XL | `58540dcc` | 846 | 1,918 | 8.57 s | 66.9 | 69.7 | 67.8 |
|
||
| UD-Q4_K_XL | `3eb539d8` | 847 | 1,918 | 8.57 s | 65.7 | 68.9 | 76.3 |
|
||
|
||
- **UD-Q4_K_XL is much slower:** prompt −51% at 4K and −37% at 16K (+3.1 s to first token at 16K),
|
||
decode −40% (~66–76 vs ~111–125 tok/s). Both repeats agree to within 0.2% on prompt speed.
|
||
- Why: its experts are 71.7 GiB vs 47 GiB, so only 5,328 fit in VRAM (15.6 GiB) vs 9,508 for IQ3_S, and
|
||
every miss moves more bytes. Load took 2 min 22 s; 82 GB of the guest's RAM in use, 27 GB available.
|
||
- The GPUs work less hard: mean 211–227 W while busy (IQ3_S 228–239 W), peaks 54 / 63°C.
|
||
- All 12 sanity checks passed in both runs. Quality differences these checks can't see are left to
|
||
husky-bench; Strata's yardstick puts UD-Q4_K_XL at 93.1% same-top-token as Q8_0 vs 87.7% for IQ3_S.
|
||
|
||
## Not tried yet
|
||
|
||
- Huge pages for the expert arena: the engine logs `MAP_HUGETLB unavailable (needed 23983 2 MiB pages,
|
||
vm.nr_hugepages=0)`. That would mean reserving ~47 GB of huge pages in the guest (a NixOS change and a
|
||
reboot).
|
||
- Power cap above 275 W (300–350 W). gpu-unlock GEMM was +40% at 350 W, but 275 W gave only +4.6% prompt speed.
|
||
- Other quants: UD-IQ4_XS (barely better than IQ3_S on Strata's yardstick), IQ3_XXS / IQ2_XS (smaller,
|
||
faster, worse).
|