lovey/benchmarks/standard

Standard benchmark runs

Strata's standard-v1 suite (strata.py bench run, protocol strata-qwen38-thinking-v1-draft.1): 3× short / 4K / 16K prompts with a 512-token cap at the Qwen model-card sampling (temperature 1.0, thinking on, effort xhigh), then 12 sanity cases (instruction, retrieval, tool calls). Each folder is the runner's own output: README.md (its report), results.json, config.json, outputs.jsonl, provenance/.

Engine 0.1.51 at 4b9e05a2, gpu-unlock, idle server, run from the guest over loopback. Tuning arms and the power-cap runs use IQ3_S (strata-iq3_s.json); the quant comparison at the end says which config each run used.

Tuning arms

Each arm = the setup config (strata/strata-iq3_s.json) plus one change, a fresh engine restart, then the suite. Median of 3; decode and prompt are the engine's timings.

Arm	Folder	Prompt 4K tok/s	Prompt 16K tok/s	TTFT 16K	Decode short	Decode 4K	Decode 16K
Warm baseline (15 h uptime, + concurrency stage)	0f07e24d	1,776	2,938	5.61 s	130.7	123.2	124.5
Restart baseline	87dfe6da	1,712	2,881	5.71 s	122.5	110.1	116.6
--adapt-async 1	1ecae1bc	1,712	2,870	5.73 s	125.1	115.0	108.6
--pipeline-windows 2	5f934909	1,705	2,878	5.71 s	110.0	104.2	102.7
--prefill auto:32768	92972582	1,702	2,239	7.35 s	122.7	111.6	105.9
all three	53bbc5d8	1,699	2,243	7.34 s	121.1	121.7	113.4

All 12 sanity checks passed in every arm. The arms report "nonconforming" only because they used --skip-model-hash and --track custom. GPU temps peaked at 56 / 64°C (GPU1 ~230 W).

Four requests at once (warm baseline, server capacity 1): 115.5 / 126.5 / 126.2 tok/s total per round, so concurrent clients just queue.

Conclusions

• Keep the setup defaults. None of the three flags helped on this machine.
• --prefill auto:32768 is clearly worse at 16K: −22% prompt speed, +1.6 s to first token. Strata's docs say chunks above 8192 aren't bit-exact on CUDA anyway. The gains reported elsewhere were at 74K prompts on AMD; very long prompts were not tested here.
• --pipeline-windows 2 left prompt speed unchanged and cost decode speed (−10% short, −12% at 16K). The community 2×3090 report's +17% prefill did not reproduce (their cards were x16 + x4; ours are x8 + x8).
• --adapt-async 1 is within noise.
• Noise: decode varies 94–135 tok/s within an arm (sampling at temperature 1.0 changes how many draft tokens are accepted), so decode differences under ~10% between single runs aren't meaningful. Prompt speed is tight (±2%) and is the reliable comparison.
• Prompt speed is already above the community 2×3090 report (1,798 tok/s at 19.9K). The 915 tok/s in the earlier notes was a first cold prompt right after load, not a steady-state number.

Power cap: 250 W vs 275 W

2026-10-27, setup defaults, case closed, caps set with nvidia-smi -pl at runtime. Runs alternated 250 / 275 / 250 / 275, each after a fresh restart. GPU temp, power, SM clock and throttle reasons were sampled every 2 s.

Cap	Folder	Prompt 4K tok/s	Prompt 16K tok/s	TTFT 16K	Decode short	Decode 4K	Decode 16K
250 W	6a05efdc	1,719	2,897	5.69 s	122.9	112.9	108.2
275 W	5c64d992	1,742	3,030	5.44 s	125.3	115.3	116.1
250 W	ae1396b5	1,712	2,887	5.71 s	116.3	122.1	106.8
275 W	16ea71d0	1,743	3,016	5.46 s	114.2	111.6	111.0

While busy (samples above 150 W):

Cap	GPU	Peak temp	Mean / peak W	Mean SM MHz	Power-throttled
250 W	0	51°C	220 / 250	1,858	62 of 80 samples
250 W	1	59°C	223 / 249	1,807	82 of 86
275 W	0	53°C	228 / 273	1,867	45 of 78
275 W	1	63°C	239 / 272	1,839	80 of 87

• 275 W: +4.6% prompt speed at 16K (2,892 → 3,023 tok/s, −0.25 s to first token), +1.6% at 4K. Both repeats agree to within 0.5%, so this is real and not noise.
• Decode is unchanged within noise; decode isn't power-bound.
• Cost: +2–4°C peak and +8–16 W mean per card while busy. All 12 sanity checks passed in every run.
• GPU1 still hits its power limit most of the time at 275 W, so a higher cap would probably buy a little more prompt speed.

Quant: IQ3_S vs UD-Q4_K_XL

2026-10-27, 275 W, case closed, setup defaults for each quant (strata-iq3_s.json, strata-unsloth-ud-q4_k_xl.json), fresh restart before each run. IQ3_S rows are the two 275 W runs from the power-cap test.

Quant	Folder	Prompt 4K tok/s	Prompt 16K tok/s	TTFT 16K	Decode short	Decode 4K	Decode 16K
IQ3_S	5c64d992	1,742	3,030	5.44 s	125.3	115.3	116.1
IQ3_S	16ea71d0	1,743	3,016	5.46 s	114.2	111.6	111.0
UD-Q4_K_XL	58540dcc	846	1,918	8.57 s	66.9	69.7	67.8
UD-Q4_K_XL	3eb539d8	847	1,918	8.57 s	65.7	68.9	76.3

• UD-Q4_K_XL is much slower: prompt −51% at 4K and −37% at 16K (+3.1 s to first token at 16K), decode −40% (~66–76 vs ~111–125 tok/s). Both repeats agree to within 0.2% on prompt speed.
• Why: its experts are 71.7 GiB vs 47 GiB, so only 5,328 fit in VRAM (15.6 GiB) vs 9,508 for IQ3_S, and every miss moves more bytes. Load took 2 min 22 s; 82 GB of the guest's RAM in use, 27 GB available.
• The GPUs work less hard: mean 211–227 W while busy (IQ3_S 228–239 W), peaks 54 / 63°C.
• All 12 sanity checks passed in both runs. Quality differences these checks can't see are left to husky-bench; Strata's yardstick puts UD-Q4_K_XL at 93.1% same-top-token as Q8_0 vs 87.7% for IQ3_S.

Not tried yet

• Huge pages for the expert arena: the engine logs MAP_HUGETLB unavailable (needed 23983 2 MiB pages, vm.nr_hugepages=0). That would mean reserving ~47 GB of huge pages in the guest (a NixOS change and a reboot).
• Power cap above 275 W (300–350 W). gpu-unlock GEMM was +40% at 350 W, but 275 W gave only +4.6% prompt speed.
• Other quants: UD-IQ4_XS (barely better than IQ3_S on Strata's yardstick), IQ3_XXS / IQ2_XS (smaller, faster, worse).