Jason WittyandClaude Opus 5.5
d525f315f4
standard-v1 on the idle server: ~2,900 prompt tok/s at 16K, decode ~110-131. --adapt-async 1 neutral, --pipeline-windows 2 slower decode, --prefill auto:32768 22% slower prompt at 16K. Setup defaults kept and the original config restored. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
76 lines
4.4 KiB
Markdown
76 lines
4.4 KiB
Markdown
# <user> standard benchmark: NVIDIA GeForce RTX 3090
|
|
|
|
Protocol `strata-qwen38-thinking-v1-draft.1` (draft), run `20261026T235926Z-5f934909`, started 2026-10-26T23:59:26Z. **Status: nonconforming.**
|
|
|
|
Written by `<user> bench run`. The machine-readable result is [results.json](results.json); every request, its raw usage and timings and the client event offsets are in [outputs.jsonl](outputs.jsonl).
|
|
|
|
This run is not a complete standard result:
|
|
|
|
- no model file could be hashed
|
|
|
|
## Hardware and software
|
|
|
|
- GPU: NVIDIA GeForce RTX 3090 (24.0 GiB dedicated)
|
|
- CPU: Intel(R) Core(TM) i9-14900KS; RAM: 110.0 GiB
|
|
- Storage of the model files: not measured
|
|
- OS: NixOS 26.05 (Yarara) 6.18.55; drivers: gpu-driver 595.71.05; runtime: cuda 13.2
|
|
- <user> commit: `4b9e05a2a806a3fe24f7b2560d98626c37dade35`; engine 0.1.51; source build
|
|
- Background work: not measured
|
|
|
|
## Model and configuration
|
|
|
|
- Model: qwen3.8-flash-next-iq3_s; quantization IQ3_S; revision not measured; source revision not measured
|
|
- Files: not measured
|
|
- Track: **custom** (arm pipeline-windows: ["--pipeline-windows","2"])
|
|
- Context 131072 tokens; server capacity 1 slot(s)
|
|
- Sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0, repetition_penalty 1.0, reasoning_effort xhigh; thinking on, preserved
|
|
- Engine options: `config.json`; chat template: `provenance/chat_template.jinja`
|
|
|
|
## Method
|
|
|
|
The frozen requests of `bench/prompts/standard-v1` (corpus `e75eff8b9feea04f...`) were sent over loopback HTTP, streaming, one at a time, in this order: 1 warm-up (excluded), 3 repetitions of the short, 4K and 16K prompts, and the twelve sanity checks. Prompt and decode speed are the engine's own `timings`; time to first token is the client's monotonic clock from dispatch to the first nonempty reasoning, answer or tool-call delta. No request was retried. Time after the model loaded: 1 min 37 s.
|
|
|
|
## Results
|
|
|
|
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
|
|
| --- | --- | --- | --- | --- | --- | --- |
|
|
| short | n/a | 110.0 (102.5-113.8) n=3 | 0.47 (0.39-0.48) n=3 | 1.26 (0.93-3.65) n=3 | 3.8 (3.7-5.5) n=3 | 1/3 |
|
|
| prompt-4k | 1705.1 (1540.3-1743.3) n=3 | 104.2 (95.9-134.5) n=3 | 2.42 (2.37-2.68) n=3 | 5.98 (4.03-7.92) n=2 | 7.3 (6.2-8.0) n=3 | 3/3 |
|
|
| prompt-16k | 2878.4 (2844.7-2883.2) n=3 | 102.7 (100.7-103.3) n=3 | 5.71 (5.71-5.78) n=3 | 10.47 (10.11-10.84) n=2 | 10.7 (10.7-10.9) n=3 | 3/3 |
|
|
|
|
Cells are median (min-max) n=samples. Short is a prompt of about 80 tokens: its prompt rate is dominated by the fixed cost of a request, so it is not shown; read prompt speed from prompt-4k and prompt-16k. Capped = requests that reached the 512-token cap, reasoning and answer together. A capped request's total time is the time to the cap, not to an answer, and it has no first-answer time when it was still reasoning.
|
|
|
|
| Phase | Planned | Completed | Truncated | Failed | Skipped |
|
|
| --- | ---: | ---: | ---: | ---: | ---: |
|
|
| warmup | 1 | 0 | 1 | 0 | 0 |
|
|
| performance | 9 | 2 | 7 | 0 | 0 |
|
|
| sanity | 12 | 12 | 0 | 0 | 0 |
|
|
| concurrency | 0 | 0 | 0 | 0 | 0 |
|
|
|
|
Sanity checks: **12 of 12**.
|
|
|
|
| Case | Result | Reason |
|
|
| --- | --- | --- |
|
|
| I1 | pass | the expected JSON value |
|
|
| I2 | pass | the expected JSON value |
|
|
| I3 | pass | the expected JSON value |
|
|
| I4 | pass | the exact expected text |
|
|
| R1 | pass | the exact expected text |
|
|
| R2 | pass | the exact expected text |
|
|
| R3 | pass | the exact expected text |
|
|
| R4 | pass | the expected JSON value |
|
|
| T1 | pass | the expected tool call |
|
|
| T2 | pass | the expected tool call |
|
|
| T3 | pass | the expected tool call |
|
|
| T4 | pass | the expected tool call |
|
|
|
|
## Failures, exclusions and limitations
|
|
|
|
- No request failed.
|
|
- Excluded from speed summaries: performance-prompt-16k-r3: first_answer_s is not available for this request
|
|
- Excluded from speed summaries: performance-prompt-4k-r2: first_answer_s is not available for this request
|
|
- Prompt-prefix reuse is the shared template prefix only; the cache was not emptied between requests, and the expert cache stays warm and evolves. These are not cold-model measurements.
|
|
- Output caps include reasoning; a request that reaches its cap is `truncated` and fails its sanity check.
|
|
- The runner cannot see other programs on the machine, the repeat-stop and loop-recovery switches of the server, or the quantization repository's revision; see `missing` in results.json for every value it could not read.
|
|
- The server had answered no request before this run (a freshly loaded engine).
|