Add standard benchmark baseline and tuning arms
standard-v1 on the idle server: ~2,900 prompt tok/s at 16K, decode ~110-131. --adapt-async 1 neutral, --pipeline-windows 2 slower decode, --prefill auto:32768 22% slower prompt at 16K. Setup defaults kept and the original config restored. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jason Witty
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
f983b8e0ac
commit
d525f315f4
@@ -42,8 +42,8 @@ serving Qwen3.8-Flash-Next IQ3_S on an OpenAI/Anthropic-compatible API, used fro
|
||||
| Client | pi coding agent 1.4.0 | [docs/pi-client.md](docs/pi-client.md) |
|
||||
| Fans | `lovey-fans.py` on the host, fed GPU temps by the guest | [docs/proxmox-host.md](docs/proxmox-host.md#fan-control) |
|
||||
|
||||
Current numbers: decode ~127–145 tok/s (MTP draft acceptance ~83%), cold prompt ~915 tok/s on 17K
|
||||
tokens, model load ~2 min. See [docs/strata.md](docs/strata.md#benchmarks).
|
||||
Current numbers (standard benchmark, idle): prompt ~1,750 tok/s at 4K and ~2,900 tok/s at 16K, decode
|
||||
~110–131 tok/s, model load ~2 min. See [benchmarks/standard](benchmarks/standard/README.md).
|
||||
|
||||
## Repo layout
|
||||
|
||||
@@ -74,7 +74,7 @@ memory-channel fault, the 4-DIMM instability on the Z790, and the pi auto-compac
|
||||
|
||||
## Next up
|
||||
|
||||
- Strata tuning: `--prefill auto:32768`, `--pipeline-windows 2`, `--adapt-async 1` (community 2×3090
|
||||
reports ~1,800–2,300 prompt tok/s vs our 915).
|
||||
- Strata tuning: the three documented flags gave nothing ([results](benchmarks/standard/README.md)).
|
||||
Next levers: huge pages for the expert arena, a higher power cap for prompt speed.
|
||||
- Try other quants.
|
||||
- Planned services as OCI LXCs on the host: SearXNG + degoog, Qdrant, Open WebUI, Tika, Whisper.
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
# <user> standard benchmark: NVIDIA GeForce RTX 3090
|
||||
|
||||
Protocol `strata-qwen38-thinking-v1-draft.1` (draft), run `20261026T234702Z-0f07e24d`, started 2026-10-26T23:47:02Z. **Status: complete.**
|
||||
|
||||
Written by `<user> bench run`. The machine-readable result is [results.json](results.json); every request, its raw usage and timings and the client event offsets are in [outputs.jsonl](outputs.jsonl).
|
||||
|
||||
## Hardware and software
|
||||
|
||||
- GPU: NVIDIA GeForce RTX 3090 (24.0 GiB dedicated)
|
||||
- CPU: Intel(R) Core(TM) i9-14900KS; RAM: 110.0 GiB
|
||||
- Storage of the model files: not measured
|
||||
- OS: NixOS 26.05 (Yarara) 6.18.55; drivers: gpu-driver 595.71.05; runtime: cuda 13.2
|
||||
- <user> commit: `4b9e05a2a806a3fe24f7b2560d98626c37dade35`; engine 0.1.51; source build
|
||||
- Background work: not measured
|
||||
|
||||
## Model and configuration
|
||||
|
||||
- Model: qwen3.8-flash-next-iq3_s; quantization IQ3_S; revision not measured; source revision not measured
|
||||
- Files: `Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf` `4c1eb2ceb4915e11...`; `Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf` `316b46f3a2dbd68c...`
|
||||
- Track: **default** (lovey 2x3090, 250 W cap, gpu-unlock, setup defaults + --max-context 131072)
|
||||
- Context 131072 tokens; server capacity 1 slot(s)
|
||||
- Sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0, repetition_penalty 1.0, reasoning_effort xhigh; thinking on, preserved
|
||||
- Engine options: `config.json`; chat template: `provenance/chat_template.jinja`
|
||||
|
||||
## Method
|
||||
|
||||
The frozen requests of `bench/prompts/standard-v1` (corpus `e75eff8b9feea04f...`) were sent over loopback HTTP, streaming, one at a time, in this order: 1 warm-up (excluded), 3 repetitions of the short, 4K and 16K prompts, and the twelve sanity checks. Prompt and decode speed are the engine's own `timings`; time to first token is the client's monotonic clock from dispatch to the first nonempty reasoning, answer or tool-call delta. No request was retried. Time after the model loaded: 1 min 31 s.
|
||||
|
||||
## Results
|
||||
|
||||
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| short | n/a | 130.7 (124.1-135.6) n=3 | 0.35 (0.34-0.41) n=3 | 1.17 (0.89-2.55) n=3 | 4.2 (3.1-4.2) n=3 | 1/3 |
|
||||
| prompt-4k | 1776.1 (1774.2-1778.0) n=3 | 123.2 (112.6-128.4) n=3 | 2.32 (2.32-2.33) n=3 | 5.30 (4.96-5.64) n=2 | 6.5 (6.3-6.9) n=3 | 3/3 |
|
||||
| prompt-16k | 2937.9 (2934.5-2945.0) n=3 | 124.5 (115.0-131.4) n=3 | 5.61 (5.59-5.61) n=3 | 7.97 (7.79-8.14) n=2 | 9.7 (9.5-10.1) n=3 | 3/3 |
|
||||
|
||||
Cells are median (min-max) n=samples. Short is a prompt of about 80 tokens: its prompt rate is dominated by the fixed cost of a request, so it is not shown; read prompt speed from prompt-4k and prompt-16k. Capped = requests that reached the 512-token cap, reasoning and answer together. A capped request's total time is the time to the cap, not to an answer, and it has no first-answer time when it was still reasoning.
|
||||
|
||||
| Phase | Planned | Completed | Truncated | Failed | Skipped |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| warmup | 1 | 0 | 1 | 0 | 0 |
|
||||
| performance | 9 | 2 | 7 | 0 | 0 |
|
||||
| sanity | 12 | 12 | 0 | 0 | 0 |
|
||||
| concurrency | 12 | 3 | 9 | 0 | 0 |
|
||||
|
||||
Sanity checks: **12 of 12**.
|
||||
|
||||
| Case | Result | Reason |
|
||||
| --- | --- | --- |
|
||||
| I1 | pass | the expected JSON value |
|
||||
| I2 | pass | the expected JSON value |
|
||||
| I3 | pass | the expected JSON value |
|
||||
| I4 | pass | the exact expected text |
|
||||
| R1 | pass | the exact expected text |
|
||||
| R2 | pass | the exact expected text |
|
||||
| R3 | pass | the exact expected text |
|
||||
| R4 | pass | the expected JSON value |
|
||||
| T1 | pass | the expected tool call |
|
||||
| T2 | pass | the expected tool call |
|
||||
| T3 | pass | the expected tool call |
|
||||
| T4 | pass | the expected tool call |
|
||||
|
||||
Concurrency rounds (four requests released together):
|
||||
|
||||
| Round | Wall s | Output tokens | Aggregate tok/s | Failed |
|
||||
| ---: | ---: | ---: | ---: | ---: |
|
||||
| 1 | 16.25344 | 1877 | 115.483245 | 0 |
|
||||
| 2 | 15.310917 | 1937 | 126.511038 | 0 |
|
||||
| 3 | 16.230166 | 2048 | 126.184785 | 0 |
|
||||
|
||||
## Failures, exclusions and limitations
|
||||
|
||||
- No request failed.
|
||||
- Excluded from speed summaries: performance-prompt-16k-r3: first_answer_s is not available for this request
|
||||
- Excluded from speed summaries: performance-prompt-4k-r1: first_answer_s is not available for this request
|
||||
- Prompt-prefix reuse is the shared template prefix only; the cache was not emptied between requests, and the expert cache stays warm and evolves. These are not cold-model measurements.
|
||||
- Output caps include reasoning; a request that reaches its cap is `truncated` and fails its sanity check.
|
||||
- The runner cannot see other programs on the machine, the repeat-stop and loop-recovery switches of the server, or the quantization repository's revision; see `missing` in results.json for every value it could not read.
|
||||
- The server had answered 1122 request(s) before this run.
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"engine_options": null,
|
||||
"other_engine_options": {},
|
||||
"server_defaults": {
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
"top_k": 20,
|
||||
"n_predict": -1
|
||||
},
|
||||
"total_slots": 1,
|
||||
"context": 131072,
|
||||
"source": "the server was already running; its engine options are not exposed"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
+184
@@ -0,0 +1,184 @@
|
||||
{%- set image_count = namespace(value=0) %}
|
||||
{%- set video_count = namespace(value=0) %}
|
||||
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
||||
{%- if content is string %}
|
||||
{{- content }}
|
||||
{%- elif content is iterable and content is not mapping %}
|
||||
{%- for item in content %}
|
||||
{%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain images.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set image_count.value = image_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
||||
{%- elif 'video' in item or item.type == 'video' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain videos.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set video_count.value = video_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
||||
{%- elif 'text' in item %}
|
||||
{{- item.text }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected item type in content.') }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- elif content is none or content is undefined %}
|
||||
{{- '' }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected content type.') }}
|
||||
{%- endif %}
|
||||
{%- endmacro %}
|
||||
{%- if not messages %}
|
||||
{{- raise_exception('No messages provided.') }}
|
||||
{%- endif %}
|
||||
{%- set sysns = namespace(count=0, text='') %}
|
||||
{%- for message in messages %}
|
||||
{%- if sysns.count == loop.index0 and (message.role == 'system' or message.role == 'developer') %}
|
||||
{%- set sys_content = render_content(message.content, false, true)|trim %}
|
||||
{%- if sys_content %}
|
||||
{%- set sysns.text = sysns.text + ('\n' if sysns.text else '') + sys_content %}
|
||||
{%- endif %}
|
||||
{%- set sysns.count = sysns.count + 1 %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set num_sys = sysns.count %}
|
||||
{%- set merged_system = sysns.text %}
|
||||
{%- set reasoning_instructions = '' %}
|
||||
{%- if enable_thinking is undefined or enable_thinking is true %}
|
||||
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
|
||||
{%- if resolved_reasoning_effort == 'high' %}
|
||||
{%- set resolved_reasoning_effort = 'xhigh' %}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
|
||||
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort == 'xhigh' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
|
||||
{%- elif resolved_reasoning_effort == 'low' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if tools and tools is iterable and tools is not mapping %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if reasoning_instructions %}
|
||||
{{- reasoning_instructions + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
|
||||
{%- for tool in tools %}
|
||||
{{- "\n" }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- "\n</tools>" }}
|
||||
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
||||
{%- if merged_system %}
|
||||
{{- '\n\n' + merged_system }}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{%- if merged_system %}
|
||||
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + merged_system + '<|im_end|>\n' }}
|
||||
{%- elif reasoning_instructions %}
|
||||
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
||||
{%- for message in messages[::-1] %}
|
||||
{%- set index = (messages|length - 1) - loop.index0 %}
|
||||
{%- if ns.multi_step_tool and message.role == "user" %}
|
||||
{%- set content = render_content(message.content, false)|trim %}
|
||||
{%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
|
||||
{%- set ns.multi_step_tool = false %}
|
||||
{%- set ns.last_query_index = index %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- for message in messages %}
|
||||
{%- if loop.index0 >= num_sys %}
|
||||
{%- set content = render_content(message.content, true)|trim %}
|
||||
{%- if message.role == "system" or message.role == "developer" %}
|
||||
{{- raise_exception('System message must be at the beginning.') }}
|
||||
{%- elif message.role == "user" %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||
{%- elif message.role == "assistant" %}
|
||||
{%- set reasoning_content = '' %}
|
||||
{%- if message.reasoning_content is string %}
|
||||
{%- set reasoning_content = message.reasoning_content %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = reasoning_content|trim %}
|
||||
{%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
|
||||
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||
{%- endif %}
|
||||
{%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if tool_call.function is defined %}
|
||||
{%- set tool_call = tool_call.function %}
|
||||
{%- endif %}
|
||||
{%- if tool_call.name is not defined or tool_call.name is none %}
|
||||
{{- raise_exception('Tool call is missing a function name.') }}
|
||||
{%- endif %}
|
||||
{%- if loop.first %}
|
||||
{%- if content|trim %}
|
||||
{{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- else %}
|
||||
{{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- if tool_call.arguments is mapping %}
|
||||
{%- for args_name, args_value in tool_call.arguments|items %}
|
||||
{{- '<parameter=' + args_name + '>\n' }}
|
||||
{%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
|
||||
{{- args_value }}
|
||||
{{- '\n</parameter>\n' }}
|
||||
{%- endfor %}
|
||||
{%- elif tool_call.arguments is string %}
|
||||
{%- if tool_call.arguments|trim %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" were passed as a JSON string. Parse them into an object before calling apply_chat_template.') }}
|
||||
{%- endif %}
|
||||
{%- elif tool_call.arguments is defined and tool_call.arguments is not none %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" must be an object/mapping or a JSON string.') }}
|
||||
{%- endif %}
|
||||
{{- '</function>\n</tool_call>' }}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == "tool" %}
|
||||
{%- if loop.previtem and loop.previtem.role != "tool" %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' }}
|
||||
{{- content }}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if not loop.last and loop.nextitem.role != "tool" %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif loop.last %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected message role.') }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- if enable_thinking is defined and enable_thinking is false %}
|
||||
{{- '<think>\n\n</think>\n\n' }}
|
||||
{%- else %}
|
||||
{{- '<think>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{#- Unsloth fixes - developer role, merged system messages, tool calling #}
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
{
|
||||
"protocol_id": "strata-qwen38-thinking-v1-draft.1",
|
||||
"bundle": "standard-v1",
|
||||
"profile": "qwen3.8-flash-next-thinking",
|
||||
"tokenizer_files": {
|
||||
"vocab.json": "4ba64f0332abcfb0b600b7df1537d1e836e68009d5fb7cdb77339738dc6365c4",
|
||||
"merges.txt": "080cf95432173729d160346b7882bd2a644b07f7a3813a3eb73f4e66787bfde9",
|
||||
"token_type.json": "5088c8c298fc06af8382ddb3b76c888703ac2634e82263b97eedcc2ad202738b"
|
||||
},
|
||||
"template": {
|
||||
"path": "serve/chat_template.jinja",
|
||||
"sha256": "ae9644304511ef71f28e036b6f65b4c709880bc5a23b7d0b42125ec63a8c70ca"
|
||||
},
|
||||
"template_kwargs": {
|
||||
"enable_thinking": true,
|
||||
"preserve_thinking": true,
|
||||
"reasoning_effort": "xhigh"
|
||||
},
|
||||
"record_format": "record{:06d}: alpha beta gamma delta epsilon zeta eta theta.",
|
||||
"short_task": "Explain how a hash table finds a value, including collisions and one small example.",
|
||||
"summary_question": "Summarize the structure of these records and explain how to look up a record by its ID.",
|
||||
"record_counts": {
|
||||
"prompt-4k": {
|
||||
"records": 211,
|
||||
"target_tokens": 4096
|
||||
},
|
||||
"prompt-16k": {
|
||||
"records": 858,
|
||||
"target_tokens": 16384
|
||||
}
|
||||
},
|
||||
"length_tolerance": 0.02,
|
||||
"requests": [
|
||||
{
|
||||
"id": "warmup",
|
||||
"input_tokens": 75,
|
||||
"max_shared_prefix_tokens": 47
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r1",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r1",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r1",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r2",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r2",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r2",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r3",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r3",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r3",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "sanity-I1",
|
||||
"input_tokens": 82,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I2",
|
||||
"input_tokens": 90,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I3",
|
||||
"input_tokens": 85,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I4",
|
||||
"input_tokens": 71,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R1",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R2",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R3",
|
||||
"input_tokens": 16382,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R4",
|
||||
"input_tokens": 16390,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-T1",
|
||||
"input_tokens": 325,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T2",
|
||||
"input_tokens": 335,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "sanity-T3",
|
||||
"input_tokens": 352,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T4",
|
||||
"input_tokens": 339,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
}
|
||||
],
|
||||
"note": "input_tokens counts the rendered thinking request with the generation prompt. max_shared_prefix_tokens is the longest token prefix this request shares with any other request of the bundle: cache reuse above it means the user payload was read from the cache."
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
bc6c1f81d86863c3dc36a31f58010701d8ab05180f69f86c6978214409e43af2 build.json
|
||||
f724855774dfbea89fade712a70d534a085c88cda6abc123db419fd79eefab49 requests/concurrency-r1-c1.json
|
||||
2c85f877abbf144ca065c58efc2a7568aaf2c8dc317ad8266e78c149f8245abb requests/concurrency-r1-c2.json
|
||||
264097f61645a0d5089562dc003b7748b1c20339c2df1290ad6664fc1834068a requests/concurrency-r1-c3.json
|
||||
65895c5a5fab53c06e9358d88d921f1bfc9dc7cc8b6e8a2521ac90de080ff470 requests/concurrency-r1-c4.json
|
||||
2c08d50e0c95e345512ec918f58fa716780b79900f09bbfabc37519c8d86543a requests/concurrency-r2-c1.json
|
||||
8aba49693701b9315b2bc5ad55e97a8b03496baedd05f370c49c2c77f45ce6cd requests/concurrency-r2-c2.json
|
||||
7cfb4a5a8fb5702244ce42e0dba9ccd3064d4023d700b409a1a472790a82809c requests/concurrency-r2-c3.json
|
||||
3ab56fbba5fd405945e53cff01b59c3ef966b0f08a0a1b0b50633d26e90cad62 requests/concurrency-r2-c4.json
|
||||
3adb684cf2cb84d4e514de30a4c6f8ccebdd8471160cfcdfc0cdb8ca19ac2137 requests/concurrency-r3-c1.json
|
||||
650379ef9a4da667f6581ae9573eb9af2db2a1fc617701e50536590b18f70077 requests/concurrency-r3-c2.json
|
||||
111415388c5e69ecbd9ab66efe85eb8f3db9142201e1548d03203d3acc05eca1 requests/concurrency-r3-c3.json
|
||||
691a8d27fd06e9beab57e6efb3d82c2a7ed9fc26635fd51b1b1d017117440c1e requests/concurrency-r3-c4.json
|
||||
06776ef38485dc4bb7e18809534fe16a57d421d163f81d4d1532c4b08225968e requests/performance-prompt-16k-r1.json
|
||||
4995c5f3c47c1682f4dc753a9f3a1fba6920fea14691e1ded2da6208ef5fe628 requests/performance-prompt-16k-r2.json
|
||||
7af4c49f99192007be27110296b5a0bcc2efbc91c7ff1c337a6cac6732050559 requests/performance-prompt-16k-r3.json
|
||||
bda0e60c2de0703899ac1bd5711d793b9b325c7f8ba6be6c967f22dd07d8dda8 requests/performance-prompt-4k-r1.json
|
||||
e898eec0ee22499473179a719505c9feb9d3772658833f607dffb34c3727150e requests/performance-prompt-4k-r2.json
|
||||
6d3b3300eace318574490f4379119409920cbe7e88fdaff28eb6f5c79430c3e3 requests/performance-prompt-4k-r3.json
|
||||
483b4f18fcc64ca802fde3377c59bf7b114ec5a5c96fe781bd2be4665ef7b898 requests/performance-short-r1.json
|
||||
b0ec85f40117303d19c8c17a4813625c8554f59caddbe69e7d8aafdcf51d501a requests/performance-short-r2.json
|
||||
e210da20b80c61147da5102fcba65131dafbbb4ceee952fa6b91ce2bb06581ed requests/performance-short-r3.json
|
||||
70ebcd31af5ae1bc7573136d9a257469d8ad9e85b5d0233980998e957c36d6c2 requests/sanity-I1.json
|
||||
1cc6e4bc56b5a22043b6e384b3036170dc6d3edeb403be391a8b802b5613f4cb requests/sanity-I2.json
|
||||
c29b13e6e31b674ff733202b3c0b1f9410768b9ea9368a4a6a13430b3e011fb8 requests/sanity-I3.json
|
||||
780fbd1d02f119323cf50ac4dabcfffbd23dd84115efbe18152e7ee9112b041a requests/sanity-I4.json
|
||||
3c1e0718ad06e591e47cf8df9c36bab2d13332d44b05e440ec17d0402bf9facc requests/sanity-R1.json
|
||||
352e054c48e08e11d339d3a02a14f4f50125071710679b8d04b77aba711fbafc requests/sanity-R2.json
|
||||
36770434f1ea046ec84b32691d644e8416e89fbcbb04b8f6032da66016d78291 requests/sanity-R3.json
|
||||
a63df33c784732cc6b4d8a0f2258b7226c8df5a3b4f6818f9e886bbf09604999 requests/sanity-R4.json
|
||||
6b272ef684b43c1e0004a40b958c807b84dcd9de579d4b2d8819e39ae4196fb2 requests/sanity-T1.json
|
||||
8253ee4aa6b19e61e28d64406807218c9aa10cf2cf682101e8a2d6a50564de46 requests/sanity-T2.json
|
||||
aa2be8761793f51eab6ef04bc54c96e018f6981b7a6a26b9e50cc3c7ac856803 requests/sanity-T3.json
|
||||
04043bdec397df487bebd2999e280a9fae5925e5efec1a10fa96055ec3430faa requests/sanity-T4.json
|
||||
d3365c57db8be1fd0dd9771f74f264b1f6994a396e472c3e5fa4ede2641ffb5c requests/warmup.json
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,75 @@
|
||||
# <user> standard benchmark: NVIDIA GeForce RTX 3090
|
||||
|
||||
Protocol `strata-qwen38-thinking-v1-draft.1` (draft), run `20261026T235539Z-1ecae1bc`, started 2026-10-26T23:55:39Z. **Status: nonconforming.**
|
||||
|
||||
Written by `<user> bench run`. The machine-readable result is [results.json](results.json); every request, its raw usage and timings and the client event offsets are in [outputs.jsonl](outputs.jsonl).
|
||||
|
||||
This run is not a complete standard result:
|
||||
|
||||
- no model file could be hashed
|
||||
|
||||
## Hardware and software
|
||||
|
||||
- GPU: NVIDIA GeForce RTX 3090 (24.0 GiB dedicated)
|
||||
- CPU: Intel(R) Core(TM) i9-14900KS; RAM: 110.0 GiB
|
||||
- Storage of the model files: not measured
|
||||
- OS: NixOS 26.05 (Yarara) 6.18.55; drivers: gpu-driver 595.71.05; runtime: cuda 13.2
|
||||
- <user> commit: `4b9e05a2a806a3fe24f7b2560d98626c37dade35`; engine 0.1.51; source build
|
||||
- Background work: not measured
|
||||
|
||||
## Model and configuration
|
||||
|
||||
- Model: qwen3.8-flash-next-iq3_s; quantization IQ3_S; revision not measured; source revision not measured
|
||||
- Files: not measured
|
||||
- Track: **custom** (arm adapt-async: ["--adapt-async","1"])
|
||||
- Context 131072 tokens; server capacity 1 slot(s)
|
||||
- Sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0, repetition_penalty 1.0, reasoning_effort xhigh; thinking on, preserved
|
||||
- Engine options: `config.json`; chat template: `provenance/chat_template.jinja`
|
||||
|
||||
## Method
|
||||
|
||||
The frozen requests of `bench/prompts/standard-v1` (corpus `e75eff8b9feea04f...`) were sent over loopback HTTP, streaming, one at a time, in this order: 1 warm-up (excluded), 3 repetitions of the short, 4K and 16K prompts, and the twelve sanity checks. Prompt and decode speed are the engine's own `timings`; time to first token is the client's monotonic clock from dispatch to the first nonempty reasoning, answer or tool-call delta. No request was retried. Time after the model loaded: 1 min 35 s.
|
||||
|
||||
## Results
|
||||
|
||||
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| short | n/a | 125.1 (123.6-129.8) n=3 | 0.46 (0.38-0.48) n=3 | 1.26 (0.88-1.64) n=3 | 4.0 (3.6-4.2) n=3 | 0/3 |
|
||||
| prompt-4k | 1711.7 (1548.2-1748.8) n=3 | 115.0 (105.0-118.2) n=3 | 2.41 (2.36-2.66) n=3 | 5.82 (5.58-6.07) n=2 | 7.0 (6.8-7.3) n=3 | 3/3 |
|
||||
| prompt-16k | 2870.3 (2837.4-2870.6) n=3 | 108.6 (99.3-130.6) n=3 | 5.73 (5.73-5.81) n=3 | 8.77 (7.39-10.16) n=2 | 10.4 (9.6-11.0) n=3 | 3/3 |
|
||||
|
||||
Cells are median (min-max) n=samples. Short is a prompt of about 80 tokens: its prompt rate is dominated by the fixed cost of a request, so it is not shown; read prompt speed from prompt-4k and prompt-16k. Capped = requests that reached the 512-token cap, reasoning and answer together. A capped request's total time is the time to the cap, not to an answer, and it has no first-answer time when it was still reasoning.
|
||||
|
||||
| Phase | Planned | Completed | Truncated | Failed | Skipped |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| warmup | 1 | 0 | 1 | 0 | 0 |
|
||||
| performance | 9 | 3 | 6 | 0 | 0 |
|
||||
| sanity | 12 | 12 | 0 | 0 | 0 |
|
||||
| concurrency | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
Sanity checks: **12 of 12**.
|
||||
|
||||
| Case | Result | Reason |
|
||||
| --- | --- | --- |
|
||||
| I1 | pass | the expected JSON value |
|
||||
| I2 | pass | the expected JSON value |
|
||||
| I3 | pass | the expected JSON value |
|
||||
| I4 | pass | the exact expected text |
|
||||
| R1 | pass | the exact expected text |
|
||||
| R2 | pass | the exact expected text |
|
||||
| R3 | pass | the exact expected text |
|
||||
| R4 | pass | the expected JSON value |
|
||||
| T1 | pass | the expected tool call |
|
||||
| T2 | pass | the expected tool call |
|
||||
| T3 | pass | the expected tool call |
|
||||
| T4 | pass | the expected tool call |
|
||||
|
||||
## Failures, exclusions and limitations
|
||||
|
||||
- No request failed.
|
||||
- Excluded from speed summaries: performance-prompt-16k-r1: first_answer_s is not available for this request
|
||||
- Excluded from speed summaries: performance-prompt-4k-r2: first_answer_s is not available for this request
|
||||
- Prompt-prefix reuse is the shared template prefix only; the cache was not emptied between requests, and the expert cache stays warm and evolves. These are not cold-model measurements.
|
||||
- Output caps include reasoning; a request that reaches its cap is `truncated` and fails its sanity check.
|
||||
- The runner cannot see other programs on the machine, the repeat-stop and loop-recovery switches of the server, or the quantization repository's revision; see `missing` in results.json for every value it could not read.
|
||||
- The server had answered no request before this run (a freshly loaded engine).
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"engine_options": null,
|
||||
"other_engine_options": {},
|
||||
"server_defaults": {
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
"top_k": 20,
|
||||
"n_predict": -1
|
||||
},
|
||||
"total_slots": 1,
|
||||
"context": 131072,
|
||||
"source": "the server was already running; its engine options are not exposed"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
+184
@@ -0,0 +1,184 @@
|
||||
{%- set image_count = namespace(value=0) %}
|
||||
{%- set video_count = namespace(value=0) %}
|
||||
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
||||
{%- if content is string %}
|
||||
{{- content }}
|
||||
{%- elif content is iterable and content is not mapping %}
|
||||
{%- for item in content %}
|
||||
{%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain images.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set image_count.value = image_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
||||
{%- elif 'video' in item or item.type == 'video' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain videos.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set video_count.value = video_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
||||
{%- elif 'text' in item %}
|
||||
{{- item.text }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected item type in content.') }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- elif content is none or content is undefined %}
|
||||
{{- '' }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected content type.') }}
|
||||
{%- endif %}
|
||||
{%- endmacro %}
|
||||
{%- if not messages %}
|
||||
{{- raise_exception('No messages provided.') }}
|
||||
{%- endif %}
|
||||
{%- set sysns = namespace(count=0, text='') %}
|
||||
{%- for message in messages %}
|
||||
{%- if sysns.count == loop.index0 and (message.role == 'system' or message.role == 'developer') %}
|
||||
{%- set sys_content = render_content(message.content, false, true)|trim %}
|
||||
{%- if sys_content %}
|
||||
{%- set sysns.text = sysns.text + ('\n' if sysns.text else '') + sys_content %}
|
||||
{%- endif %}
|
||||
{%- set sysns.count = sysns.count + 1 %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set num_sys = sysns.count %}
|
||||
{%- set merged_system = sysns.text %}
|
||||
{%- set reasoning_instructions = '' %}
|
||||
{%- if enable_thinking is undefined or enable_thinking is true %}
|
||||
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
|
||||
{%- if resolved_reasoning_effort == 'high' %}
|
||||
{%- set resolved_reasoning_effort = 'xhigh' %}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
|
||||
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort == 'xhigh' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
|
||||
{%- elif resolved_reasoning_effort == 'low' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if tools and tools is iterable and tools is not mapping %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if reasoning_instructions %}
|
||||
{{- reasoning_instructions + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
|
||||
{%- for tool in tools %}
|
||||
{{- "\n" }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- "\n</tools>" }}
|
||||
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
||||
{%- if merged_system %}
|
||||
{{- '\n\n' + merged_system }}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{%- if merged_system %}
|
||||
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + merged_system + '<|im_end|>\n' }}
|
||||
{%- elif reasoning_instructions %}
|
||||
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
||||
{%- for message in messages[::-1] %}
|
||||
{%- set index = (messages|length - 1) - loop.index0 %}
|
||||
{%- if ns.multi_step_tool and message.role == "user" %}
|
||||
{%- set content = render_content(message.content, false)|trim %}
|
||||
{%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
|
||||
{%- set ns.multi_step_tool = false %}
|
||||
{%- set ns.last_query_index = index %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- for message in messages %}
|
||||
{%- if loop.index0 >= num_sys %}
|
||||
{%- set content = render_content(message.content, true)|trim %}
|
||||
{%- if message.role == "system" or message.role == "developer" %}
|
||||
{{- raise_exception('System message must be at the beginning.') }}
|
||||
{%- elif message.role == "user" %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||
{%- elif message.role == "assistant" %}
|
||||
{%- set reasoning_content = '' %}
|
||||
{%- if message.reasoning_content is string %}
|
||||
{%- set reasoning_content = message.reasoning_content %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = reasoning_content|trim %}
|
||||
{%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
|
||||
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||
{%- endif %}
|
||||
{%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if tool_call.function is defined %}
|
||||
{%- set tool_call = tool_call.function %}
|
||||
{%- endif %}
|
||||
{%- if tool_call.name is not defined or tool_call.name is none %}
|
||||
{{- raise_exception('Tool call is missing a function name.') }}
|
||||
{%- endif %}
|
||||
{%- if loop.first %}
|
||||
{%- if content|trim %}
|
||||
{{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- else %}
|
||||
{{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- if tool_call.arguments is mapping %}
|
||||
{%- for args_name, args_value in tool_call.arguments|items %}
|
||||
{{- '<parameter=' + args_name + '>\n' }}
|
||||
{%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
|
||||
{{- args_value }}
|
||||
{{- '\n</parameter>\n' }}
|
||||
{%- endfor %}
|
||||
{%- elif tool_call.arguments is string %}
|
||||
{%- if tool_call.arguments|trim %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" were passed as a JSON string. Parse them into an object before calling apply_chat_template.') }}
|
||||
{%- endif %}
|
||||
{%- elif tool_call.arguments is defined and tool_call.arguments is not none %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" must be an object/mapping or a JSON string.') }}
|
||||
{%- endif %}
|
||||
{{- '</function>\n</tool_call>' }}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == "tool" %}
|
||||
{%- if loop.previtem and loop.previtem.role != "tool" %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' }}
|
||||
{{- content }}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if not loop.last and loop.nextitem.role != "tool" %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif loop.last %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected message role.') }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- if enable_thinking is defined and enable_thinking is false %}
|
||||
{{- '<think>\n\n</think>\n\n' }}
|
||||
{%- else %}
|
||||
{{- '<think>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{#- Unsloth fixes - developer role, merged system messages, tool calling #}
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
{
|
||||
"protocol_id": "strata-qwen38-thinking-v1-draft.1",
|
||||
"bundle": "standard-v1",
|
||||
"profile": "qwen3.8-flash-next-thinking",
|
||||
"tokenizer_files": {
|
||||
"vocab.json": "4ba64f0332abcfb0b600b7df1537d1e836e68009d5fb7cdb77339738dc6365c4",
|
||||
"merges.txt": "080cf95432173729d160346b7882bd2a644b07f7a3813a3eb73f4e66787bfde9",
|
||||
"token_type.json": "5088c8c298fc06af8382ddb3b76c888703ac2634e82263b97eedcc2ad202738b"
|
||||
},
|
||||
"template": {
|
||||
"path": "serve/chat_template.jinja",
|
||||
"sha256": "ae9644304511ef71f28e036b6f65b4c709880bc5a23b7d0b42125ec63a8c70ca"
|
||||
},
|
||||
"template_kwargs": {
|
||||
"enable_thinking": true,
|
||||
"preserve_thinking": true,
|
||||
"reasoning_effort": "xhigh"
|
||||
},
|
||||
"record_format": "record{:06d}: alpha beta gamma delta epsilon zeta eta theta.",
|
||||
"short_task": "Explain how a hash table finds a value, including collisions and one small example.",
|
||||
"summary_question": "Summarize the structure of these records and explain how to look up a record by its ID.",
|
||||
"record_counts": {
|
||||
"prompt-4k": {
|
||||
"records": 211,
|
||||
"target_tokens": 4096
|
||||
},
|
||||
"prompt-16k": {
|
||||
"records": 858,
|
||||
"target_tokens": 16384
|
||||
}
|
||||
},
|
||||
"length_tolerance": 0.02,
|
||||
"requests": [
|
||||
{
|
||||
"id": "warmup",
|
||||
"input_tokens": 75,
|
||||
"max_shared_prefix_tokens": 47
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r1",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r1",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r1",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r2",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r2",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r2",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r3",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r3",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r3",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "sanity-I1",
|
||||
"input_tokens": 82,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I2",
|
||||
"input_tokens": 90,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I3",
|
||||
"input_tokens": 85,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I4",
|
||||
"input_tokens": 71,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R1",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R2",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R3",
|
||||
"input_tokens": 16382,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R4",
|
||||
"input_tokens": 16390,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-T1",
|
||||
"input_tokens": 325,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T2",
|
||||
"input_tokens": 335,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "sanity-T3",
|
||||
"input_tokens": 352,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T4",
|
||||
"input_tokens": 339,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
}
|
||||
],
|
||||
"note": "input_tokens counts the rendered thinking request with the generation prompt. max_shared_prefix_tokens is the longest token prefix this request shares with any other request of the bundle: cache reuse above it means the user payload was read from the cache."
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
bc6c1f81d86863c3dc36a31f58010701d8ab05180f69f86c6978214409e43af2 build.json
|
||||
f724855774dfbea89fade712a70d534a085c88cda6abc123db419fd79eefab49 requests/concurrency-r1-c1.json
|
||||
2c85f877abbf144ca065c58efc2a7568aaf2c8dc317ad8266e78c149f8245abb requests/concurrency-r1-c2.json
|
||||
264097f61645a0d5089562dc003b7748b1c20339c2df1290ad6664fc1834068a requests/concurrency-r1-c3.json
|
||||
65895c5a5fab53c06e9358d88d921f1bfc9dc7cc8b6e8a2521ac90de080ff470 requests/concurrency-r1-c4.json
|
||||
2c08d50e0c95e345512ec918f58fa716780b79900f09bbfabc37519c8d86543a requests/concurrency-r2-c1.json
|
||||
8aba49693701b9315b2bc5ad55e97a8b03496baedd05f370c49c2c77f45ce6cd requests/concurrency-r2-c2.json
|
||||
7cfb4a5a8fb5702244ce42e0dba9ccd3064d4023d700b409a1a472790a82809c requests/concurrency-r2-c3.json
|
||||
3ab56fbba5fd405945e53cff01b59c3ef966b0f08a0a1b0b50633d26e90cad62 requests/concurrency-r2-c4.json
|
||||
3adb684cf2cb84d4e514de30a4c6f8ccebdd8471160cfcdfc0cdb8ca19ac2137 requests/concurrency-r3-c1.json
|
||||
650379ef9a4da667f6581ae9573eb9af2db2a1fc617701e50536590b18f70077 requests/concurrency-r3-c2.json
|
||||
111415388c5e69ecbd9ab66efe85eb8f3db9142201e1548d03203d3acc05eca1 requests/concurrency-r3-c3.json
|
||||
691a8d27fd06e9beab57e6efb3d82c2a7ed9fc26635fd51b1b1d017117440c1e requests/concurrency-r3-c4.json
|
||||
06776ef38485dc4bb7e18809534fe16a57d421d163f81d4d1532c4b08225968e requests/performance-prompt-16k-r1.json
|
||||
4995c5f3c47c1682f4dc753a9f3a1fba6920fea14691e1ded2da6208ef5fe628 requests/performance-prompt-16k-r2.json
|
||||
7af4c49f99192007be27110296b5a0bcc2efbc91c7ff1c337a6cac6732050559 requests/performance-prompt-16k-r3.json
|
||||
bda0e60c2de0703899ac1bd5711d793b9b325c7f8ba6be6c967f22dd07d8dda8 requests/performance-prompt-4k-r1.json
|
||||
e898eec0ee22499473179a719505c9feb9d3772658833f607dffb34c3727150e requests/performance-prompt-4k-r2.json
|
||||
6d3b3300eace318574490f4379119409920cbe7e88fdaff28eb6f5c79430c3e3 requests/performance-prompt-4k-r3.json
|
||||
483b4f18fcc64ca802fde3377c59bf7b114ec5a5c96fe781bd2be4665ef7b898 requests/performance-short-r1.json
|
||||
b0ec85f40117303d19c8c17a4813625c8554f59caddbe69e7d8aafdcf51d501a requests/performance-short-r2.json
|
||||
e210da20b80c61147da5102fcba65131dafbbb4ceee952fa6b91ce2bb06581ed requests/performance-short-r3.json
|
||||
70ebcd31af5ae1bc7573136d9a257469d8ad9e85b5d0233980998e957c36d6c2 requests/sanity-I1.json
|
||||
1cc6e4bc56b5a22043b6e384b3036170dc6d3edeb403be391a8b802b5613f4cb requests/sanity-I2.json
|
||||
c29b13e6e31b674ff733202b3c0b1f9410768b9ea9368a4a6a13430b3e011fb8 requests/sanity-I3.json
|
||||
780fbd1d02f119323cf50ac4dabcfffbd23dd84115efbe18152e7ee9112b041a requests/sanity-I4.json
|
||||
3c1e0718ad06e591e47cf8df9c36bab2d13332d44b05e440ec17d0402bf9facc requests/sanity-R1.json
|
||||
352e054c48e08e11d339d3a02a14f4f50125071710679b8d04b77aba711fbafc requests/sanity-R2.json
|
||||
36770434f1ea046ec84b32691d644e8416e89fbcbb04b8f6032da66016d78291 requests/sanity-R3.json
|
||||
a63df33c784732cc6b4d8a0f2258b7226c8df5a3b4f6818f9e886bbf09604999 requests/sanity-R4.json
|
||||
6b272ef684b43c1e0004a40b958c807b84dcd9de579d4b2d8819e39ae4196fb2 requests/sanity-T1.json
|
||||
8253ee4aa6b19e61e28d64406807218c9aa10cf2cf682101e8a2d6a50564de46 requests/sanity-T2.json
|
||||
aa2be8761793f51eab6ef04bc54c96e018f6981b7a6a26b9e50cc3c7ac856803 requests/sanity-T3.json
|
||||
04043bdec397df487bebd2999e280a9fae5925e5efec1a10fa96055ec3430faa requests/sanity-T4.json
|
||||
d3365c57db8be1fd0dd9771f74f264b1f6994a396e472c3e5fa4ede2641ffb5c requests/warmup.json
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,75 @@
|
||||
# <user> standard benchmark: NVIDIA GeForce RTX 3090
|
||||
|
||||
Protocol `strata-qwen38-thinking-v1-draft.1` (draft), run `20261026T235926Z-5f934909`, started 2026-10-26T23:59:26Z. **Status: nonconforming.**
|
||||
|
||||
Written by `<user> bench run`. The machine-readable result is [results.json](results.json); every request, its raw usage and timings and the client event offsets are in [outputs.jsonl](outputs.jsonl).
|
||||
|
||||
This run is not a complete standard result:
|
||||
|
||||
- no model file could be hashed
|
||||
|
||||
## Hardware and software
|
||||
|
||||
- GPU: NVIDIA GeForce RTX 3090 (24.0 GiB dedicated)
|
||||
- CPU: Intel(R) Core(TM) i9-14900KS; RAM: 110.0 GiB
|
||||
- Storage of the model files: not measured
|
||||
- OS: NixOS 26.05 (Yarara) 6.18.55; drivers: gpu-driver 595.71.05; runtime: cuda 13.2
|
||||
- <user> commit: `4b9e05a2a806a3fe24f7b2560d98626c37dade35`; engine 0.1.51; source build
|
||||
- Background work: not measured
|
||||
|
||||
## Model and configuration
|
||||
|
||||
- Model: qwen3.8-flash-next-iq3_s; quantization IQ3_S; revision not measured; source revision not measured
|
||||
- Files: not measured
|
||||
- Track: **custom** (arm pipeline-windows: ["--pipeline-windows","2"])
|
||||
- Context 131072 tokens; server capacity 1 slot(s)
|
||||
- Sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0, repetition_penalty 1.0, reasoning_effort xhigh; thinking on, preserved
|
||||
- Engine options: `config.json`; chat template: `provenance/chat_template.jinja`
|
||||
|
||||
## Method
|
||||
|
||||
The frozen requests of `bench/prompts/standard-v1` (corpus `e75eff8b9feea04f...`) were sent over loopback HTTP, streaming, one at a time, in this order: 1 warm-up (excluded), 3 repetitions of the short, 4K and 16K prompts, and the twelve sanity checks. Prompt and decode speed are the engine's own `timings`; time to first token is the client's monotonic clock from dispatch to the first nonempty reasoning, answer or tool-call delta. No request was retried. Time after the model loaded: 1 min 37 s.
|
||||
|
||||
## Results
|
||||
|
||||
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| short | n/a | 110.0 (102.5-113.8) n=3 | 0.47 (0.39-0.48) n=3 | 1.26 (0.93-3.65) n=3 | 3.8 (3.7-5.5) n=3 | 1/3 |
|
||||
| prompt-4k | 1705.1 (1540.3-1743.3) n=3 | 104.2 (95.9-134.5) n=3 | 2.42 (2.37-2.68) n=3 | 5.98 (4.03-7.92) n=2 | 7.3 (6.2-8.0) n=3 | 3/3 |
|
||||
| prompt-16k | 2878.4 (2844.7-2883.2) n=3 | 102.7 (100.7-103.3) n=3 | 5.71 (5.71-5.78) n=3 | 10.47 (10.11-10.84) n=2 | 10.7 (10.7-10.9) n=3 | 3/3 |
|
||||
|
||||
Cells are median (min-max) n=samples. Short is a prompt of about 80 tokens: its prompt rate is dominated by the fixed cost of a request, so it is not shown; read prompt speed from prompt-4k and prompt-16k. Capped = requests that reached the 512-token cap, reasoning and answer together. A capped request's total time is the time to the cap, not to an answer, and it has no first-answer time when it was still reasoning.
|
||||
|
||||
| Phase | Planned | Completed | Truncated | Failed | Skipped |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| warmup | 1 | 0 | 1 | 0 | 0 |
|
||||
| performance | 9 | 2 | 7 | 0 | 0 |
|
||||
| sanity | 12 | 12 | 0 | 0 | 0 |
|
||||
| concurrency | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
Sanity checks: **12 of 12**.
|
||||
|
||||
| Case | Result | Reason |
|
||||
| --- | --- | --- |
|
||||
| I1 | pass | the expected JSON value |
|
||||
| I2 | pass | the expected JSON value |
|
||||
| I3 | pass | the expected JSON value |
|
||||
| I4 | pass | the exact expected text |
|
||||
| R1 | pass | the exact expected text |
|
||||
| R2 | pass | the exact expected text |
|
||||
| R3 | pass | the exact expected text |
|
||||
| R4 | pass | the expected JSON value |
|
||||
| T1 | pass | the expected tool call |
|
||||
| T2 | pass | the expected tool call |
|
||||
| T3 | pass | the expected tool call |
|
||||
| T4 | pass | the expected tool call |
|
||||
|
||||
## Failures, exclusions and limitations
|
||||
|
||||
- No request failed.
|
||||
- Excluded from speed summaries: performance-prompt-16k-r3: first_answer_s is not available for this request
|
||||
- Excluded from speed summaries: performance-prompt-4k-r2: first_answer_s is not available for this request
|
||||
- Prompt-prefix reuse is the shared template prefix only; the cache was not emptied between requests, and the expert cache stays warm and evolves. These are not cold-model measurements.
|
||||
- Output caps include reasoning; a request that reaches its cap is `truncated` and fails its sanity check.
|
||||
- The runner cannot see other programs on the machine, the repeat-stop and loop-recovery switches of the server, or the quantization repository's revision; see `missing` in results.json for every value it could not read.
|
||||
- The server had answered no request before this run (a freshly loaded engine).
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"engine_options": null,
|
||||
"other_engine_options": {},
|
||||
"server_defaults": {
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
"top_k": 20,
|
||||
"n_predict": -1
|
||||
},
|
||||
"total_slots": 1,
|
||||
"context": 131072,
|
||||
"source": "the server was already running; its engine options are not exposed"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
+184
@@ -0,0 +1,184 @@
|
||||
{%- set image_count = namespace(value=0) %}
|
||||
{%- set video_count = namespace(value=0) %}
|
||||
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
||||
{%- if content is string %}
|
||||
{{- content }}
|
||||
{%- elif content is iterable and content is not mapping %}
|
||||
{%- for item in content %}
|
||||
{%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain images.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set image_count.value = image_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
||||
{%- elif 'video' in item or item.type == 'video' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain videos.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set video_count.value = video_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
||||
{%- elif 'text' in item %}
|
||||
{{- item.text }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected item type in content.') }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- elif content is none or content is undefined %}
|
||||
{{- '' }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected content type.') }}
|
||||
{%- endif %}
|
||||
{%- endmacro %}
|
||||
{%- if not messages %}
|
||||
{{- raise_exception('No messages provided.') }}
|
||||
{%- endif %}
|
||||
{%- set sysns = namespace(count=0, text='') %}
|
||||
{%- for message in messages %}
|
||||
{%- if sysns.count == loop.index0 and (message.role == 'system' or message.role == 'developer') %}
|
||||
{%- set sys_content = render_content(message.content, false, true)|trim %}
|
||||
{%- if sys_content %}
|
||||
{%- set sysns.text = sysns.text + ('\n' if sysns.text else '') + sys_content %}
|
||||
{%- endif %}
|
||||
{%- set sysns.count = sysns.count + 1 %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set num_sys = sysns.count %}
|
||||
{%- set merged_system = sysns.text %}
|
||||
{%- set reasoning_instructions = '' %}
|
||||
{%- if enable_thinking is undefined or enable_thinking is true %}
|
||||
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
|
||||
{%- if resolved_reasoning_effort == 'high' %}
|
||||
{%- set resolved_reasoning_effort = 'xhigh' %}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
|
||||
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort == 'xhigh' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
|
||||
{%- elif resolved_reasoning_effort == 'low' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if tools and tools is iterable and tools is not mapping %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if reasoning_instructions %}
|
||||
{{- reasoning_instructions + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
|
||||
{%- for tool in tools %}
|
||||
{{- "\n" }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- "\n</tools>" }}
|
||||
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
||||
{%- if merged_system %}
|
||||
{{- '\n\n' + merged_system }}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{%- if merged_system %}
|
||||
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + merged_system + '<|im_end|>\n' }}
|
||||
{%- elif reasoning_instructions %}
|
||||
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
||||
{%- for message in messages[::-1] %}
|
||||
{%- set index = (messages|length - 1) - loop.index0 %}
|
||||
{%- if ns.multi_step_tool and message.role == "user" %}
|
||||
{%- set content = render_content(message.content, false)|trim %}
|
||||
{%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
|
||||
{%- set ns.multi_step_tool = false %}
|
||||
{%- set ns.last_query_index = index %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- for message in messages %}
|
||||
{%- if loop.index0 >= num_sys %}
|
||||
{%- set content = render_content(message.content, true)|trim %}
|
||||
{%- if message.role == "system" or message.role == "developer" %}
|
||||
{{- raise_exception('System message must be at the beginning.') }}
|
||||
{%- elif message.role == "user" %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||
{%- elif message.role == "assistant" %}
|
||||
{%- set reasoning_content = '' %}
|
||||
{%- if message.reasoning_content is string %}
|
||||
{%- set reasoning_content = message.reasoning_content %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = reasoning_content|trim %}
|
||||
{%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
|
||||
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||
{%- endif %}
|
||||
{%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if tool_call.function is defined %}
|
||||
{%- set tool_call = tool_call.function %}
|
||||
{%- endif %}
|
||||
{%- if tool_call.name is not defined or tool_call.name is none %}
|
||||
{{- raise_exception('Tool call is missing a function name.') }}
|
||||
{%- endif %}
|
||||
{%- if loop.first %}
|
||||
{%- if content|trim %}
|
||||
{{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- else %}
|
||||
{{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- if tool_call.arguments is mapping %}
|
||||
{%- for args_name, args_value in tool_call.arguments|items %}
|
||||
{{- '<parameter=' + args_name + '>\n' }}
|
||||
{%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
|
||||
{{- args_value }}
|
||||
{{- '\n</parameter>\n' }}
|
||||
{%- endfor %}
|
||||
{%- elif tool_call.arguments is string %}
|
||||
{%- if tool_call.arguments|trim %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" were passed as a JSON string. Parse them into an object before calling apply_chat_template.') }}
|
||||
{%- endif %}
|
||||
{%- elif tool_call.arguments is defined and tool_call.arguments is not none %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" must be an object/mapping or a JSON string.') }}
|
||||
{%- endif %}
|
||||
{{- '</function>\n</tool_call>' }}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == "tool" %}
|
||||
{%- if loop.previtem and loop.previtem.role != "tool" %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' }}
|
||||
{{- content }}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if not loop.last and loop.nextitem.role != "tool" %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif loop.last %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected message role.') }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- if enable_thinking is defined and enable_thinking is false %}
|
||||
{{- '<think>\n\n</think>\n\n' }}
|
||||
{%- else %}
|
||||
{{- '<think>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{#- Unsloth fixes - developer role, merged system messages, tool calling #}
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
{
|
||||
"protocol_id": "strata-qwen38-thinking-v1-draft.1",
|
||||
"bundle": "standard-v1",
|
||||
"profile": "qwen3.8-flash-next-thinking",
|
||||
"tokenizer_files": {
|
||||
"vocab.json": "4ba64f0332abcfb0b600b7df1537d1e836e68009d5fb7cdb77339738dc6365c4",
|
||||
"merges.txt": "080cf95432173729d160346b7882bd2a644b07f7a3813a3eb73f4e66787bfde9",
|
||||
"token_type.json": "5088c8c298fc06af8382ddb3b76c888703ac2634e82263b97eedcc2ad202738b"
|
||||
},
|
||||
"template": {
|
||||
"path": "serve/chat_template.jinja",
|
||||
"sha256": "ae9644304511ef71f28e036b6f65b4c709880bc5a23b7d0b42125ec63a8c70ca"
|
||||
},
|
||||
"template_kwargs": {
|
||||
"enable_thinking": true,
|
||||
"preserve_thinking": true,
|
||||
"reasoning_effort": "xhigh"
|
||||
},
|
||||
"record_format": "record{:06d}: alpha beta gamma delta epsilon zeta eta theta.",
|
||||
"short_task": "Explain how a hash table finds a value, including collisions and one small example.",
|
||||
"summary_question": "Summarize the structure of these records and explain how to look up a record by its ID.",
|
||||
"record_counts": {
|
||||
"prompt-4k": {
|
||||
"records": 211,
|
||||
"target_tokens": 4096
|
||||
},
|
||||
"prompt-16k": {
|
||||
"records": 858,
|
||||
"target_tokens": 16384
|
||||
}
|
||||
},
|
||||
"length_tolerance": 0.02,
|
||||
"requests": [
|
||||
{
|
||||
"id": "warmup",
|
||||
"input_tokens": 75,
|
||||
"max_shared_prefix_tokens": 47
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r1",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r1",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r1",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r2",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r2",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r2",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r3",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r3",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r3",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "sanity-I1",
|
||||
"input_tokens": 82,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I2",
|
||||
"input_tokens": 90,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I3",
|
||||
"input_tokens": 85,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I4",
|
||||
"input_tokens": 71,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R1",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R2",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R3",
|
||||
"input_tokens": 16382,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R4",
|
||||
"input_tokens": 16390,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-T1",
|
||||
"input_tokens": 325,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T2",
|
||||
"input_tokens": 335,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "sanity-T3",
|
||||
"input_tokens": 352,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T4",
|
||||
"input_tokens": 339,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
}
|
||||
],
|
||||
"note": "input_tokens counts the rendered thinking request with the generation prompt. max_shared_prefix_tokens is the longest token prefix this request shares with any other request of the bundle: cache reuse above it means the user payload was read from the cache."
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
bc6c1f81d86863c3dc36a31f58010701d8ab05180f69f86c6978214409e43af2 build.json
|
||||
f724855774dfbea89fade712a70d534a085c88cda6abc123db419fd79eefab49 requests/concurrency-r1-c1.json
|
||||
2c85f877abbf144ca065c58efc2a7568aaf2c8dc317ad8266e78c149f8245abb requests/concurrency-r1-c2.json
|
||||
264097f61645a0d5089562dc003b7748b1c20339c2df1290ad6664fc1834068a requests/concurrency-r1-c3.json
|
||||
65895c5a5fab53c06e9358d88d921f1bfc9dc7cc8b6e8a2521ac90de080ff470 requests/concurrency-r1-c4.json
|
||||
2c08d50e0c95e345512ec918f58fa716780b79900f09bbfabc37519c8d86543a requests/concurrency-r2-c1.json
|
||||
8aba49693701b9315b2bc5ad55e97a8b03496baedd05f370c49c2c77f45ce6cd requests/concurrency-r2-c2.json
|
||||
7cfb4a5a8fb5702244ce42e0dba9ccd3064d4023d700b409a1a472790a82809c requests/concurrency-r2-c3.json
|
||||
3ab56fbba5fd405945e53cff01b59c3ef966b0f08a0a1b0b50633d26e90cad62 requests/concurrency-r2-c4.json
|
||||
3adb684cf2cb84d4e514de30a4c6f8ccebdd8471160cfcdfc0cdb8ca19ac2137 requests/concurrency-r3-c1.json
|
||||
650379ef9a4da667f6581ae9573eb9af2db2a1fc617701e50536590b18f70077 requests/concurrency-r3-c2.json
|
||||
111415388c5e69ecbd9ab66efe85eb8f3db9142201e1548d03203d3acc05eca1 requests/concurrency-r3-c3.json
|
||||
691a8d27fd06e9beab57e6efb3d82c2a7ed9fc26635fd51b1b1d017117440c1e requests/concurrency-r3-c4.json
|
||||
06776ef38485dc4bb7e18809534fe16a57d421d163f81d4d1532c4b08225968e requests/performance-prompt-16k-r1.json
|
||||
4995c5f3c47c1682f4dc753a9f3a1fba6920fea14691e1ded2da6208ef5fe628 requests/performance-prompt-16k-r2.json
|
||||
7af4c49f99192007be27110296b5a0bcc2efbc91c7ff1c337a6cac6732050559 requests/performance-prompt-16k-r3.json
|
||||
bda0e60c2de0703899ac1bd5711d793b9b325c7f8ba6be6c967f22dd07d8dda8 requests/performance-prompt-4k-r1.json
|
||||
e898eec0ee22499473179a719505c9feb9d3772658833f607dffb34c3727150e requests/performance-prompt-4k-r2.json
|
||||
6d3b3300eace318574490f4379119409920cbe7e88fdaff28eb6f5c79430c3e3 requests/performance-prompt-4k-r3.json
|
||||
483b4f18fcc64ca802fde3377c59bf7b114ec5a5c96fe781bd2be4665ef7b898 requests/performance-short-r1.json
|
||||
b0ec85f40117303d19c8c17a4813625c8554f59caddbe69e7d8aafdcf51d501a requests/performance-short-r2.json
|
||||
e210da20b80c61147da5102fcba65131dafbbb4ceee952fa6b91ce2bb06581ed requests/performance-short-r3.json
|
||||
70ebcd31af5ae1bc7573136d9a257469d8ad9e85b5d0233980998e957c36d6c2 requests/sanity-I1.json
|
||||
1cc6e4bc56b5a22043b6e384b3036170dc6d3edeb403be391a8b802b5613f4cb requests/sanity-I2.json
|
||||
c29b13e6e31b674ff733202b3c0b1f9410768b9ea9368a4a6a13430b3e011fb8 requests/sanity-I3.json
|
||||
780fbd1d02f119323cf50ac4dabcfffbd23dd84115efbe18152e7ee9112b041a requests/sanity-I4.json
|
||||
3c1e0718ad06e591e47cf8df9c36bab2d13332d44b05e440ec17d0402bf9facc requests/sanity-R1.json
|
||||
352e054c48e08e11d339d3a02a14f4f50125071710679b8d04b77aba711fbafc requests/sanity-R2.json
|
||||
36770434f1ea046ec84b32691d644e8416e89fbcbb04b8f6032da66016d78291 requests/sanity-R3.json
|
||||
a63df33c784732cc6b4d8a0f2258b7226c8df5a3b4f6818f9e886bbf09604999 requests/sanity-R4.json
|
||||
6b272ef684b43c1e0004a40b958c807b84dcd9de579d4b2d8819e39ae4196fb2 requests/sanity-T1.json
|
||||
8253ee4aa6b19e61e28d64406807218c9aa10cf2cf682101e8a2d6a50564de46 requests/sanity-T2.json
|
||||
aa2be8761793f51eab6ef04bc54c96e018f6981b7a6a26b9e50cc3c7ac856803 requests/sanity-T3.json
|
||||
04043bdec397df487bebd2999e280a9fae5925e5efec1a10fa96055ec3430faa requests/sanity-T4.json
|
||||
d3365c57db8be1fd0dd9771f74f264b1f6994a396e472c3e5fa4ede2641ffb5c requests/warmup.json
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,75 @@
|
||||
# <user> standard benchmark: NVIDIA GeForce RTX 3090
|
||||
|
||||
Protocol `strata-qwen38-thinking-v1-draft.1` (draft), run `20261026T235154Z-87dfe6da`, started 2026-10-26T23:51:54Z. **Status: nonconforming.**
|
||||
|
||||
Written by `<user> bench run`. The machine-readable result is [results.json](results.json); every request, its raw usage and timings and the client event offsets are in [outputs.jsonl](outputs.jsonl).
|
||||
|
||||
This run is not a complete standard result:
|
||||
|
||||
- no model file could be hashed
|
||||
|
||||
## Hardware and software
|
||||
|
||||
- GPU: NVIDIA GeForce RTX 3090 (24.0 GiB dedicated)
|
||||
- CPU: Intel(R) Core(TM) i9-14900KS; RAM: 110.0 GiB
|
||||
- Storage of the model files: not measured
|
||||
- OS: NixOS 26.05 (Yarara) 6.18.55; drivers: gpu-driver 595.71.05; runtime: cuda 13.2
|
||||
- <user> commit: `4b9e05a2a806a3fe24f7b2560d98626c37dade35`; engine 0.1.51; source build
|
||||
- Background work: not measured
|
||||
|
||||
## Model and configuration
|
||||
|
||||
- Model: qwen3.8-flash-next-iq3_s; quantization IQ3_S; revision not measured; source revision not measured
|
||||
- Files: not measured
|
||||
- Track: **custom** (arm restart-baseline: [])
|
||||
- Context 131072 tokens; server capacity 1 slot(s)
|
||||
- Sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0, repetition_penalty 1.0, reasoning_effort xhigh; thinking on, preserved
|
||||
- Engine options: `config.json`; chat template: `provenance/chat_template.jinja`
|
||||
|
||||
## Method
|
||||
|
||||
The frozen requests of `bench/prompts/standard-v1` (corpus `e75eff8b9feea04f...`) were sent over loopback HTTP, streaming, one at a time, in this order: 1 warm-up (excluded), 3 repetitions of the short, 4K and 16K prompts, and the twelve sanity checks. Prompt and decode speed are the engine's own `timings`; time to first token is the client's monotonic clock from dispatch to the first nonempty reasoning, answer or tool-call delta. No request was retried. Time after the model loaded: 1 min 34 s.
|
||||
|
||||
## Results
|
||||
|
||||
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| short | n/a | 122.5 (106.5-132.7) n=3 | 0.47 (0.39-0.47) n=3 | 1.12 (0.99-1.82) n=3 | 3.9 (3.1-4.0) n=3 | 0/3 |
|
||||
| prompt-4k | 1712.2 (1540.6-1750.0) n=3 | 110.1 (93.8-130.0) n=3 | 2.41 (2.36-2.67) n=3 | 4.33 (4.33-4.33) n=1 | 7.0 (6.3-8.1) n=3 | 3/3 |
|
||||
| prompt-16k | 2880.8 (2838.2-2880.9) n=3 | 116.6 (114.3-124.7) n=3 | 5.71 (5.71-5.79) n=3 | 8.56 (7.45-8.83) n=3 | 10.1 (9.8-10.3) n=3 | 3/3 |
|
||||
|
||||
Cells are median (min-max) n=samples. Short is a prompt of about 80 tokens: its prompt rate is dominated by the fixed cost of a request, so it is not shown; read prompt speed from prompt-4k and prompt-16k. Capped = requests that reached the 512-token cap, reasoning and answer together. A capped request's total time is the time to the cap, not to an answer, and it has no first-answer time when it was still reasoning.
|
||||
|
||||
| Phase | Planned | Completed | Truncated | Failed | Skipped |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| warmup | 1 | 0 | 1 | 0 | 0 |
|
||||
| performance | 9 | 3 | 6 | 0 | 0 |
|
||||
| sanity | 12 | 12 | 0 | 0 | 0 |
|
||||
| concurrency | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
Sanity checks: **12 of 12**.
|
||||
|
||||
| Case | Result | Reason |
|
||||
| --- | --- | --- |
|
||||
| I1 | pass | the expected JSON value |
|
||||
| I2 | pass | the expected JSON value |
|
||||
| I3 | pass | the expected JSON value |
|
||||
| I4 | pass | the exact expected text |
|
||||
| R1 | pass | the exact expected text |
|
||||
| R2 | pass | the exact expected text |
|
||||
| R3 | pass | the exact expected text |
|
||||
| R4 | pass | the expected JSON value |
|
||||
| T1 | pass | the expected tool call |
|
||||
| T2 | pass | the expected tool call |
|
||||
| T3 | pass | the expected tool call |
|
||||
| T4 | pass | the expected tool call |
|
||||
|
||||
## Failures, exclusions and limitations
|
||||
|
||||
- No request failed.
|
||||
- Excluded from speed summaries: performance-prompt-4k-r1: first_answer_s is not available for this request
|
||||
- Excluded from speed summaries: performance-prompt-4k-r2: first_answer_s is not available for this request
|
||||
- Prompt-prefix reuse is the shared template prefix only; the cache was not emptied between requests, and the expert cache stays warm and evolves. These are not cold-model measurements.
|
||||
- Output caps include reasoning; a request that reaches its cap is `truncated` and fails its sanity check.
|
||||
- The runner cannot see other programs on the machine, the repeat-stop and loop-recovery switches of the server, or the quantization repository's revision; see `missing` in results.json for every value it could not read.
|
||||
- The server had answered no request before this run (a freshly loaded engine).
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"engine_options": null,
|
||||
"other_engine_options": {},
|
||||
"server_defaults": {
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
"top_k": 20,
|
||||
"n_predict": -1
|
||||
},
|
||||
"total_slots": 1,
|
||||
"context": 131072,
|
||||
"source": "the server was already running; its engine options are not exposed"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
+184
@@ -0,0 +1,184 @@
|
||||
{%- set image_count = namespace(value=0) %}
|
||||
{%- set video_count = namespace(value=0) %}
|
||||
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
||||
{%- if content is string %}
|
||||
{{- content }}
|
||||
{%- elif content is iterable and content is not mapping %}
|
||||
{%- for item in content %}
|
||||
{%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain images.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set image_count.value = image_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
||||
{%- elif 'video' in item or item.type == 'video' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain videos.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set video_count.value = video_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
||||
{%- elif 'text' in item %}
|
||||
{{- item.text }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected item type in content.') }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- elif content is none or content is undefined %}
|
||||
{{- '' }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected content type.') }}
|
||||
{%- endif %}
|
||||
{%- endmacro %}
|
||||
{%- if not messages %}
|
||||
{{- raise_exception('No messages provided.') }}
|
||||
{%- endif %}
|
||||
{%- set sysns = namespace(count=0, text='') %}
|
||||
{%- for message in messages %}
|
||||
{%- if sysns.count == loop.index0 and (message.role == 'system' or message.role == 'developer') %}
|
||||
{%- set sys_content = render_content(message.content, false, true)|trim %}
|
||||
{%- if sys_content %}
|
||||
{%- set sysns.text = sysns.text + ('\n' if sysns.text else '') + sys_content %}
|
||||
{%- endif %}
|
||||
{%- set sysns.count = sysns.count + 1 %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set num_sys = sysns.count %}
|
||||
{%- set merged_system = sysns.text %}
|
||||
{%- set reasoning_instructions = '' %}
|
||||
{%- if enable_thinking is undefined or enable_thinking is true %}
|
||||
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
|
||||
{%- if resolved_reasoning_effort == 'high' %}
|
||||
{%- set resolved_reasoning_effort = 'xhigh' %}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
|
||||
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort == 'xhigh' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
|
||||
{%- elif resolved_reasoning_effort == 'low' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if tools and tools is iterable and tools is not mapping %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if reasoning_instructions %}
|
||||
{{- reasoning_instructions + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
|
||||
{%- for tool in tools %}
|
||||
{{- "\n" }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- "\n</tools>" }}
|
||||
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
||||
{%- if merged_system %}
|
||||
{{- '\n\n' + merged_system }}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{%- if merged_system %}
|
||||
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + merged_system + '<|im_end|>\n' }}
|
||||
{%- elif reasoning_instructions %}
|
||||
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
||||
{%- for message in messages[::-1] %}
|
||||
{%- set index = (messages|length - 1) - loop.index0 %}
|
||||
{%- if ns.multi_step_tool and message.role == "user" %}
|
||||
{%- set content = render_content(message.content, false)|trim %}
|
||||
{%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
|
||||
{%- set ns.multi_step_tool = false %}
|
||||
{%- set ns.last_query_index = index %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- for message in messages %}
|
||||
{%- if loop.index0 >= num_sys %}
|
||||
{%- set content = render_content(message.content, true)|trim %}
|
||||
{%- if message.role == "system" or message.role == "developer" %}
|
||||
{{- raise_exception('System message must be at the beginning.') }}
|
||||
{%- elif message.role == "user" %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||
{%- elif message.role == "assistant" %}
|
||||
{%- set reasoning_content = '' %}
|
||||
{%- if message.reasoning_content is string %}
|
||||
{%- set reasoning_content = message.reasoning_content %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = reasoning_content|trim %}
|
||||
{%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
|
||||
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||
{%- endif %}
|
||||
{%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if tool_call.function is defined %}
|
||||
{%- set tool_call = tool_call.function %}
|
||||
{%- endif %}
|
||||
{%- if tool_call.name is not defined or tool_call.name is none %}
|
||||
{{- raise_exception('Tool call is missing a function name.') }}
|
||||
{%- endif %}
|
||||
{%- if loop.first %}
|
||||
{%- if content|trim %}
|
||||
{{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- else %}
|
||||
{{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- if tool_call.arguments is mapping %}
|
||||
{%- for args_name, args_value in tool_call.arguments|items %}
|
||||
{{- '<parameter=' + args_name + '>\n' }}
|
||||
{%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
|
||||
{{- args_value }}
|
||||
{{- '\n</parameter>\n' }}
|
||||
{%- endfor %}
|
||||
{%- elif tool_call.arguments is string %}
|
||||
{%- if tool_call.arguments|trim %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" were passed as a JSON string. Parse them into an object before calling apply_chat_template.') }}
|
||||
{%- endif %}
|
||||
{%- elif tool_call.arguments is defined and tool_call.arguments is not none %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" must be an object/mapping or a JSON string.') }}
|
||||
{%- endif %}
|
||||
{{- '</function>\n</tool_call>' }}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == "tool" %}
|
||||
{%- if loop.previtem and loop.previtem.role != "tool" %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' }}
|
||||
{{- content }}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if not loop.last and loop.nextitem.role != "tool" %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif loop.last %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected message role.') }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- if enable_thinking is defined and enable_thinking is false %}
|
||||
{{- '<think>\n\n</think>\n\n' }}
|
||||
{%- else %}
|
||||
{{- '<think>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{#- Unsloth fixes - developer role, merged system messages, tool calling #}
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
{
|
||||
"protocol_id": "strata-qwen38-thinking-v1-draft.1",
|
||||
"bundle": "standard-v1",
|
||||
"profile": "qwen3.8-flash-next-thinking",
|
||||
"tokenizer_files": {
|
||||
"vocab.json": "4ba64f0332abcfb0b600b7df1537d1e836e68009d5fb7cdb77339738dc6365c4",
|
||||
"merges.txt": "080cf95432173729d160346b7882bd2a644b07f7a3813a3eb73f4e66787bfde9",
|
||||
"token_type.json": "5088c8c298fc06af8382ddb3b76c888703ac2634e82263b97eedcc2ad202738b"
|
||||
},
|
||||
"template": {
|
||||
"path": "serve/chat_template.jinja",
|
||||
"sha256": "ae9644304511ef71f28e036b6f65b4c709880bc5a23b7d0b42125ec63a8c70ca"
|
||||
},
|
||||
"template_kwargs": {
|
||||
"enable_thinking": true,
|
||||
"preserve_thinking": true,
|
||||
"reasoning_effort": "xhigh"
|
||||
},
|
||||
"record_format": "record{:06d}: alpha beta gamma delta epsilon zeta eta theta.",
|
||||
"short_task": "Explain how a hash table finds a value, including collisions and one small example.",
|
||||
"summary_question": "Summarize the structure of these records and explain how to look up a record by its ID.",
|
||||
"record_counts": {
|
||||
"prompt-4k": {
|
||||
"records": 211,
|
||||
"target_tokens": 4096
|
||||
},
|
||||
"prompt-16k": {
|
||||
"records": 858,
|
||||
"target_tokens": 16384
|
||||
}
|
||||
},
|
||||
"length_tolerance": 0.02,
|
||||
"requests": [
|
||||
{
|
||||
"id": "warmup",
|
||||
"input_tokens": 75,
|
||||
"max_shared_prefix_tokens": 47
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r1",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r1",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r1",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r2",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r2",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r2",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r3",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r3",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r3",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "sanity-I1",
|
||||
"input_tokens": 82,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I2",
|
||||
"input_tokens": 90,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I3",
|
||||
"input_tokens": 85,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I4",
|
||||
"input_tokens": 71,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R1",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R2",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R3",
|
||||
"input_tokens": 16382,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R4",
|
||||
"input_tokens": 16390,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-T1",
|
||||
"input_tokens": 325,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T2",
|
||||
"input_tokens": 335,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "sanity-T3",
|
||||
"input_tokens": 352,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T4",
|
||||
"input_tokens": 339,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
}
|
||||
],
|
||||
"note": "input_tokens counts the rendered thinking request with the generation prompt. max_shared_prefix_tokens is the longest token prefix this request shares with any other request of the bundle: cache reuse above it means the user payload was read from the cache."
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
bc6c1f81d86863c3dc36a31f58010701d8ab05180f69f86c6978214409e43af2 build.json
|
||||
f724855774dfbea89fade712a70d534a085c88cda6abc123db419fd79eefab49 requests/concurrency-r1-c1.json
|
||||
2c85f877abbf144ca065c58efc2a7568aaf2c8dc317ad8266e78c149f8245abb requests/concurrency-r1-c2.json
|
||||
264097f61645a0d5089562dc003b7748b1c20339c2df1290ad6664fc1834068a requests/concurrency-r1-c3.json
|
||||
65895c5a5fab53c06e9358d88d921f1bfc9dc7cc8b6e8a2521ac90de080ff470 requests/concurrency-r1-c4.json
|
||||
2c08d50e0c95e345512ec918f58fa716780b79900f09bbfabc37519c8d86543a requests/concurrency-r2-c1.json
|
||||
8aba49693701b9315b2bc5ad55e97a8b03496baedd05f370c49c2c77f45ce6cd requests/concurrency-r2-c2.json
|
||||
7cfb4a5a8fb5702244ce42e0dba9ccd3064d4023d700b409a1a472790a82809c requests/concurrency-r2-c3.json
|
||||
3ab56fbba5fd405945e53cff01b59c3ef966b0f08a0a1b0b50633d26e90cad62 requests/concurrency-r2-c4.json
|
||||
3adb684cf2cb84d4e514de30a4c6f8ccebdd8471160cfcdfc0cdb8ca19ac2137 requests/concurrency-r3-c1.json
|
||||
650379ef9a4da667f6581ae9573eb9af2db2a1fc617701e50536590b18f70077 requests/concurrency-r3-c2.json
|
||||
111415388c5e69ecbd9ab66efe85eb8f3db9142201e1548d03203d3acc05eca1 requests/concurrency-r3-c3.json
|
||||
691a8d27fd06e9beab57e6efb3d82c2a7ed9fc26635fd51b1b1d017117440c1e requests/concurrency-r3-c4.json
|
||||
06776ef38485dc4bb7e18809534fe16a57d421d163f81d4d1532c4b08225968e requests/performance-prompt-16k-r1.json
|
||||
4995c5f3c47c1682f4dc753a9f3a1fba6920fea14691e1ded2da6208ef5fe628 requests/performance-prompt-16k-r2.json
|
||||
7af4c49f99192007be27110296b5a0bcc2efbc91c7ff1c337a6cac6732050559 requests/performance-prompt-16k-r3.json
|
||||
bda0e60c2de0703899ac1bd5711d793b9b325c7f8ba6be6c967f22dd07d8dda8 requests/performance-prompt-4k-r1.json
|
||||
e898eec0ee22499473179a719505c9feb9d3772658833f607dffb34c3727150e requests/performance-prompt-4k-r2.json
|
||||
6d3b3300eace318574490f4379119409920cbe7e88fdaff28eb6f5c79430c3e3 requests/performance-prompt-4k-r3.json
|
||||
483b4f18fcc64ca802fde3377c59bf7b114ec5a5c96fe781bd2be4665ef7b898 requests/performance-short-r1.json
|
||||
b0ec85f40117303d19c8c17a4813625c8554f59caddbe69e7d8aafdcf51d501a requests/performance-short-r2.json
|
||||
e210da20b80c61147da5102fcba65131dafbbb4ceee952fa6b91ce2bb06581ed requests/performance-short-r3.json
|
||||
70ebcd31af5ae1bc7573136d9a257469d8ad9e85b5d0233980998e957c36d6c2 requests/sanity-I1.json
|
||||
1cc6e4bc56b5a22043b6e384b3036170dc6d3edeb403be391a8b802b5613f4cb requests/sanity-I2.json
|
||||
c29b13e6e31b674ff733202b3c0b1f9410768b9ea9368a4a6a13430b3e011fb8 requests/sanity-I3.json
|
||||
780fbd1d02f119323cf50ac4dabcfffbd23dd84115efbe18152e7ee9112b041a requests/sanity-I4.json
|
||||
3c1e0718ad06e591e47cf8df9c36bab2d13332d44b05e440ec17d0402bf9facc requests/sanity-R1.json
|
||||
352e054c48e08e11d339d3a02a14f4f50125071710679b8d04b77aba711fbafc requests/sanity-R2.json
|
||||
36770434f1ea046ec84b32691d644e8416e89fbcbb04b8f6032da66016d78291 requests/sanity-R3.json
|
||||
a63df33c784732cc6b4d8a0f2258b7226c8df5a3b4f6818f9e886bbf09604999 requests/sanity-R4.json
|
||||
6b272ef684b43c1e0004a40b958c807b84dcd9de579d4b2d8819e39ae4196fb2 requests/sanity-T1.json
|
||||
8253ee4aa6b19e61e28d64406807218c9aa10cf2cf682101e8a2d6a50564de46 requests/sanity-T2.json
|
||||
aa2be8761793f51eab6ef04bc54c96e018f6981b7a6a26b9e50cc3c7ac856803 requests/sanity-T3.json
|
||||
04043bdec397df487bebd2999e280a9fae5925e5efec1a10fa96055ec3430faa requests/sanity-T4.json
|
||||
d3365c57db8be1fd0dd9771f74f264b1f6994a396e472c3e5fa4ede2641ffb5c requests/warmup.json
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,74 @@
|
||||
# <user> standard benchmark: NVIDIA GeForce RTX 3090
|
||||
|
||||
Protocol `strata-qwen38-thinking-v1-draft.1` (draft), run `20261027T000709Z-53bbc5d8`, started 2026-10-27T00:07:09Z. **Status: nonconforming.**
|
||||
|
||||
Written by `<user> bench run`. The machine-readable result is [results.json](results.json); every request, its raw usage and timings and the client event offsets are in [outputs.jsonl](outputs.jsonl).
|
||||
|
||||
This run is not a complete standard result:
|
||||
|
||||
- no model file could be hashed
|
||||
|
||||
## Hardware and software
|
||||
|
||||
- GPU: NVIDIA GeForce RTX 3090 (24.0 GiB dedicated)
|
||||
- CPU: Intel(R) Core(TM) i9-14900KS; RAM: 110.0 GiB
|
||||
- Storage of the model files: not measured
|
||||
- OS: NixOS 26.05 (Yarara) 6.18.55; drivers: gpu-driver 595.71.05; runtime: cuda 13.2
|
||||
- <user> commit: `4b9e05a2a806a3fe24f7b2560d98626c37dade35`; engine 0.1.51; source build
|
||||
- Background work: not measured
|
||||
|
||||
## Model and configuration
|
||||
|
||||
- Model: qwen3.8-flash-next-iq3_s; quantization IQ3_S; revision not measured; source revision not measured
|
||||
- Files: not measured
|
||||
- Track: **custom** (arm combined: ["--adapt-async","1","--pipeline-windows","2","--prefill","auto:32768"])
|
||||
- Context 131072 tokens; server capacity 1 slot(s)
|
||||
- Sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0, repetition_penalty 1.0, reasoning_effort xhigh; thinking on, preserved
|
||||
- Engine options: `config.json`; chat template: `provenance/chat_template.jinja`
|
||||
|
||||
## Method
|
||||
|
||||
The frozen requests of `bench/prompts/standard-v1` (corpus `e75eff8b9feea04f...`) were sent over loopback HTTP, streaming, one at a time, in this order: 1 warm-up (excluded), 3 repetitions of the short, 4K and 16K prompts, and the twelve sanity checks. Prompt and decode speed are the engine's own `timings`; time to first token is the client's monotonic clock from dispatch to the first nonempty reasoning, answer or tool-call delta. No request was retried. Time after the model loaded: 1 min 42 s.
|
||||
|
||||
## Results
|
||||
|
||||
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| short | n/a | 121.1 (107.1-121.8) n=3 | 0.48 (0.39-0.49) n=3 | 1.43 (1.27-1.57) n=3 | 3.9 (3.1-4.0) n=3 | 0/3 |
|
||||
| prompt-4k | 1699.4 (1541.2-1740.4) n=3 | 121.7 (95.1-125.9) n=3 | 2.43 (2.37-2.67) n=3 | 4.20 (4.12-6.68) n=3 | 6.4 (5.9-8.0) n=3 | 2/3 |
|
||||
| prompt-16k | 2242.5 (2163.6-2259.4) n=3 | 113.4 (101.7-116.8) n=3 | 7.34 (7.28-7.60) n=3 | 10.75 (10.70-10.81) n=2 | 11.8 (11.7-12.6) n=3 | 3/3 |
|
||||
|
||||
Cells are median (min-max) n=samples. Short is a prompt of about 80 tokens: its prompt rate is dominated by the fixed cost of a request, so it is not shown; read prompt speed from prompt-4k and prompt-16k. Capped = requests that reached the 512-token cap, reasoning and answer together. A capped request's total time is the time to the cap, not to an answer, and it has no first-answer time when it was still reasoning.
|
||||
|
||||
| Phase | Planned | Completed | Truncated | Failed | Skipped |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| warmup | 1 | 0 | 1 | 0 | 0 |
|
||||
| performance | 9 | 4 | 5 | 0 | 0 |
|
||||
| sanity | 12 | 12 | 0 | 0 | 0 |
|
||||
| concurrency | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
Sanity checks: **12 of 12**.
|
||||
|
||||
| Case | Result | Reason |
|
||||
| --- | --- | --- |
|
||||
| I1 | pass | the expected JSON value |
|
||||
| I2 | pass | the expected JSON value |
|
||||
| I3 | pass | the expected JSON value |
|
||||
| I4 | pass | the exact expected text |
|
||||
| R1 | pass | the exact expected text |
|
||||
| R2 | pass | the exact expected text |
|
||||
| R3 | pass | the exact expected text |
|
||||
| R4 | pass | the expected JSON value |
|
||||
| T1 | pass | the expected tool call |
|
||||
| T2 | pass | the expected tool call |
|
||||
| T3 | pass | the expected tool call |
|
||||
| T4 | pass | the expected tool call |
|
||||
|
||||
## Failures, exclusions and limitations
|
||||
|
||||
- No request failed.
|
||||
- Excluded from speed summaries: performance-prompt-16k-r1: first_answer_s is not available for this request
|
||||
- Prompt-prefix reuse is the shared template prefix only; the cache was not emptied between requests, and the expert cache stays warm and evolves. These are not cold-model measurements.
|
||||
- Output caps include reasoning; a request that reaches its cap is `truncated` and fails its sanity check.
|
||||
- The runner cannot see other programs on the machine, the repeat-stop and loop-recovery switches of the server, or the quantization repository's revision; see `missing` in results.json for every value it could not read.
|
||||
- The server had answered no request before this run (a freshly loaded engine).
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"engine_options": null,
|
||||
"other_engine_options": {},
|
||||
"server_defaults": {
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
"top_k": 20,
|
||||
"n_predict": -1
|
||||
},
|
||||
"total_slots": 1,
|
||||
"context": 131072,
|
||||
"source": "the server was already running; its engine options are not exposed"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
+184
@@ -0,0 +1,184 @@
|
||||
{%- set image_count = namespace(value=0) %}
|
||||
{%- set video_count = namespace(value=0) %}
|
||||
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
||||
{%- if content is string %}
|
||||
{{- content }}
|
||||
{%- elif content is iterable and content is not mapping %}
|
||||
{%- for item in content %}
|
||||
{%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain images.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set image_count.value = image_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
||||
{%- elif 'video' in item or item.type == 'video' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain videos.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set video_count.value = video_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
||||
{%- elif 'text' in item %}
|
||||
{{- item.text }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected item type in content.') }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- elif content is none or content is undefined %}
|
||||
{{- '' }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected content type.') }}
|
||||
{%- endif %}
|
||||
{%- endmacro %}
|
||||
{%- if not messages %}
|
||||
{{- raise_exception('No messages provided.') }}
|
||||
{%- endif %}
|
||||
{%- set sysns = namespace(count=0, text='') %}
|
||||
{%- for message in messages %}
|
||||
{%- if sysns.count == loop.index0 and (message.role == 'system' or message.role == 'developer') %}
|
||||
{%- set sys_content = render_content(message.content, false, true)|trim %}
|
||||
{%- if sys_content %}
|
||||
{%- set sysns.text = sysns.text + ('\n' if sysns.text else '') + sys_content %}
|
||||
{%- endif %}
|
||||
{%- set sysns.count = sysns.count + 1 %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set num_sys = sysns.count %}
|
||||
{%- set merged_system = sysns.text %}
|
||||
{%- set reasoning_instructions = '' %}
|
||||
{%- if enable_thinking is undefined or enable_thinking is true %}
|
||||
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
|
||||
{%- if resolved_reasoning_effort == 'high' %}
|
||||
{%- set resolved_reasoning_effort = 'xhigh' %}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
|
||||
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort == 'xhigh' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
|
||||
{%- elif resolved_reasoning_effort == 'low' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if tools and tools is iterable and tools is not mapping %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if reasoning_instructions %}
|
||||
{{- reasoning_instructions + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
|
||||
{%- for tool in tools %}
|
||||
{{- "\n" }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- "\n</tools>" }}
|
||||
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
||||
{%- if merged_system %}
|
||||
{{- '\n\n' + merged_system }}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{%- if merged_system %}
|
||||
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + merged_system + '<|im_end|>\n' }}
|
||||
{%- elif reasoning_instructions %}
|
||||
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
||||
{%- for message in messages[::-1] %}
|
||||
{%- set index = (messages|length - 1) - loop.index0 %}
|
||||
{%- if ns.multi_step_tool and message.role == "user" %}
|
||||
{%- set content = render_content(message.content, false)|trim %}
|
||||
{%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
|
||||
{%- set ns.multi_step_tool = false %}
|
||||
{%- set ns.last_query_index = index %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- for message in messages %}
|
||||
{%- if loop.index0 >= num_sys %}
|
||||
{%- set content = render_content(message.content, true)|trim %}
|
||||
{%- if message.role == "system" or message.role == "developer" %}
|
||||
{{- raise_exception('System message must be at the beginning.') }}
|
||||
{%- elif message.role == "user" %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||
{%- elif message.role == "assistant" %}
|
||||
{%- set reasoning_content = '' %}
|
||||
{%- if message.reasoning_content is string %}
|
||||
{%- set reasoning_content = message.reasoning_content %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = reasoning_content|trim %}
|
||||
{%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
|
||||
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||
{%- endif %}
|
||||
{%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if tool_call.function is defined %}
|
||||
{%- set tool_call = tool_call.function %}
|
||||
{%- endif %}
|
||||
{%- if tool_call.name is not defined or tool_call.name is none %}
|
||||
{{- raise_exception('Tool call is missing a function name.') }}
|
||||
{%- endif %}
|
||||
{%- if loop.first %}
|
||||
{%- if content|trim %}
|
||||
{{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- else %}
|
||||
{{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- if tool_call.arguments is mapping %}
|
||||
{%- for args_name, args_value in tool_call.arguments|items %}
|
||||
{{- '<parameter=' + args_name + '>\n' }}
|
||||
{%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
|
||||
{{- args_value }}
|
||||
{{- '\n</parameter>\n' }}
|
||||
{%- endfor %}
|
||||
{%- elif tool_call.arguments is string %}
|
||||
{%- if tool_call.arguments|trim %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" were passed as a JSON string. Parse them into an object before calling apply_chat_template.') }}
|
||||
{%- endif %}
|
||||
{%- elif tool_call.arguments is defined and tool_call.arguments is not none %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" must be an object/mapping or a JSON string.') }}
|
||||
{%- endif %}
|
||||
{{- '</function>\n</tool_call>' }}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == "tool" %}
|
||||
{%- if loop.previtem and loop.previtem.role != "tool" %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' }}
|
||||
{{- content }}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if not loop.last and loop.nextitem.role != "tool" %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif loop.last %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected message role.') }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- if enable_thinking is defined and enable_thinking is false %}
|
||||
{{- '<think>\n\n</think>\n\n' }}
|
||||
{%- else %}
|
||||
{{- '<think>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{#- Unsloth fixes - developer role, merged system messages, tool calling #}
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
{
|
||||
"protocol_id": "strata-qwen38-thinking-v1-draft.1",
|
||||
"bundle": "standard-v1",
|
||||
"profile": "qwen3.8-flash-next-thinking",
|
||||
"tokenizer_files": {
|
||||
"vocab.json": "4ba64f0332abcfb0b600b7df1537d1e836e68009d5fb7cdb77339738dc6365c4",
|
||||
"merges.txt": "080cf95432173729d160346b7882bd2a644b07f7a3813a3eb73f4e66787bfde9",
|
||||
"token_type.json": "5088c8c298fc06af8382ddb3b76c888703ac2634e82263b97eedcc2ad202738b"
|
||||
},
|
||||
"template": {
|
||||
"path": "serve/chat_template.jinja",
|
||||
"sha256": "ae9644304511ef71f28e036b6f65b4c709880bc5a23b7d0b42125ec63a8c70ca"
|
||||
},
|
||||
"template_kwargs": {
|
||||
"enable_thinking": true,
|
||||
"preserve_thinking": true,
|
||||
"reasoning_effort": "xhigh"
|
||||
},
|
||||
"record_format": "record{:06d}: alpha beta gamma delta epsilon zeta eta theta.",
|
||||
"short_task": "Explain how a hash table finds a value, including collisions and one small example.",
|
||||
"summary_question": "Summarize the structure of these records and explain how to look up a record by its ID.",
|
||||
"record_counts": {
|
||||
"prompt-4k": {
|
||||
"records": 211,
|
||||
"target_tokens": 4096
|
||||
},
|
||||
"prompt-16k": {
|
||||
"records": 858,
|
||||
"target_tokens": 16384
|
||||
}
|
||||
},
|
||||
"length_tolerance": 0.02,
|
||||
"requests": [
|
||||
{
|
||||
"id": "warmup",
|
||||
"input_tokens": 75,
|
||||
"max_shared_prefix_tokens": 47
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r1",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r1",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r1",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r2",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r2",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r2",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r3",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r3",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r3",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "sanity-I1",
|
||||
"input_tokens": 82,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I2",
|
||||
"input_tokens": 90,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I3",
|
||||
"input_tokens": 85,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I4",
|
||||
"input_tokens": 71,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R1",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R2",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R3",
|
||||
"input_tokens": 16382,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R4",
|
||||
"input_tokens": 16390,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-T1",
|
||||
"input_tokens": 325,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T2",
|
||||
"input_tokens": 335,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "sanity-T3",
|
||||
"input_tokens": 352,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T4",
|
||||
"input_tokens": 339,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
}
|
||||
],
|
||||
"note": "input_tokens counts the rendered thinking request with the generation prompt. max_shared_prefix_tokens is the longest token prefix this request shares with any other request of the bundle: cache reuse above it means the user payload was read from the cache."
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
bc6c1f81d86863c3dc36a31f58010701d8ab05180f69f86c6978214409e43af2 build.json
|
||||
f724855774dfbea89fade712a70d534a085c88cda6abc123db419fd79eefab49 requests/concurrency-r1-c1.json
|
||||
2c85f877abbf144ca065c58efc2a7568aaf2c8dc317ad8266e78c149f8245abb requests/concurrency-r1-c2.json
|
||||
264097f61645a0d5089562dc003b7748b1c20339c2df1290ad6664fc1834068a requests/concurrency-r1-c3.json
|
||||
65895c5a5fab53c06e9358d88d921f1bfc9dc7cc8b6e8a2521ac90de080ff470 requests/concurrency-r1-c4.json
|
||||
2c08d50e0c95e345512ec918f58fa716780b79900f09bbfabc37519c8d86543a requests/concurrency-r2-c1.json
|
||||
8aba49693701b9315b2bc5ad55e97a8b03496baedd05f370c49c2c77f45ce6cd requests/concurrency-r2-c2.json
|
||||
7cfb4a5a8fb5702244ce42e0dba9ccd3064d4023d700b409a1a472790a82809c requests/concurrency-r2-c3.json
|
||||
3ab56fbba5fd405945e53cff01b59c3ef966b0f08a0a1b0b50633d26e90cad62 requests/concurrency-r2-c4.json
|
||||
3adb684cf2cb84d4e514de30a4c6f8ccebdd8471160cfcdfc0cdb8ca19ac2137 requests/concurrency-r3-c1.json
|
||||
650379ef9a4da667f6581ae9573eb9af2db2a1fc617701e50536590b18f70077 requests/concurrency-r3-c2.json
|
||||
111415388c5e69ecbd9ab66efe85eb8f3db9142201e1548d03203d3acc05eca1 requests/concurrency-r3-c3.json
|
||||
691a8d27fd06e9beab57e6efb3d82c2a7ed9fc26635fd51b1b1d017117440c1e requests/concurrency-r3-c4.json
|
||||
06776ef38485dc4bb7e18809534fe16a57d421d163f81d4d1532c4b08225968e requests/performance-prompt-16k-r1.json
|
||||
4995c5f3c47c1682f4dc753a9f3a1fba6920fea14691e1ded2da6208ef5fe628 requests/performance-prompt-16k-r2.json
|
||||
7af4c49f99192007be27110296b5a0bcc2efbc91c7ff1c337a6cac6732050559 requests/performance-prompt-16k-r3.json
|
||||
bda0e60c2de0703899ac1bd5711d793b9b325c7f8ba6be6c967f22dd07d8dda8 requests/performance-prompt-4k-r1.json
|
||||
e898eec0ee22499473179a719505c9feb9d3772658833f607dffb34c3727150e requests/performance-prompt-4k-r2.json
|
||||
6d3b3300eace318574490f4379119409920cbe7e88fdaff28eb6f5c79430c3e3 requests/performance-prompt-4k-r3.json
|
||||
483b4f18fcc64ca802fde3377c59bf7b114ec5a5c96fe781bd2be4665ef7b898 requests/performance-short-r1.json
|
||||
b0ec85f40117303d19c8c17a4813625c8554f59caddbe69e7d8aafdcf51d501a requests/performance-short-r2.json
|
||||
e210da20b80c61147da5102fcba65131dafbbb4ceee952fa6b91ce2bb06581ed requests/performance-short-r3.json
|
||||
70ebcd31af5ae1bc7573136d9a257469d8ad9e85b5d0233980998e957c36d6c2 requests/sanity-I1.json
|
||||
1cc6e4bc56b5a22043b6e384b3036170dc6d3edeb403be391a8b802b5613f4cb requests/sanity-I2.json
|
||||
c29b13e6e31b674ff733202b3c0b1f9410768b9ea9368a4a6a13430b3e011fb8 requests/sanity-I3.json
|
||||
780fbd1d02f119323cf50ac4dabcfffbd23dd84115efbe18152e7ee9112b041a requests/sanity-I4.json
|
||||
3c1e0718ad06e591e47cf8df9c36bab2d13332d44b05e440ec17d0402bf9facc requests/sanity-R1.json
|
||||
352e054c48e08e11d339d3a02a14f4f50125071710679b8d04b77aba711fbafc requests/sanity-R2.json
|
||||
36770434f1ea046ec84b32691d644e8416e89fbcbb04b8f6032da66016d78291 requests/sanity-R3.json
|
||||
a63df33c784732cc6b4d8a0f2258b7226c8df5a3b4f6818f9e886bbf09604999 requests/sanity-R4.json
|
||||
6b272ef684b43c1e0004a40b958c807b84dcd9de579d4b2d8819e39ae4196fb2 requests/sanity-T1.json
|
||||
8253ee4aa6b19e61e28d64406807218c9aa10cf2cf682101e8a2d6a50564de46 requests/sanity-T2.json
|
||||
aa2be8761793f51eab6ef04bc54c96e018f6981b7a6a26b9e50cc3c7ac856803 requests/sanity-T3.json
|
||||
04043bdec397df487bebd2999e280a9fae5925e5efec1a10fa96055ec3430faa requests/sanity-T4.json
|
||||
d3365c57db8be1fd0dd9771f74f264b1f6994a396e472c3e5fa4ede2641ffb5c requests/warmup.json
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,75 @@
|
||||
# <user> standard benchmark: NVIDIA GeForce RTX 3090
|
||||
|
||||
Protocol `strata-qwen38-thinking-v1-draft.1` (draft), run `20261027T000315Z-92972582`, started 2026-10-27T00:03:15Z. **Status: nonconforming.**
|
||||
|
||||
Written by `<user> bench run`. The machine-readable result is [results.json](results.json); every request, its raw usage and timings and the client event offsets are in [outputs.jsonl](outputs.jsonl).
|
||||
|
||||
This run is not a complete standard result:
|
||||
|
||||
- no model file could be hashed
|
||||
|
||||
## Hardware and software
|
||||
|
||||
- GPU: NVIDIA GeForce RTX 3090 (24.0 GiB dedicated)
|
||||
- CPU: Intel(R) Core(TM) i9-14900KS; RAM: 110.0 GiB
|
||||
- Storage of the model files: not measured
|
||||
- OS: NixOS 26.05 (Yarara) 6.18.55; drivers: gpu-driver 595.71.05; runtime: cuda 13.2
|
||||
- <user> commit: `4b9e05a2a806a3fe24f7b2560d98626c37dade35`; engine 0.1.51; source build
|
||||
- Background work: not measured
|
||||
|
||||
## Model and configuration
|
||||
|
||||
- Model: qwen3.8-flash-next-iq3_s; quantization IQ3_S; revision not measured; source revision not measured
|
||||
- Files: not measured
|
||||
- Track: **custom** (arm prefill-32k: ["--prefill","auto:32768"])
|
||||
- Context 131072 tokens; server capacity 1 slot(s)
|
||||
- Sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0, repetition_penalty 1.0, reasoning_effort xhigh; thinking on, preserved
|
||||
- Engine options: `config.json`; chat template: `provenance/chat_template.jinja`
|
||||
|
||||
## Method
|
||||
|
||||
The frozen requests of `bench/prompts/standard-v1` (corpus `e75eff8b9feea04f...`) were sent over loopback HTTP, streaming, one at a time, in this order: 1 warm-up (excluded), 3 repetitions of the short, 4K and 16K prompts, and the twelve sanity checks. Prompt and decode speed are the engine's own `timings`; time to first token is the client's monotonic clock from dispatch to the first nonempty reasoning, answer or tool-call delta. No request was retried. Time after the model loaded: 1 min 42 s.
|
||||
|
||||
## Results
|
||||
|
||||
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| short | n/a | 122.7 (119.5-128.2) n=3 | 0.48 (0.39-0.49) n=3 | 1.11 (0.89-1.71) n=3 | 4.0 (2.5-4.2) n=3 | 0/3 |
|
||||
| prompt-4k | 1702.4 (1545.2-1743.6) n=3 | 111.6 (103.0-126.5) n=3 | 2.42 (2.36-2.67) n=3 | 5.49 (4.84-6.13) n=2 | 7.2 (6.5-7.3) n=3 | 3/3 |
|
||||
| prompt-16k | 2239.4 (2145.0-2247.7) n=3 | 105.9 (99.9-118.3) n=3 | 7.35 (7.32-7.67) n=3 | 10.70 (9.77-11.62) n=2 | 12.1 (12.0-12.5) n=3 | 3/3 |
|
||||
|
||||
Cells are median (min-max) n=samples. Short is a prompt of about 80 tokens: its prompt rate is dominated by the fixed cost of a request, so it is not shown; read prompt speed from prompt-4k and prompt-16k. Capped = requests that reached the 512-token cap, reasoning and answer together. A capped request's total time is the time to the cap, not to an answer, and it has no first-answer time when it was still reasoning.
|
||||
|
||||
| Phase | Planned | Completed | Truncated | Failed | Skipped |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| warmup | 1 | 0 | 1 | 0 | 0 |
|
||||
| performance | 9 | 3 | 6 | 0 | 0 |
|
||||
| sanity | 12 | 12 | 0 | 0 | 0 |
|
||||
| concurrency | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
Sanity checks: **12 of 12**.
|
||||
|
||||
| Case | Result | Reason |
|
||||
| --- | --- | --- |
|
||||
| I1 | pass | the expected JSON value |
|
||||
| I2 | pass | the expected JSON value |
|
||||
| I3 | pass | the expected JSON value |
|
||||
| I4 | pass | the exact expected text |
|
||||
| R1 | pass | the exact expected text |
|
||||
| R2 | pass | the exact expected text |
|
||||
| R3 | pass | the exact expected text |
|
||||
| R4 | pass | the expected JSON value |
|
||||
| T1 | pass | the expected tool call |
|
||||
| T2 | pass | the expected tool call |
|
||||
| T3 | pass | the expected tool call |
|
||||
| T4 | pass | the expected tool call |
|
||||
|
||||
## Failures, exclusions and limitations
|
||||
|
||||
- No request failed.
|
||||
- Excluded from speed summaries: performance-prompt-16k-r3: first_answer_s is not available for this request
|
||||
- Excluded from speed summaries: performance-prompt-4k-r3: first_answer_s is not available for this request
|
||||
- Prompt-prefix reuse is the shared template prefix only; the cache was not emptied between requests, and the expert cache stays warm and evolves. These are not cold-model measurements.
|
||||
- Output caps include reasoning; a request that reaches its cap is `truncated` and fails its sanity check.
|
||||
- The runner cannot see other programs on the machine, the repeat-stop and loop-recovery switches of the server, or the quantization repository's revision; see `missing` in results.json for every value it could not read.
|
||||
- The server had answered no request before this run (a freshly loaded engine).
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"engine_options": null,
|
||||
"other_engine_options": {},
|
||||
"server_defaults": {
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
"top_k": 20,
|
||||
"n_predict": -1
|
||||
},
|
||||
"total_slots": 1,
|
||||
"context": 131072,
|
||||
"source": "the server was already running; its engine options are not exposed"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
+184
@@ -0,0 +1,184 @@
|
||||
{%- set image_count = namespace(value=0) %}
|
||||
{%- set video_count = namespace(value=0) %}
|
||||
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
||||
{%- if content is string %}
|
||||
{{- content }}
|
||||
{%- elif content is iterable and content is not mapping %}
|
||||
{%- for item in content %}
|
||||
{%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain images.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set image_count.value = image_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
||||
{%- elif 'video' in item or item.type == 'video' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain videos.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set video_count.value = video_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
||||
{%- elif 'text' in item %}
|
||||
{{- item.text }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected item type in content.') }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- elif content is none or content is undefined %}
|
||||
{{- '' }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected content type.') }}
|
||||
{%- endif %}
|
||||
{%- endmacro %}
|
||||
{%- if not messages %}
|
||||
{{- raise_exception('No messages provided.') }}
|
||||
{%- endif %}
|
||||
{%- set sysns = namespace(count=0, text='') %}
|
||||
{%- for message in messages %}
|
||||
{%- if sysns.count == loop.index0 and (message.role == 'system' or message.role == 'developer') %}
|
||||
{%- set sys_content = render_content(message.content, false, true)|trim %}
|
||||
{%- if sys_content %}
|
||||
{%- set sysns.text = sysns.text + ('\n' if sysns.text else '') + sys_content %}
|
||||
{%- endif %}
|
||||
{%- set sysns.count = sysns.count + 1 %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set num_sys = sysns.count %}
|
||||
{%- set merged_system = sysns.text %}
|
||||
{%- set reasoning_instructions = '' %}
|
||||
{%- if enable_thinking is undefined or enable_thinking is true %}
|
||||
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
|
||||
{%- if resolved_reasoning_effort == 'high' %}
|
||||
{%- set resolved_reasoning_effort = 'xhigh' %}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
|
||||
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort == 'xhigh' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
|
||||
{%- elif resolved_reasoning_effort == 'low' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if tools and tools is iterable and tools is not mapping %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if reasoning_instructions %}
|
||||
{{- reasoning_instructions + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
|
||||
{%- for tool in tools %}
|
||||
{{- "\n" }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- "\n</tools>" }}
|
||||
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
||||
{%- if merged_system %}
|
||||
{{- '\n\n' + merged_system }}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{%- if merged_system %}
|
||||
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + merged_system + '<|im_end|>\n' }}
|
||||
{%- elif reasoning_instructions %}
|
||||
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
||||
{%- for message in messages[::-1] %}
|
||||
{%- set index = (messages|length - 1) - loop.index0 %}
|
||||
{%- if ns.multi_step_tool and message.role == "user" %}
|
||||
{%- set content = render_content(message.content, false)|trim %}
|
||||
{%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
|
||||
{%- set ns.multi_step_tool = false %}
|
||||
{%- set ns.last_query_index = index %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- for message in messages %}
|
||||
{%- if loop.index0 >= num_sys %}
|
||||
{%- set content = render_content(message.content, true)|trim %}
|
||||
{%- if message.role == "system" or message.role == "developer" %}
|
||||
{{- raise_exception('System message must be at the beginning.') }}
|
||||
{%- elif message.role == "user" %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||
{%- elif message.role == "assistant" %}
|
||||
{%- set reasoning_content = '' %}
|
||||
{%- if message.reasoning_content is string %}
|
||||
{%- set reasoning_content = message.reasoning_content %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = reasoning_content|trim %}
|
||||
{%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
|
||||
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||
{%- endif %}
|
||||
{%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if tool_call.function is defined %}
|
||||
{%- set tool_call = tool_call.function %}
|
||||
{%- endif %}
|
||||
{%- if tool_call.name is not defined or tool_call.name is none %}
|
||||
{{- raise_exception('Tool call is missing a function name.') }}
|
||||
{%- endif %}
|
||||
{%- if loop.first %}
|
||||
{%- if content|trim %}
|
||||
{{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- else %}
|
||||
{{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- if tool_call.arguments is mapping %}
|
||||
{%- for args_name, args_value in tool_call.arguments|items %}
|
||||
{{- '<parameter=' + args_name + '>\n' }}
|
||||
{%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
|
||||
{{- args_value }}
|
||||
{{- '\n</parameter>\n' }}
|
||||
{%- endfor %}
|
||||
{%- elif tool_call.arguments is string %}
|
||||
{%- if tool_call.arguments|trim %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" were passed as a JSON string. Parse them into an object before calling apply_chat_template.') }}
|
||||
{%- endif %}
|
||||
{%- elif tool_call.arguments is defined and tool_call.arguments is not none %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" must be an object/mapping or a JSON string.') }}
|
||||
{%- endif %}
|
||||
{{- '</function>\n</tool_call>' }}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == "tool" %}
|
||||
{%- if loop.previtem and loop.previtem.role != "tool" %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' }}
|
||||
{{- content }}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if not loop.last and loop.nextitem.role != "tool" %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif loop.last %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected message role.') }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- if enable_thinking is defined and enable_thinking is false %}
|
||||
{{- '<think>\n\n</think>\n\n' }}
|
||||
{%- else %}
|
||||
{{- '<think>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{#- Unsloth fixes - developer role, merged system messages, tool calling #}
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
{
|
||||
"protocol_id": "strata-qwen38-thinking-v1-draft.1",
|
||||
"bundle": "standard-v1",
|
||||
"profile": "qwen3.8-flash-next-thinking",
|
||||
"tokenizer_files": {
|
||||
"vocab.json": "4ba64f0332abcfb0b600b7df1537d1e836e68009d5fb7cdb77339738dc6365c4",
|
||||
"merges.txt": "080cf95432173729d160346b7882bd2a644b07f7a3813a3eb73f4e66787bfde9",
|
||||
"token_type.json": "5088c8c298fc06af8382ddb3b76c888703ac2634e82263b97eedcc2ad202738b"
|
||||
},
|
||||
"template": {
|
||||
"path": "serve/chat_template.jinja",
|
||||
"sha256": "ae9644304511ef71f28e036b6f65b4c709880bc5a23b7d0b42125ec63a8c70ca"
|
||||
},
|
||||
"template_kwargs": {
|
||||
"enable_thinking": true,
|
||||
"preserve_thinking": true,
|
||||
"reasoning_effort": "xhigh"
|
||||
},
|
||||
"record_format": "record{:06d}: alpha beta gamma delta epsilon zeta eta theta.",
|
||||
"short_task": "Explain how a hash table finds a value, including collisions and one small example.",
|
||||
"summary_question": "Summarize the structure of these records and explain how to look up a record by its ID.",
|
||||
"record_counts": {
|
||||
"prompt-4k": {
|
||||
"records": 211,
|
||||
"target_tokens": 4096
|
||||
},
|
||||
"prompt-16k": {
|
||||
"records": 858,
|
||||
"target_tokens": 16384
|
||||
}
|
||||
},
|
||||
"length_tolerance": 0.02,
|
||||
"requests": [
|
||||
{
|
||||
"id": "warmup",
|
||||
"input_tokens": 75,
|
||||
"max_shared_prefix_tokens": 47
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r1",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r1",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r1",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r2",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r2",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r2",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r3",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r3",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r3",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "sanity-I1",
|
||||
"input_tokens": 82,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I2",
|
||||
"input_tokens": 90,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I3",
|
||||
"input_tokens": 85,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I4",
|
||||
"input_tokens": 71,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R1",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R2",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R3",
|
||||
"input_tokens": 16382,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R4",
|
||||
"input_tokens": 16390,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-T1",
|
||||
"input_tokens": 325,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T2",
|
||||
"input_tokens": 335,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "sanity-T3",
|
||||
"input_tokens": 352,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T4",
|
||||
"input_tokens": 339,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
}
|
||||
],
|
||||
"note": "input_tokens counts the rendered thinking request with the generation prompt. max_shared_prefix_tokens is the longest token prefix this request shares with any other request of the bundle: cache reuse above it means the user payload was read from the cache."
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
bc6c1f81d86863c3dc36a31f58010701d8ab05180f69f86c6978214409e43af2 build.json
|
||||
f724855774dfbea89fade712a70d534a085c88cda6abc123db419fd79eefab49 requests/concurrency-r1-c1.json
|
||||
2c85f877abbf144ca065c58efc2a7568aaf2c8dc317ad8266e78c149f8245abb requests/concurrency-r1-c2.json
|
||||
264097f61645a0d5089562dc003b7748b1c20339c2df1290ad6664fc1834068a requests/concurrency-r1-c3.json
|
||||
65895c5a5fab53c06e9358d88d921f1bfc9dc7cc8b6e8a2521ac90de080ff470 requests/concurrency-r1-c4.json
|
||||
2c08d50e0c95e345512ec918f58fa716780b79900f09bbfabc37519c8d86543a requests/concurrency-r2-c1.json
|
||||
8aba49693701b9315b2bc5ad55e97a8b03496baedd05f370c49c2c77f45ce6cd requests/concurrency-r2-c2.json
|
||||
7cfb4a5a8fb5702244ce42e0dba9ccd3064d4023d700b409a1a472790a82809c requests/concurrency-r2-c3.json
|
||||
3ab56fbba5fd405945e53cff01b59c3ef966b0f08a0a1b0b50633d26e90cad62 requests/concurrency-r2-c4.json
|
||||
3adb684cf2cb84d4e514de30a4c6f8ccebdd8471160cfcdfc0cdb8ca19ac2137 requests/concurrency-r3-c1.json
|
||||
650379ef9a4da667f6581ae9573eb9af2db2a1fc617701e50536590b18f70077 requests/concurrency-r3-c2.json
|
||||
111415388c5e69ecbd9ab66efe85eb8f3db9142201e1548d03203d3acc05eca1 requests/concurrency-r3-c3.json
|
||||
691a8d27fd06e9beab57e6efb3d82c2a7ed9fc26635fd51b1b1d017117440c1e requests/concurrency-r3-c4.json
|
||||
06776ef38485dc4bb7e18809534fe16a57d421d163f81d4d1532c4b08225968e requests/performance-prompt-16k-r1.json
|
||||
4995c5f3c47c1682f4dc753a9f3a1fba6920fea14691e1ded2da6208ef5fe628 requests/performance-prompt-16k-r2.json
|
||||
7af4c49f99192007be27110296b5a0bcc2efbc91c7ff1c337a6cac6732050559 requests/performance-prompt-16k-r3.json
|
||||
bda0e60c2de0703899ac1bd5711d793b9b325c7f8ba6be6c967f22dd07d8dda8 requests/performance-prompt-4k-r1.json
|
||||
e898eec0ee22499473179a719505c9feb9d3772658833f607dffb34c3727150e requests/performance-prompt-4k-r2.json
|
||||
6d3b3300eace318574490f4379119409920cbe7e88fdaff28eb6f5c79430c3e3 requests/performance-prompt-4k-r3.json
|
||||
483b4f18fcc64ca802fde3377c59bf7b114ec5a5c96fe781bd2be4665ef7b898 requests/performance-short-r1.json
|
||||
b0ec85f40117303d19c8c17a4813625c8554f59caddbe69e7d8aafdcf51d501a requests/performance-short-r2.json
|
||||
e210da20b80c61147da5102fcba65131dafbbb4ceee952fa6b91ce2bb06581ed requests/performance-short-r3.json
|
||||
70ebcd31af5ae1bc7573136d9a257469d8ad9e85b5d0233980998e957c36d6c2 requests/sanity-I1.json
|
||||
1cc6e4bc56b5a22043b6e384b3036170dc6d3edeb403be391a8b802b5613f4cb requests/sanity-I2.json
|
||||
c29b13e6e31b674ff733202b3c0b1f9410768b9ea9368a4a6a13430b3e011fb8 requests/sanity-I3.json
|
||||
780fbd1d02f119323cf50ac4dabcfffbd23dd84115efbe18152e7ee9112b041a requests/sanity-I4.json
|
||||
3c1e0718ad06e591e47cf8df9c36bab2d13332d44b05e440ec17d0402bf9facc requests/sanity-R1.json
|
||||
352e054c48e08e11d339d3a02a14f4f50125071710679b8d04b77aba711fbafc requests/sanity-R2.json
|
||||
36770434f1ea046ec84b32691d644e8416e89fbcbb04b8f6032da66016d78291 requests/sanity-R3.json
|
||||
a63df33c784732cc6b4d8a0f2258b7226c8df5a3b4f6818f9e886bbf09604999 requests/sanity-R4.json
|
||||
6b272ef684b43c1e0004a40b958c807b84dcd9de579d4b2d8819e39ae4196fb2 requests/sanity-T1.json
|
||||
8253ee4aa6b19e61e28d64406807218c9aa10cf2cf682101e8a2d6a50564de46 requests/sanity-T2.json
|
||||
aa2be8761793f51eab6ef04bc54c96e018f6981b7a6a26b9e50cc3c7ac856803 requests/sanity-T3.json
|
||||
04043bdec397df487bebd2999e280a9fae5925e5efec1a10fa96055ec3430faa requests/sanity-T4.json
|
||||
d3365c57db8be1fd0dd9771f74f264b1f6994a396e472c3e5fa4ede2641ffb5c requests/warmup.json
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,52 @@
|
||||
# Standard benchmark runs
|
||||
|
||||
Strata's `standard-v1` suite (`strata.py bench run`, protocol `strata-qwen38-thinking-v1-draft.1`): 3×
|
||||
short / 4K / 16K prompts with a 512-token cap at the Qwen model-card sampling (temperature 1.0, thinking
|
||||
on, effort xhigh), then 12 sanity cases (instruction, retrieval, tool calls). Each folder is the runner's
|
||||
own output: `README.md` (its report), `results.json`, `config.json`, `outputs.jsonl`, `provenance/`.
|
||||
|
||||
All runs 2026-10-26, IQ3_S, engine 0.1.51 at `4b9e05a2`, 250 W cap, gpu-unlock, idle server, run from
|
||||
the guest over loopback.
|
||||
|
||||
## Tuning arms
|
||||
|
||||
Each arm = the setup config ([`strata/strata-iq3_s.json`](../../strata/strata-iq3_s.json)) plus one
|
||||
change, a fresh engine restart, then the suite. Median of 3; decode and prompt are the engine's timings.
|
||||
|
||||
| Arm | Folder | Prompt 4K tok/s | Prompt 16K tok/s | TTFT 16K | Decode short | Decode 4K | Decode 16K |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|
|
||||
| Warm baseline (15 h uptime, + concurrency stage) | `0f07e24d` | 1,776 | 2,938 | 5.61 s | 130.7 | 123.2 | 124.5 |
|
||||
| **Restart baseline** | `87dfe6da` | **1,712** | **2,881** | **5.71 s** | **122.5** | **110.1** | **116.6** |
|
||||
| `--adapt-async 1` | `1ecae1bc` | 1,712 | 2,870 | 5.73 s | 125.1 | 115.0 | 108.6 |
|
||||
| `--pipeline-windows 2` | `5f934909` | 1,705 | 2,878 | 5.71 s | 110.0 | 104.2 | 102.7 |
|
||||
| `--prefill auto:32768` | `92972582` | 1,702 | **2,239** | **7.35 s** | 122.7 | 111.6 | 105.9 |
|
||||
| all three | `53bbc5d8` | 1,699 | 2,243 | 7.34 s | 121.1 | 121.7 | 113.4 |
|
||||
|
||||
All 12 sanity checks passed in every arm. The arms report "nonconforming" only because they used
|
||||
`--skip-model-hash` and `--track custom`. GPU temps peaked at 56 / 64°C (GPU1 ~230 W).
|
||||
|
||||
Four requests at once (warm baseline, server capacity 1): 115.5 / 126.5 / 126.2 tok/s total per round,
|
||||
so concurrent clients just queue.
|
||||
|
||||
## Conclusions
|
||||
|
||||
- **Keep the setup defaults.** None of the three flags helped on this machine.
|
||||
- `--prefill auto:32768` is clearly worse at 16K: −22% prompt speed, +1.6 s to first token. Strata's
|
||||
docs say chunks above 8192 aren't bit-exact on CUDA anyway. The gains reported elsewhere were at 74K
|
||||
prompts on AMD; very long prompts were not tested here.
|
||||
- `--pipeline-windows 2` left prompt speed unchanged and cost decode speed (−10% short, −12% at 16K). The
|
||||
community 2×3090 report's +17% prefill did not reproduce (their cards were x16 + x4; ours are x8 + x8).
|
||||
- `--adapt-async 1` is within noise.
|
||||
- **Noise:** decode varies 94–135 tok/s *within* an arm (sampling at temperature 1.0 changes how many
|
||||
draft tokens are accepted), so decode differences under ~10% between single runs aren't meaningful.
|
||||
Prompt speed is tight (±2%) and is the reliable comparison.
|
||||
- Prompt speed is already above the community 2×3090 report (1,798 tok/s at 19.9K). The 915 tok/s in
|
||||
the earlier notes was a first cold prompt right after load, not a steady-state number.
|
||||
|
||||
## Not tried yet
|
||||
|
||||
- Huge pages for the expert arena: the engine logs `MAP_HUGETLB unavailable (needed 23983 2 MiB pages,
|
||||
vm.nr_hugepages=0)`. That would mean reserving ~47 GB of huge pages in the guest (a NixOS change and a
|
||||
reboot).
|
||||
- Raising the power cap from 250 W toward 300–350 W for prompt speed (gpu-unlock GEMM was +40% at 350 W).
|
||||
- Other quants.
|
||||
@@ -2,6 +2,15 @@
|
||||
|
||||
Newest first. Each entry: what broke, the cause, and what fixed it.
|
||||
|
||||
## 2026-10-26 (night): first standard benchmark and tuning arms
|
||||
|
||||
Ran Strata's `standard-v1` suite on the idle server, then one arm per tuning flag with a fresh restart
|
||||
each. Baseline: 2,881–2,938 prompt tok/s at 16K, decode ~110–131. `--adapt-async 1` neutral,
|
||||
`--pipeline-windows 2` −10–12% decode, `--prefill auto:32768` −22% prompt at 16K. No config change kept;
|
||||
the original `strata-iq3_s.json` was restored (byte-identical, backup at
|
||||
`/var/lib/strata/strata-iq3_s.json.orig-2026-10-26`). Results:
|
||||
[benchmarks/standard](../benchmarks/standard/README.md).
|
||||
|
||||
## 2026-10-26 (evening): compaction loop again, summary snowball
|
||||
|
||||
**Symptom:** the same "generation hit the token cap" loop, now at the 32K cap, retrying every ~4 min
|
||||
|
||||
+5
-2
@@ -74,8 +74,11 @@ The service is `systemd.services.strata` in [`nixos/configuration.nix`](../nixos
|
||||
| 17K-token cold prompt | 915 tok/s prompt, 113 tok/s output; repeats reuse the cached prefix |
|
||||
| Community 2×3090 reference | ~137 decode, 1,800–2,300 prompt tok/s with tuning |
|
||||
|
||||
Untried tuning: `--prefill auto:32768` (setup's tip, +21–35% prompt), `--pipeline-windows 2`,
|
||||
`--adapt-async 1`.
|
||||
**Standard benchmark** (`standard-v1`, idle server, 2026-10-26): prompt 1,712–1,776 tok/s at 4K and
|
||||
2,881–2,938 tok/s at 16K, decode ~110–131 tok/s at temperature 1.0, 12/12 sanity checks. The 915 tok/s
|
||||
above was a first cold prompt right after load. Tuning arms (`--adapt-async 1`, `--pipeline-windows 2`,
|
||||
`--prefill auto:32768`) gave no gain, and the 32K prefill chunk was 22% slower at 16K: see
|
||||
[benchmarks/standard](../benchmarks/standard/README.md). The setup defaults stay.
|
||||
|
||||
Memory at idle with the model loaded: both GPUs ~23.8 of 24.5 GB used, guest ~76 GB RAM used with
|
||||
~33 GB available.
|
||||
|
||||
Reference in New Issue
Block a user