Benchmark power cap 250 W vs 275 W (case closed)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jason Witty
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
d525f315f4
commit
0c519cd9a3
@@ -0,0 +1,77 @@
|
||||
# <user> standard benchmark: NVIDIA GeForce RTX 3090
|
||||
|
||||
Protocol `strata-qwen38-thinking-v1-draft.1` (draft), run `20261027T035636Z-16ea71d0`, started 2026-10-27T03:56:36Z. **Status: nonconforming.**
|
||||
|
||||
Written by `<user> bench run`. The machine-readable result is [results.json](results.json); every request, its raw usage and timings and the client event offsets are in [outputs.jsonl](outputs.jsonl).
|
||||
|
||||
This run is not a complete standard result:
|
||||
|
||||
- no model file could be hashed
|
||||
|
||||
## Hardware and software
|
||||
|
||||
- GPU: NVIDIA GeForce RTX 3090 (24.0 GiB dedicated)
|
||||
- CPU: Intel(R) Core(TM) i9-14900KS; RAM: 110.0 GiB
|
||||
- Storage of the model files: not measured
|
||||
- OS: NixOS 26.05 (Yarara) 6.18.55; drivers: gpu-driver 595.71.05; runtime: cuda 13.2
|
||||
- <user> commit: `4b9e05a2a806a3fe24f7b2560d98626c37dade35`; engine 0.1.51; source build
|
||||
- Background work: not measured
|
||||
|
||||
## Model and configuration
|
||||
|
||||
- Model: qwen3.8-flash-next-iq3_s; quantization IQ3_S; revision not measured; source revision not measured
|
||||
- Files: not measured
|
||||
- Track: **custom** (power cap 275W rep 2, case closed)
|
||||
- Context 131072 tokens; server capacity 1 slot(s)
|
||||
- Sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0, repetition_penalty 1.0, reasoning_effort xhigh; thinking on, preserved
|
||||
- Engine options: `config.json`; chat template: `provenance/chat_template.jinja`
|
||||
|
||||
## Method
|
||||
|
||||
The frozen requests of `bench/prompts/standard-v1` (corpus `e75eff8b9feea04f...`) were sent over loopback HTTP, streaming, one at a time, in this order: 1 warm-up (excluded), 3 repetitions of the short, 4K and 16K prompts, and the twelve sanity checks. Prompt and decode speed are the engine's own `timings`; time to first token is the client's monotonic clock from dispatch to the first nonempty reasoning, answer or tool-call delta. No request was retried. Time after the model loaded: 1 min 35 s.
|
||||
|
||||
## Results
|
||||
|
||||
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| short | n/a | 114.2 (112.4-124.9) n=3 | 0.47 (0.38-0.47) n=3 | 1.17 (0.88-3.13) n=3 | 4.2 (3.7-4.8) n=3 | 0/3 |
|
||||
| prompt-4k | 1743.3 (1568.8-1782.7) n=3 | 111.6 (98.0-112.0) n=3 | 2.37 (2.31-2.63) n=3 | 6.05 (6.05-6.05) n=1 | 6.9 (6.9-7.8) n=3 | 3/3 |
|
||||
| prompt-16k | 3015.5 (2964.3-3019.2) n=3 | 111.0 (102.8-130.2) n=3 | 5.46 (5.45-5.56) n=3 | 7.63 (7.63-7.63) n=1 | 10.1 (9.4-10.5) n=3 | 3/3 |
|
||||
|
||||
Cells are median (min-max) n=samples. Short is a prompt of about 80 tokens: its prompt rate is dominated by the fixed cost of a request, so it is not shown; read prompt speed from prompt-4k and prompt-16k. Capped = requests that reached the 512-token cap, reasoning and answer together. A capped request's total time is the time to the cap, not to an answer, and it has no first-answer time when it was still reasoning.
|
||||
|
||||
| Phase | Planned | Completed | Truncated | Failed | Skipped |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| warmup | 1 | 0 | 1 | 0 | 0 |
|
||||
| performance | 9 | 3 | 6 | 0 | 0 |
|
||||
| sanity | 12 | 12 | 0 | 0 | 0 |
|
||||
| concurrency | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
Sanity checks: **12 of 12**.
|
||||
|
||||
| Case | Result | Reason |
|
||||
| --- | --- | --- |
|
||||
| I1 | pass | the expected JSON value |
|
||||
| I2 | pass | the expected JSON value |
|
||||
| I3 | pass | the expected JSON value |
|
||||
| I4 | pass | the exact expected text |
|
||||
| R1 | pass | the exact expected text |
|
||||
| R2 | pass | the exact expected text |
|
||||
| R3 | pass | the exact expected text |
|
||||
| R4 | pass | the expected JSON value |
|
||||
| T1 | pass | the expected tool call |
|
||||
| T2 | pass | the expected tool call |
|
||||
| T3 | pass | the expected tool call |
|
||||
| T4 | pass | the expected tool call |
|
||||
|
||||
## Failures, exclusions and limitations
|
||||
|
||||
- No request failed.
|
||||
- Excluded from speed summaries: performance-prompt-16k-r1: first_answer_s is not available for this request
|
||||
- Excluded from speed summaries: performance-prompt-16k-r2: first_answer_s is not available for this request
|
||||
- Excluded from speed summaries: performance-prompt-4k-r1: first_answer_s is not available for this request
|
||||
- Excluded from speed summaries: performance-prompt-4k-r3: first_answer_s is not available for this request
|
||||
- Prompt-prefix reuse is the shared template prefix only; the cache was not emptied between requests, and the expert cache stays warm and evolves. These are not cold-model measurements.
|
||||
- Output caps include reasoning; a request that reaches its cap is `truncated` and fails its sanity check.
|
||||
- The runner cannot see other programs on the machine, the repeat-stop and loop-recovery switches of the server, or the quantization repository's revision; see `missing` in results.json for every value it could not read.
|
||||
- The server had answered no request before this run (a freshly loaded engine).
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"engine_options": null,
|
||||
"other_engine_options": {},
|
||||
"server_defaults": {
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
"top_k": 20,
|
||||
"n_predict": -1
|
||||
},
|
||||
"total_slots": 1,
|
||||
"context": 131072,
|
||||
"source": "the server was already running; its engine options are not exposed"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
+184
@@ -0,0 +1,184 @@
|
||||
{%- set image_count = namespace(value=0) %}
|
||||
{%- set video_count = namespace(value=0) %}
|
||||
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
||||
{%- if content is string %}
|
||||
{{- content }}
|
||||
{%- elif content is iterable and content is not mapping %}
|
||||
{%- for item in content %}
|
||||
{%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain images.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set image_count.value = image_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
||||
{%- elif 'video' in item or item.type == 'video' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain videos.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set video_count.value = video_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
||||
{%- elif 'text' in item %}
|
||||
{{- item.text }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected item type in content.') }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- elif content is none or content is undefined %}
|
||||
{{- '' }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected content type.') }}
|
||||
{%- endif %}
|
||||
{%- endmacro %}
|
||||
{%- if not messages %}
|
||||
{{- raise_exception('No messages provided.') }}
|
||||
{%- endif %}
|
||||
{%- set sysns = namespace(count=0, text='') %}
|
||||
{%- for message in messages %}
|
||||
{%- if sysns.count == loop.index0 and (message.role == 'system' or message.role == 'developer') %}
|
||||
{%- set sys_content = render_content(message.content, false, true)|trim %}
|
||||
{%- if sys_content %}
|
||||
{%- set sysns.text = sysns.text + ('\n' if sysns.text else '') + sys_content %}
|
||||
{%- endif %}
|
||||
{%- set sysns.count = sysns.count + 1 %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set num_sys = sysns.count %}
|
||||
{%- set merged_system = sysns.text %}
|
||||
{%- set reasoning_instructions = '' %}
|
||||
{%- if enable_thinking is undefined or enable_thinking is true %}
|
||||
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
|
||||
{%- if resolved_reasoning_effort == 'high' %}
|
||||
{%- set resolved_reasoning_effort = 'xhigh' %}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
|
||||
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort == 'xhigh' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
|
||||
{%- elif resolved_reasoning_effort == 'low' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if tools and tools is iterable and tools is not mapping %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if reasoning_instructions %}
|
||||
{{- reasoning_instructions + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
|
||||
{%- for tool in tools %}
|
||||
{{- "\n" }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- "\n</tools>" }}
|
||||
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
||||
{%- if merged_system %}
|
||||
{{- '\n\n' + merged_system }}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{%- if merged_system %}
|
||||
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + merged_system + '<|im_end|>\n' }}
|
||||
{%- elif reasoning_instructions %}
|
||||
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
||||
{%- for message in messages[::-1] %}
|
||||
{%- set index = (messages|length - 1) - loop.index0 %}
|
||||
{%- if ns.multi_step_tool and message.role == "user" %}
|
||||
{%- set content = render_content(message.content, false)|trim %}
|
||||
{%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
|
||||
{%- set ns.multi_step_tool = false %}
|
||||
{%- set ns.last_query_index = index %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- for message in messages %}
|
||||
{%- if loop.index0 >= num_sys %}
|
||||
{%- set content = render_content(message.content, true)|trim %}
|
||||
{%- if message.role == "system" or message.role == "developer" %}
|
||||
{{- raise_exception('System message must be at the beginning.') }}
|
||||
{%- elif message.role == "user" %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||
{%- elif message.role == "assistant" %}
|
||||
{%- set reasoning_content = '' %}
|
||||
{%- if message.reasoning_content is string %}
|
||||
{%- set reasoning_content = message.reasoning_content %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = reasoning_content|trim %}
|
||||
{%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
|
||||
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||
{%- endif %}
|
||||
{%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if tool_call.function is defined %}
|
||||
{%- set tool_call = tool_call.function %}
|
||||
{%- endif %}
|
||||
{%- if tool_call.name is not defined or tool_call.name is none %}
|
||||
{{- raise_exception('Tool call is missing a function name.') }}
|
||||
{%- endif %}
|
||||
{%- if loop.first %}
|
||||
{%- if content|trim %}
|
||||
{{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- else %}
|
||||
{{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- if tool_call.arguments is mapping %}
|
||||
{%- for args_name, args_value in tool_call.arguments|items %}
|
||||
{{- '<parameter=' + args_name + '>\n' }}
|
||||
{%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
|
||||
{{- args_value }}
|
||||
{{- '\n</parameter>\n' }}
|
||||
{%- endfor %}
|
||||
{%- elif tool_call.arguments is string %}
|
||||
{%- if tool_call.arguments|trim %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" were passed as a JSON string. Parse them into an object before calling apply_chat_template.') }}
|
||||
{%- endif %}
|
||||
{%- elif tool_call.arguments is defined and tool_call.arguments is not none %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" must be an object/mapping or a JSON string.') }}
|
||||
{%- endif %}
|
||||
{{- '</function>\n</tool_call>' }}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == "tool" %}
|
||||
{%- if loop.previtem and loop.previtem.role != "tool" %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' }}
|
||||
{{- content }}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if not loop.last and loop.nextitem.role != "tool" %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif loop.last %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected message role.') }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- if enable_thinking is defined and enable_thinking is false %}
|
||||
{{- '<think>\n\n</think>\n\n' }}
|
||||
{%- else %}
|
||||
{{- '<think>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{#- Unsloth fixes - developer role, merged system messages, tool calling #}
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
{
|
||||
"protocol_id": "strata-qwen38-thinking-v1-draft.1",
|
||||
"bundle": "standard-v1",
|
||||
"profile": "qwen3.8-flash-next-thinking",
|
||||
"tokenizer_files": {
|
||||
"vocab.json": "4ba64f0332abcfb0b600b7df1537d1e836e68009d5fb7cdb77339738dc6365c4",
|
||||
"merges.txt": "080cf95432173729d160346b7882bd2a644b07f7a3813a3eb73f4e66787bfde9",
|
||||
"token_type.json": "5088c8c298fc06af8382ddb3b76c888703ac2634e82263b97eedcc2ad202738b"
|
||||
},
|
||||
"template": {
|
||||
"path": "serve/chat_template.jinja",
|
||||
"sha256": "ae9644304511ef71f28e036b6f65b4c709880bc5a23b7d0b42125ec63a8c70ca"
|
||||
},
|
||||
"template_kwargs": {
|
||||
"enable_thinking": true,
|
||||
"preserve_thinking": true,
|
||||
"reasoning_effort": "xhigh"
|
||||
},
|
||||
"record_format": "record{:06d}: alpha beta gamma delta epsilon zeta eta theta.",
|
||||
"short_task": "Explain how a hash table finds a value, including collisions and one small example.",
|
||||
"summary_question": "Summarize the structure of these records and explain how to look up a record by its ID.",
|
||||
"record_counts": {
|
||||
"prompt-4k": {
|
||||
"records": 211,
|
||||
"target_tokens": 4096
|
||||
},
|
||||
"prompt-16k": {
|
||||
"records": 858,
|
||||
"target_tokens": 16384
|
||||
}
|
||||
},
|
||||
"length_tolerance": 0.02,
|
||||
"requests": [
|
||||
{
|
||||
"id": "warmup",
|
||||
"input_tokens": 75,
|
||||
"max_shared_prefix_tokens": 47
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r1",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r1",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r1",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r2",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r2",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r2",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r3",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r3",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r3",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "sanity-I1",
|
||||
"input_tokens": 82,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I2",
|
||||
"input_tokens": 90,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I3",
|
||||
"input_tokens": 85,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I4",
|
||||
"input_tokens": 71,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R1",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R2",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R3",
|
||||
"input_tokens": 16382,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R4",
|
||||
"input_tokens": 16390,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-T1",
|
||||
"input_tokens": 325,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T2",
|
||||
"input_tokens": 335,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "sanity-T3",
|
||||
"input_tokens": 352,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T4",
|
||||
"input_tokens": 339,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
}
|
||||
],
|
||||
"note": "input_tokens counts the rendered thinking request with the generation prompt. max_shared_prefix_tokens is the longest token prefix this request shares with any other request of the bundle: cache reuse above it means the user payload was read from the cache."
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
bc6c1f81d86863c3dc36a31f58010701d8ab05180f69f86c6978214409e43af2 build.json
|
||||
f724855774dfbea89fade712a70d534a085c88cda6abc123db419fd79eefab49 requests/concurrency-r1-c1.json
|
||||
2c85f877abbf144ca065c58efc2a7568aaf2c8dc317ad8266e78c149f8245abb requests/concurrency-r1-c2.json
|
||||
264097f61645a0d5089562dc003b7748b1c20339c2df1290ad6664fc1834068a requests/concurrency-r1-c3.json
|
||||
65895c5a5fab53c06e9358d88d921f1bfc9dc7cc8b6e8a2521ac90de080ff470 requests/concurrency-r1-c4.json
|
||||
2c08d50e0c95e345512ec918f58fa716780b79900f09bbfabc37519c8d86543a requests/concurrency-r2-c1.json
|
||||
8aba49693701b9315b2bc5ad55e97a8b03496baedd05f370c49c2c77f45ce6cd requests/concurrency-r2-c2.json
|
||||
7cfb4a5a8fb5702244ce42e0dba9ccd3064d4023d700b409a1a472790a82809c requests/concurrency-r2-c3.json
|
||||
3ab56fbba5fd405945e53cff01b59c3ef966b0f08a0a1b0b50633d26e90cad62 requests/concurrency-r2-c4.json
|
||||
3adb684cf2cb84d4e514de30a4c6f8ccebdd8471160cfcdfc0cdb8ca19ac2137 requests/concurrency-r3-c1.json
|
||||
650379ef9a4da667f6581ae9573eb9af2db2a1fc617701e50536590b18f70077 requests/concurrency-r3-c2.json
|
||||
111415388c5e69ecbd9ab66efe85eb8f3db9142201e1548d03203d3acc05eca1 requests/concurrency-r3-c3.json
|
||||
691a8d27fd06e9beab57e6efb3d82c2a7ed9fc26635fd51b1b1d017117440c1e requests/concurrency-r3-c4.json
|
||||
06776ef38485dc4bb7e18809534fe16a57d421d163f81d4d1532c4b08225968e requests/performance-prompt-16k-r1.json
|
||||
4995c5f3c47c1682f4dc753a9f3a1fba6920fea14691e1ded2da6208ef5fe628 requests/performance-prompt-16k-r2.json
|
||||
7af4c49f99192007be27110296b5a0bcc2efbc91c7ff1c337a6cac6732050559 requests/performance-prompt-16k-r3.json
|
||||
bda0e60c2de0703899ac1bd5711d793b9b325c7f8ba6be6c967f22dd07d8dda8 requests/performance-prompt-4k-r1.json
|
||||
e898eec0ee22499473179a719505c9feb9d3772658833f607dffb34c3727150e requests/performance-prompt-4k-r2.json
|
||||
6d3b3300eace318574490f4379119409920cbe7e88fdaff28eb6f5c79430c3e3 requests/performance-prompt-4k-r3.json
|
||||
483b4f18fcc64ca802fde3377c59bf7b114ec5a5c96fe781bd2be4665ef7b898 requests/performance-short-r1.json
|
||||
b0ec85f40117303d19c8c17a4813625c8554f59caddbe69e7d8aafdcf51d501a requests/performance-short-r2.json
|
||||
e210da20b80c61147da5102fcba65131dafbbb4ceee952fa6b91ce2bb06581ed requests/performance-short-r3.json
|
||||
70ebcd31af5ae1bc7573136d9a257469d8ad9e85b5d0233980998e957c36d6c2 requests/sanity-I1.json
|
||||
1cc6e4bc56b5a22043b6e384b3036170dc6d3edeb403be391a8b802b5613f4cb requests/sanity-I2.json
|
||||
c29b13e6e31b674ff733202b3c0b1f9410768b9ea9368a4a6a13430b3e011fb8 requests/sanity-I3.json
|
||||
780fbd1d02f119323cf50ac4dabcfffbd23dd84115efbe18152e7ee9112b041a requests/sanity-I4.json
|
||||
3c1e0718ad06e591e47cf8df9c36bab2d13332d44b05e440ec17d0402bf9facc requests/sanity-R1.json
|
||||
352e054c48e08e11d339d3a02a14f4f50125071710679b8d04b77aba711fbafc requests/sanity-R2.json
|
||||
36770434f1ea046ec84b32691d644e8416e89fbcbb04b8f6032da66016d78291 requests/sanity-R3.json
|
||||
a63df33c784732cc6b4d8a0f2258b7226c8df5a3b4f6818f9e886bbf09604999 requests/sanity-R4.json
|
||||
6b272ef684b43c1e0004a40b958c807b84dcd9de579d4b2d8819e39ae4196fb2 requests/sanity-T1.json
|
||||
8253ee4aa6b19e61e28d64406807218c9aa10cf2cf682101e8a2d6a50564de46 requests/sanity-T2.json
|
||||
aa2be8761793f51eab6ef04bc54c96e018f6981b7a6a26b9e50cc3c7ac856803 requests/sanity-T3.json
|
||||
04043bdec397df487bebd2999e280a9fae5925e5efec1a10fa96055ec3430faa requests/sanity-T4.json
|
||||
d3365c57db8be1fd0dd9771f74f264b1f6994a396e472c3e5fa4ede2641ffb5c requests/warmup.json
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,75 @@
|
||||
# <user> standard benchmark: NVIDIA GeForce RTX 3090
|
||||
|
||||
Protocol `strata-qwen38-thinking-v1-draft.1` (draft), run `20261027T034915Z-5c64d992`, started 2026-10-27T03:49:15Z. **Status: nonconforming.**
|
||||
|
||||
Written by `<user> bench run`. The machine-readable result is [results.json](results.json); every request, its raw usage and timings and the client event offsets are in [outputs.jsonl](outputs.jsonl).
|
||||
|
||||
This run is not a complete standard result:
|
||||
|
||||
- no model file could be hashed
|
||||
|
||||
## Hardware and software
|
||||
|
||||
- GPU: NVIDIA GeForce RTX 3090 (24.0 GiB dedicated)
|
||||
- CPU: Intel(R) Core(TM) i9-14900KS; RAM: 110.0 GiB
|
||||
- Storage of the model files: not measured
|
||||
- OS: NixOS 26.05 (Yarara) 6.18.55; drivers: gpu-driver 595.71.05; runtime: cuda 13.2
|
||||
- <user> commit: `4b9e05a2a806a3fe24f7b2560d98626c37dade35`; engine 0.1.51; source build
|
||||
- Background work: not measured
|
||||
|
||||
## Model and configuration
|
||||
|
||||
- Model: qwen3.8-flash-next-iq3_s; quantization IQ3_S; revision not measured; source revision not measured
|
||||
- Files: not measured
|
||||
- Track: **custom** (power cap 275W rep 1, case closed)
|
||||
- Context 131072 tokens; server capacity 1 slot(s)
|
||||
- Sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0, repetition_penalty 1.0, reasoning_effort xhigh; thinking on, preserved
|
||||
- Engine options: `config.json`; chat template: `provenance/chat_template.jinja`
|
||||
|
||||
## Method
|
||||
|
||||
The frozen requests of `bench/prompts/standard-v1` (corpus `e75eff8b9feea04f...`) were sent over loopback HTTP, streaming, one at a time, in this order: 1 warm-up (excluded), 3 repetitions of the short, 4K and 16K prompts, and the twelve sanity checks. Prompt and decode speed are the engine's own `timings`; time to first token is the client's monotonic clock from dispatch to the first nonempty reasoning, answer or tool-call delta. No request was retried. Time after the model loaded: 1 min 33 s.
|
||||
|
||||
## Results
|
||||
|
||||
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| short | n/a | 125.3 (112.1-134.3) n=3 | 0.47 (0.38-0.47) n=3 | 1.05 (0.88-1.43) n=3 | 4.2 (2.9-4.3) n=3 | 1/3 |
|
||||
| prompt-4k | 1742.3 (1571.6-1783.6) n=3 | 115.3 (109.9-117.2) n=3 | 2.37 (2.31-2.62) n=3 | 5.44 (5.41-6.08) n=3 | 6.8 (6.7-7.3) n=3 | 3/3 |
|
||||
| prompt-16k | 3030.3 (2988.3-3036.1) n=3 | 116.1 (100.3-120.7) n=3 | 5.44 (5.43-5.51) n=3 | 8.93 (8.93-8.93) n=1 | 9.8 (9.7-10.6) n=3 | 3/3 |
|
||||
|
||||
Cells are median (min-max) n=samples. Short is a prompt of about 80 tokens: its prompt rate is dominated by the fixed cost of a request, so it is not shown; read prompt speed from prompt-4k and prompt-16k. Capped = requests that reached the 512-token cap, reasoning and answer together. A capped request's total time is the time to the cap, not to an answer, and it has no first-answer time when it was still reasoning.
|
||||
|
||||
| Phase | Planned | Completed | Truncated | Failed | Skipped |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| warmup | 1 | 0 | 1 | 0 | 0 |
|
||||
| performance | 9 | 2 | 7 | 0 | 0 |
|
||||
| sanity | 12 | 12 | 0 | 0 | 0 |
|
||||
| concurrency | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
Sanity checks: **12 of 12**.
|
||||
|
||||
| Case | Result | Reason |
|
||||
| --- | --- | --- |
|
||||
| I1 | pass | the expected JSON value |
|
||||
| I2 | pass | the expected JSON value |
|
||||
| I3 | pass | the expected JSON value |
|
||||
| I4 | pass | the exact expected text |
|
||||
| R1 | pass | the exact expected text |
|
||||
| R2 | pass | the exact expected text |
|
||||
| R3 | pass | the exact expected text |
|
||||
| R4 | pass | the expected JSON value |
|
||||
| T1 | pass | the expected tool call |
|
||||
| T2 | pass | the expected tool call |
|
||||
| T3 | pass | the expected tool call |
|
||||
| T4 | pass | the expected tool call |
|
||||
|
||||
## Failures, exclusions and limitations
|
||||
|
||||
- No request failed.
|
||||
- Excluded from speed summaries: performance-prompt-16k-r1: first_answer_s is not available for this request
|
||||
- Excluded from speed summaries: performance-prompt-16k-r3: first_answer_s is not available for this request
|
||||
- Prompt-prefix reuse is the shared template prefix only; the cache was not emptied between requests, and the expert cache stays warm and evolves. These are not cold-model measurements.
|
||||
- Output caps include reasoning; a request that reaches its cap is `truncated` and fails its sanity check.
|
||||
- The runner cannot see other programs on the machine, the repeat-stop and loop-recovery switches of the server, or the quantization repository's revision; see `missing` in results.json for every value it could not read.
|
||||
- The server had answered no request before this run (a freshly loaded engine).
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"engine_options": null,
|
||||
"other_engine_options": {},
|
||||
"server_defaults": {
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
"top_k": 20,
|
||||
"n_predict": -1
|
||||
},
|
||||
"total_slots": 1,
|
||||
"context": 131072,
|
||||
"source": "the server was already running; its engine options are not exposed"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
+184
@@ -0,0 +1,184 @@
|
||||
{%- set image_count = namespace(value=0) %}
|
||||
{%- set video_count = namespace(value=0) %}
|
||||
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
||||
{%- if content is string %}
|
||||
{{- content }}
|
||||
{%- elif content is iterable and content is not mapping %}
|
||||
{%- for item in content %}
|
||||
{%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain images.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set image_count.value = image_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
||||
{%- elif 'video' in item or item.type == 'video' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain videos.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set video_count.value = video_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
||||
{%- elif 'text' in item %}
|
||||
{{- item.text }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected item type in content.') }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- elif content is none or content is undefined %}
|
||||
{{- '' }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected content type.') }}
|
||||
{%- endif %}
|
||||
{%- endmacro %}
|
||||
{%- if not messages %}
|
||||
{{- raise_exception('No messages provided.') }}
|
||||
{%- endif %}
|
||||
{%- set sysns = namespace(count=0, text='') %}
|
||||
{%- for message in messages %}
|
||||
{%- if sysns.count == loop.index0 and (message.role == 'system' or message.role == 'developer') %}
|
||||
{%- set sys_content = render_content(message.content, false, true)|trim %}
|
||||
{%- if sys_content %}
|
||||
{%- set sysns.text = sysns.text + ('\n' if sysns.text else '') + sys_content %}
|
||||
{%- endif %}
|
||||
{%- set sysns.count = sysns.count + 1 %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set num_sys = sysns.count %}
|
||||
{%- set merged_system = sysns.text %}
|
||||
{%- set reasoning_instructions = '' %}
|
||||
{%- if enable_thinking is undefined or enable_thinking is true %}
|
||||
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
|
||||
{%- if resolved_reasoning_effort == 'high' %}
|
||||
{%- set resolved_reasoning_effort = 'xhigh' %}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
|
||||
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort == 'xhigh' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
|
||||
{%- elif resolved_reasoning_effort == 'low' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if tools and tools is iterable and tools is not mapping %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if reasoning_instructions %}
|
||||
{{- reasoning_instructions + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
|
||||
{%- for tool in tools %}
|
||||
{{- "\n" }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- "\n</tools>" }}
|
||||
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
||||
{%- if merged_system %}
|
||||
{{- '\n\n' + merged_system }}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{%- if merged_system %}
|
||||
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + merged_system + '<|im_end|>\n' }}
|
||||
{%- elif reasoning_instructions %}
|
||||
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
||||
{%- for message in messages[::-1] %}
|
||||
{%- set index = (messages|length - 1) - loop.index0 %}
|
||||
{%- if ns.multi_step_tool and message.role == "user" %}
|
||||
{%- set content = render_content(message.content, false)|trim %}
|
||||
{%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
|
||||
{%- set ns.multi_step_tool = false %}
|
||||
{%- set ns.last_query_index = index %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- for message in messages %}
|
||||
{%- if loop.index0 >= num_sys %}
|
||||
{%- set content = render_content(message.content, true)|trim %}
|
||||
{%- if message.role == "system" or message.role == "developer" %}
|
||||
{{- raise_exception('System message must be at the beginning.') }}
|
||||
{%- elif message.role == "user" %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||
{%- elif message.role == "assistant" %}
|
||||
{%- set reasoning_content = '' %}
|
||||
{%- if message.reasoning_content is string %}
|
||||
{%- set reasoning_content = message.reasoning_content %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = reasoning_content|trim %}
|
||||
{%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
|
||||
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||
{%- endif %}
|
||||
{%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if tool_call.function is defined %}
|
||||
{%- set tool_call = tool_call.function %}
|
||||
{%- endif %}
|
||||
{%- if tool_call.name is not defined or tool_call.name is none %}
|
||||
{{- raise_exception('Tool call is missing a function name.') }}
|
||||
{%- endif %}
|
||||
{%- if loop.first %}
|
||||
{%- if content|trim %}
|
||||
{{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- else %}
|
||||
{{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- if tool_call.arguments is mapping %}
|
||||
{%- for args_name, args_value in tool_call.arguments|items %}
|
||||
{{- '<parameter=' + args_name + '>\n' }}
|
||||
{%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
|
||||
{{- args_value }}
|
||||
{{- '\n</parameter>\n' }}
|
||||
{%- endfor %}
|
||||
{%- elif tool_call.arguments is string %}
|
||||
{%- if tool_call.arguments|trim %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" were passed as a JSON string. Parse them into an object before calling apply_chat_template.') }}
|
||||
{%- endif %}
|
||||
{%- elif tool_call.arguments is defined and tool_call.arguments is not none %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" must be an object/mapping or a JSON string.') }}
|
||||
{%- endif %}
|
||||
{{- '</function>\n</tool_call>' }}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == "tool" %}
|
||||
{%- if loop.previtem and loop.previtem.role != "tool" %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' }}
|
||||
{{- content }}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if not loop.last and loop.nextitem.role != "tool" %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif loop.last %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected message role.') }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- if enable_thinking is defined and enable_thinking is false %}
|
||||
{{- '<think>\n\n</think>\n\n' }}
|
||||
{%- else %}
|
||||
{{- '<think>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{#- Unsloth fixes - developer role, merged system messages, tool calling #}
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
{
|
||||
"protocol_id": "strata-qwen38-thinking-v1-draft.1",
|
||||
"bundle": "standard-v1",
|
||||
"profile": "qwen3.8-flash-next-thinking",
|
||||
"tokenizer_files": {
|
||||
"vocab.json": "4ba64f0332abcfb0b600b7df1537d1e836e68009d5fb7cdb77339738dc6365c4",
|
||||
"merges.txt": "080cf95432173729d160346b7882bd2a644b07f7a3813a3eb73f4e66787bfde9",
|
||||
"token_type.json": "5088c8c298fc06af8382ddb3b76c888703ac2634e82263b97eedcc2ad202738b"
|
||||
},
|
||||
"template": {
|
||||
"path": "serve/chat_template.jinja",
|
||||
"sha256": "ae9644304511ef71f28e036b6f65b4c709880bc5a23b7d0b42125ec63a8c70ca"
|
||||
},
|
||||
"template_kwargs": {
|
||||
"enable_thinking": true,
|
||||
"preserve_thinking": true,
|
||||
"reasoning_effort": "xhigh"
|
||||
},
|
||||
"record_format": "record{:06d}: alpha beta gamma delta epsilon zeta eta theta.",
|
||||
"short_task": "Explain how a hash table finds a value, including collisions and one small example.",
|
||||
"summary_question": "Summarize the structure of these records and explain how to look up a record by its ID.",
|
||||
"record_counts": {
|
||||
"prompt-4k": {
|
||||
"records": 211,
|
||||
"target_tokens": 4096
|
||||
},
|
||||
"prompt-16k": {
|
||||
"records": 858,
|
||||
"target_tokens": 16384
|
||||
}
|
||||
},
|
||||
"length_tolerance": 0.02,
|
||||
"requests": [
|
||||
{
|
||||
"id": "warmup",
|
||||
"input_tokens": 75,
|
||||
"max_shared_prefix_tokens": 47
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r1",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r1",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r1",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r2",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r2",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r2",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r3",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r3",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r3",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "sanity-I1",
|
||||
"input_tokens": 82,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I2",
|
||||
"input_tokens": 90,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I3",
|
||||
"input_tokens": 85,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I4",
|
||||
"input_tokens": 71,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R1",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R2",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R3",
|
||||
"input_tokens": 16382,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R4",
|
||||
"input_tokens": 16390,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-T1",
|
||||
"input_tokens": 325,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T2",
|
||||
"input_tokens": 335,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "sanity-T3",
|
||||
"input_tokens": 352,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T4",
|
||||
"input_tokens": 339,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
}
|
||||
],
|
||||
"note": "input_tokens counts the rendered thinking request with the generation prompt. max_shared_prefix_tokens is the longest token prefix this request shares with any other request of the bundle: cache reuse above it means the user payload was read from the cache."
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
bc6c1f81d86863c3dc36a31f58010701d8ab05180f69f86c6978214409e43af2 build.json
|
||||
f724855774dfbea89fade712a70d534a085c88cda6abc123db419fd79eefab49 requests/concurrency-r1-c1.json
|
||||
2c85f877abbf144ca065c58efc2a7568aaf2c8dc317ad8266e78c149f8245abb requests/concurrency-r1-c2.json
|
||||
264097f61645a0d5089562dc003b7748b1c20339c2df1290ad6664fc1834068a requests/concurrency-r1-c3.json
|
||||
65895c5a5fab53c06e9358d88d921f1bfc9dc7cc8b6e8a2521ac90de080ff470 requests/concurrency-r1-c4.json
|
||||
2c08d50e0c95e345512ec918f58fa716780b79900f09bbfabc37519c8d86543a requests/concurrency-r2-c1.json
|
||||
8aba49693701b9315b2bc5ad55e97a8b03496baedd05f370c49c2c77f45ce6cd requests/concurrency-r2-c2.json
|
||||
7cfb4a5a8fb5702244ce42e0dba9ccd3064d4023d700b409a1a472790a82809c requests/concurrency-r2-c3.json
|
||||
3ab56fbba5fd405945e53cff01b59c3ef966b0f08a0a1b0b50633d26e90cad62 requests/concurrency-r2-c4.json
|
||||
3adb684cf2cb84d4e514de30a4c6f8ccebdd8471160cfcdfc0cdb8ca19ac2137 requests/concurrency-r3-c1.json
|
||||
650379ef9a4da667f6581ae9573eb9af2db2a1fc617701e50536590b18f70077 requests/concurrency-r3-c2.json
|
||||
111415388c5e69ecbd9ab66efe85eb8f3db9142201e1548d03203d3acc05eca1 requests/concurrency-r3-c3.json
|
||||
691a8d27fd06e9beab57e6efb3d82c2a7ed9fc26635fd51b1b1d017117440c1e requests/concurrency-r3-c4.json
|
||||
06776ef38485dc4bb7e18809534fe16a57d421d163f81d4d1532c4b08225968e requests/performance-prompt-16k-r1.json
|
||||
4995c5f3c47c1682f4dc753a9f3a1fba6920fea14691e1ded2da6208ef5fe628 requests/performance-prompt-16k-r2.json
|
||||
7af4c49f99192007be27110296b5a0bcc2efbc91c7ff1c337a6cac6732050559 requests/performance-prompt-16k-r3.json
|
||||
bda0e60c2de0703899ac1bd5711d793b9b325c7f8ba6be6c967f22dd07d8dda8 requests/performance-prompt-4k-r1.json
|
||||
e898eec0ee22499473179a719505c9feb9d3772658833f607dffb34c3727150e requests/performance-prompt-4k-r2.json
|
||||
6d3b3300eace318574490f4379119409920cbe7e88fdaff28eb6f5c79430c3e3 requests/performance-prompt-4k-r3.json
|
||||
483b4f18fcc64ca802fde3377c59bf7b114ec5a5c96fe781bd2be4665ef7b898 requests/performance-short-r1.json
|
||||
b0ec85f40117303d19c8c17a4813625c8554f59caddbe69e7d8aafdcf51d501a requests/performance-short-r2.json
|
||||
e210da20b80c61147da5102fcba65131dafbbb4ceee952fa6b91ce2bb06581ed requests/performance-short-r3.json
|
||||
70ebcd31af5ae1bc7573136d9a257469d8ad9e85b5d0233980998e957c36d6c2 requests/sanity-I1.json
|
||||
1cc6e4bc56b5a22043b6e384b3036170dc6d3edeb403be391a8b802b5613f4cb requests/sanity-I2.json
|
||||
c29b13e6e31b674ff733202b3c0b1f9410768b9ea9368a4a6a13430b3e011fb8 requests/sanity-I3.json
|
||||
780fbd1d02f119323cf50ac4dabcfffbd23dd84115efbe18152e7ee9112b041a requests/sanity-I4.json
|
||||
3c1e0718ad06e591e47cf8df9c36bab2d13332d44b05e440ec17d0402bf9facc requests/sanity-R1.json
|
||||
352e054c48e08e11d339d3a02a14f4f50125071710679b8d04b77aba711fbafc requests/sanity-R2.json
|
||||
36770434f1ea046ec84b32691d644e8416e89fbcbb04b8f6032da66016d78291 requests/sanity-R3.json
|
||||
a63df33c784732cc6b4d8a0f2258b7226c8df5a3b4f6818f9e886bbf09604999 requests/sanity-R4.json
|
||||
6b272ef684b43c1e0004a40b958c807b84dcd9de579d4b2d8819e39ae4196fb2 requests/sanity-T1.json
|
||||
8253ee4aa6b19e61e28d64406807218c9aa10cf2cf682101e8a2d6a50564de46 requests/sanity-T2.json
|
||||
aa2be8761793f51eab6ef04bc54c96e018f6981b7a6a26b9e50cc3c7ac856803 requests/sanity-T3.json
|
||||
04043bdec397df487bebd2999e280a9fae5925e5efec1a10fa96055ec3430faa requests/sanity-T4.json
|
||||
d3365c57db8be1fd0dd9771f74f264b1f6994a396e472c3e5fa4ede2641ffb5c requests/warmup.json
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,75 @@
|
||||
# <user> standard benchmark: NVIDIA GeForce RTX 3090
|
||||
|
||||
Protocol `strata-qwen38-thinking-v1-draft.1` (draft), run `20261027T034528Z-6a05efdc`, started 2026-10-27T03:45:28Z. **Status: nonconforming.**
|
||||
|
||||
Written by `<user> bench run`. The machine-readable result is [results.json](results.json); every request, its raw usage and timings and the client event offsets are in [outputs.jsonl](outputs.jsonl).
|
||||
|
||||
This run is not a complete standard result:
|
||||
|
||||
- no model file could be hashed
|
||||
|
||||
## Hardware and software
|
||||
|
||||
- GPU: NVIDIA GeForce RTX 3090 (24.0 GiB dedicated)
|
||||
- CPU: Intel(R) Core(TM) i9-14900KS; RAM: 110.0 GiB
|
||||
- Storage of the model files: not measured
|
||||
- OS: NixOS 26.05 (Yarara) 6.18.55; drivers: gpu-driver 595.71.05; runtime: cuda 13.2
|
||||
- <user> commit: `4b9e05a2a806a3fe24f7b2560d98626c37dade35`; engine 0.1.51; source build
|
||||
- Background work: not measured
|
||||
|
||||
## Model and configuration
|
||||
|
||||
- Model: qwen3.8-flash-next-iq3_s; quantization IQ3_S; revision not measured; source revision not measured
|
||||
- Files: not measured
|
||||
- Track: **custom** (power cap 250W rep 1, case closed)
|
||||
- Context 131072 tokens; server capacity 1 slot(s)
|
||||
- Sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0, repetition_penalty 1.0, reasoning_effort xhigh; thinking on, preserved
|
||||
- Engine options: `config.json`; chat template: `provenance/chat_template.jinja`
|
||||
|
||||
## Method
|
||||
|
||||
The frozen requests of `bench/prompts/standard-v1` (corpus `e75eff8b9feea04f...`) were sent over loopback HTTP, streaming, one at a time, in this order: 1 warm-up (excluded), 3 repetitions of the short, 4K and 16K prompts, and the twelve sanity checks. Prompt and decode speed are the engine's own `timings`; time to first token is the client's monotonic clock from dispatch to the first nonempty reasoning, answer or tool-call delta. No request was retried. Time after the model loaded: 1 min 36 s.
|
||||
|
||||
## Results
|
||||
|
||||
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| short | n/a | 122.9 (122.0-123.8) n=3 | 0.46 (0.38-0.49) n=3 | 1.02 (0.88-1.37) n=3 | 4.0 (3.1-4.2) n=3 | 0/3 |
|
||||
| prompt-4k | 1718.7 (1558.8-1758.8) n=3 | 112.9 (110.3-125.1) n=3 | 2.40 (2.35-2.65) n=3 | 6.09 (5.14-6.36) n=3 | 6.9 (6.5-7.3) n=3 | 3/3 |
|
||||
| prompt-16k | 2896.8 (2864.8-2909.5) n=3 | 108.2 (99.8-109.8) n=3 | 5.69 (5.65-5.74) n=3 | 10.00 (10.00-10.00) n=1 | 10.4 (10.3-10.9) n=3 | 3/3 |
|
||||
|
||||
Cells are median (min-max) n=samples. Short is a prompt of about 80 tokens: its prompt rate is dominated by the fixed cost of a request, so it is not shown; read prompt speed from prompt-4k and prompt-16k. Capped = requests that reached the 512-token cap, reasoning and answer together. A capped request's total time is the time to the cap, not to an answer, and it has no first-answer time when it was still reasoning.
|
||||
|
||||
| Phase | Planned | Completed | Truncated | Failed | Skipped |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| warmup | 1 | 0 | 1 | 0 | 0 |
|
||||
| performance | 9 | 3 | 6 | 0 | 0 |
|
||||
| sanity | 12 | 12 | 0 | 0 | 0 |
|
||||
| concurrency | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
Sanity checks: **12 of 12**.
|
||||
|
||||
| Case | Result | Reason |
|
||||
| --- | --- | --- |
|
||||
| I1 | pass | the expected JSON value |
|
||||
| I2 | pass | the expected JSON value |
|
||||
| I3 | pass | the expected JSON value |
|
||||
| I4 | pass | the exact expected text |
|
||||
| R1 | pass | the exact expected text |
|
||||
| R2 | pass | the exact expected text |
|
||||
| R3 | pass | the exact expected text |
|
||||
| R4 | pass | the expected JSON value |
|
||||
| T1 | pass | the expected tool call |
|
||||
| T2 | pass | the expected tool call |
|
||||
| T3 | pass | the expected tool call |
|
||||
| T4 | pass | the expected tool call |
|
||||
|
||||
## Failures, exclusions and limitations
|
||||
|
||||
- No request failed.
|
||||
- Excluded from speed summaries: performance-prompt-16k-r1: first_answer_s is not available for this request
|
||||
- Excluded from speed summaries: performance-prompt-16k-r2: first_answer_s is not available for this request
|
||||
- Prompt-prefix reuse is the shared template prefix only; the cache was not emptied between requests, and the expert cache stays warm and evolves. These are not cold-model measurements.
|
||||
- Output caps include reasoning; a request that reaches its cap is `truncated` and fails its sanity check.
|
||||
- The runner cannot see other programs on the machine, the repeat-stop and loop-recovery switches of the server, or the quantization repository's revision; see `missing` in results.json for every value it could not read.
|
||||
- The server had answered no request before this run (a freshly loaded engine).
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"engine_options": null,
|
||||
"other_engine_options": {},
|
||||
"server_defaults": {
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
"top_k": 20,
|
||||
"n_predict": -1
|
||||
},
|
||||
"total_slots": 1,
|
||||
"context": 131072,
|
||||
"source": "the server was already running; its engine options are not exposed"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
+184
@@ -0,0 +1,184 @@
|
||||
{%- set image_count = namespace(value=0) %}
|
||||
{%- set video_count = namespace(value=0) %}
|
||||
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
||||
{%- if content is string %}
|
||||
{{- content }}
|
||||
{%- elif content is iterable and content is not mapping %}
|
||||
{%- for item in content %}
|
||||
{%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain images.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set image_count.value = image_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
||||
{%- elif 'video' in item or item.type == 'video' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain videos.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set video_count.value = video_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
||||
{%- elif 'text' in item %}
|
||||
{{- item.text }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected item type in content.') }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- elif content is none or content is undefined %}
|
||||
{{- '' }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected content type.') }}
|
||||
{%- endif %}
|
||||
{%- endmacro %}
|
||||
{%- if not messages %}
|
||||
{{- raise_exception('No messages provided.') }}
|
||||
{%- endif %}
|
||||
{%- set sysns = namespace(count=0, text='') %}
|
||||
{%- for message in messages %}
|
||||
{%- if sysns.count == loop.index0 and (message.role == 'system' or message.role == 'developer') %}
|
||||
{%- set sys_content = render_content(message.content, false, true)|trim %}
|
||||
{%- if sys_content %}
|
||||
{%- set sysns.text = sysns.text + ('\n' if sysns.text else '') + sys_content %}
|
||||
{%- endif %}
|
||||
{%- set sysns.count = sysns.count + 1 %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set num_sys = sysns.count %}
|
||||
{%- set merged_system = sysns.text %}
|
||||
{%- set reasoning_instructions = '' %}
|
||||
{%- if enable_thinking is undefined or enable_thinking is true %}
|
||||
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
|
||||
{%- if resolved_reasoning_effort == 'high' %}
|
||||
{%- set resolved_reasoning_effort = 'xhigh' %}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
|
||||
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort == 'xhigh' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
|
||||
{%- elif resolved_reasoning_effort == 'low' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if tools and tools is iterable and tools is not mapping %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if reasoning_instructions %}
|
||||
{{- reasoning_instructions + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
|
||||
{%- for tool in tools %}
|
||||
{{- "\n" }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- "\n</tools>" }}
|
||||
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
||||
{%- if merged_system %}
|
||||
{{- '\n\n' + merged_system }}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{%- if merged_system %}
|
||||
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + merged_system + '<|im_end|>\n' }}
|
||||
{%- elif reasoning_instructions %}
|
||||
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
||||
{%- for message in messages[::-1] %}
|
||||
{%- set index = (messages|length - 1) - loop.index0 %}
|
||||
{%- if ns.multi_step_tool and message.role == "user" %}
|
||||
{%- set content = render_content(message.content, false)|trim %}
|
||||
{%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
|
||||
{%- set ns.multi_step_tool = false %}
|
||||
{%- set ns.last_query_index = index %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- for message in messages %}
|
||||
{%- if loop.index0 >= num_sys %}
|
||||
{%- set content = render_content(message.content, true)|trim %}
|
||||
{%- if message.role == "system" or message.role == "developer" %}
|
||||
{{- raise_exception('System message must be at the beginning.') }}
|
||||
{%- elif message.role == "user" %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||
{%- elif message.role == "assistant" %}
|
||||
{%- set reasoning_content = '' %}
|
||||
{%- if message.reasoning_content is string %}
|
||||
{%- set reasoning_content = message.reasoning_content %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = reasoning_content|trim %}
|
||||
{%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
|
||||
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||
{%- endif %}
|
||||
{%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if tool_call.function is defined %}
|
||||
{%- set tool_call = tool_call.function %}
|
||||
{%- endif %}
|
||||
{%- if tool_call.name is not defined or tool_call.name is none %}
|
||||
{{- raise_exception('Tool call is missing a function name.') }}
|
||||
{%- endif %}
|
||||
{%- if loop.first %}
|
||||
{%- if content|trim %}
|
||||
{{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- else %}
|
||||
{{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- if tool_call.arguments is mapping %}
|
||||
{%- for args_name, args_value in tool_call.arguments|items %}
|
||||
{{- '<parameter=' + args_name + '>\n' }}
|
||||
{%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
|
||||
{{- args_value }}
|
||||
{{- '\n</parameter>\n' }}
|
||||
{%- endfor %}
|
||||
{%- elif tool_call.arguments is string %}
|
||||
{%- if tool_call.arguments|trim %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" were passed as a JSON string. Parse them into an object before calling apply_chat_template.') }}
|
||||
{%- endif %}
|
||||
{%- elif tool_call.arguments is defined and tool_call.arguments is not none %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" must be an object/mapping or a JSON string.') }}
|
||||
{%- endif %}
|
||||
{{- '</function>\n</tool_call>' }}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == "tool" %}
|
||||
{%- if loop.previtem and loop.previtem.role != "tool" %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' }}
|
||||
{{- content }}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if not loop.last and loop.nextitem.role != "tool" %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif loop.last %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected message role.') }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- if enable_thinking is defined and enable_thinking is false %}
|
||||
{{- '<think>\n\n</think>\n\n' }}
|
||||
{%- else %}
|
||||
{{- '<think>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{#- Unsloth fixes - developer role, merged system messages, tool calling #}
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
{
|
||||
"protocol_id": "strata-qwen38-thinking-v1-draft.1",
|
||||
"bundle": "standard-v1",
|
||||
"profile": "qwen3.8-flash-next-thinking",
|
||||
"tokenizer_files": {
|
||||
"vocab.json": "4ba64f0332abcfb0b600b7df1537d1e836e68009d5fb7cdb77339738dc6365c4",
|
||||
"merges.txt": "080cf95432173729d160346b7882bd2a644b07f7a3813a3eb73f4e66787bfde9",
|
||||
"token_type.json": "5088c8c298fc06af8382ddb3b76c888703ac2634e82263b97eedcc2ad202738b"
|
||||
},
|
||||
"template": {
|
||||
"path": "serve/chat_template.jinja",
|
||||
"sha256": "ae9644304511ef71f28e036b6f65b4c709880bc5a23b7d0b42125ec63a8c70ca"
|
||||
},
|
||||
"template_kwargs": {
|
||||
"enable_thinking": true,
|
||||
"preserve_thinking": true,
|
||||
"reasoning_effort": "xhigh"
|
||||
},
|
||||
"record_format": "record{:06d}: alpha beta gamma delta epsilon zeta eta theta.",
|
||||
"short_task": "Explain how a hash table finds a value, including collisions and one small example.",
|
||||
"summary_question": "Summarize the structure of these records and explain how to look up a record by its ID.",
|
||||
"record_counts": {
|
||||
"prompt-4k": {
|
||||
"records": 211,
|
||||
"target_tokens": 4096
|
||||
},
|
||||
"prompt-16k": {
|
||||
"records": 858,
|
||||
"target_tokens": 16384
|
||||
}
|
||||
},
|
||||
"length_tolerance": 0.02,
|
||||
"requests": [
|
||||
{
|
||||
"id": "warmup",
|
||||
"input_tokens": 75,
|
||||
"max_shared_prefix_tokens": 47
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r1",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r1",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r1",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r2",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r2",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r2",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r3",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r3",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r3",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "sanity-I1",
|
||||
"input_tokens": 82,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I2",
|
||||
"input_tokens": 90,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I3",
|
||||
"input_tokens": 85,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I4",
|
||||
"input_tokens": 71,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R1",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R2",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R3",
|
||||
"input_tokens": 16382,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R4",
|
||||
"input_tokens": 16390,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-T1",
|
||||
"input_tokens": 325,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T2",
|
||||
"input_tokens": 335,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "sanity-T3",
|
||||
"input_tokens": 352,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T4",
|
||||
"input_tokens": 339,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
}
|
||||
],
|
||||
"note": "input_tokens counts the rendered thinking request with the generation prompt. max_shared_prefix_tokens is the longest token prefix this request shares with any other request of the bundle: cache reuse above it means the user payload was read from the cache."
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
bc6c1f81d86863c3dc36a31f58010701d8ab05180f69f86c6978214409e43af2 build.json
|
||||
f724855774dfbea89fade712a70d534a085c88cda6abc123db419fd79eefab49 requests/concurrency-r1-c1.json
|
||||
2c85f877abbf144ca065c58efc2a7568aaf2c8dc317ad8266e78c149f8245abb requests/concurrency-r1-c2.json
|
||||
264097f61645a0d5089562dc003b7748b1c20339c2df1290ad6664fc1834068a requests/concurrency-r1-c3.json
|
||||
65895c5a5fab53c06e9358d88d921f1bfc9dc7cc8b6e8a2521ac90de080ff470 requests/concurrency-r1-c4.json
|
||||
2c08d50e0c95e345512ec918f58fa716780b79900f09bbfabc37519c8d86543a requests/concurrency-r2-c1.json
|
||||
8aba49693701b9315b2bc5ad55e97a8b03496baedd05f370c49c2c77f45ce6cd requests/concurrency-r2-c2.json
|
||||
7cfb4a5a8fb5702244ce42e0dba9ccd3064d4023d700b409a1a472790a82809c requests/concurrency-r2-c3.json
|
||||
3ab56fbba5fd405945e53cff01b59c3ef966b0f08a0a1b0b50633d26e90cad62 requests/concurrency-r2-c4.json
|
||||
3adb684cf2cb84d4e514de30a4c6f8ccebdd8471160cfcdfc0cdb8ca19ac2137 requests/concurrency-r3-c1.json
|
||||
650379ef9a4da667f6581ae9573eb9af2db2a1fc617701e50536590b18f70077 requests/concurrency-r3-c2.json
|
||||
111415388c5e69ecbd9ab66efe85eb8f3db9142201e1548d03203d3acc05eca1 requests/concurrency-r3-c3.json
|
||||
691a8d27fd06e9beab57e6efb3d82c2a7ed9fc26635fd51b1b1d017117440c1e requests/concurrency-r3-c4.json
|
||||
06776ef38485dc4bb7e18809534fe16a57d421d163f81d4d1532c4b08225968e requests/performance-prompt-16k-r1.json
|
||||
4995c5f3c47c1682f4dc753a9f3a1fba6920fea14691e1ded2da6208ef5fe628 requests/performance-prompt-16k-r2.json
|
||||
7af4c49f99192007be27110296b5a0bcc2efbc91c7ff1c337a6cac6732050559 requests/performance-prompt-16k-r3.json
|
||||
bda0e60c2de0703899ac1bd5711d793b9b325c7f8ba6be6c967f22dd07d8dda8 requests/performance-prompt-4k-r1.json
|
||||
e898eec0ee22499473179a719505c9feb9d3772658833f607dffb34c3727150e requests/performance-prompt-4k-r2.json
|
||||
6d3b3300eace318574490f4379119409920cbe7e88fdaff28eb6f5c79430c3e3 requests/performance-prompt-4k-r3.json
|
||||
483b4f18fcc64ca802fde3377c59bf7b114ec5a5c96fe781bd2be4665ef7b898 requests/performance-short-r1.json
|
||||
b0ec85f40117303d19c8c17a4813625c8554f59caddbe69e7d8aafdcf51d501a requests/performance-short-r2.json
|
||||
e210da20b80c61147da5102fcba65131dafbbb4ceee952fa6b91ce2bb06581ed requests/performance-short-r3.json
|
||||
70ebcd31af5ae1bc7573136d9a257469d8ad9e85b5d0233980998e957c36d6c2 requests/sanity-I1.json
|
||||
1cc6e4bc56b5a22043b6e384b3036170dc6d3edeb403be391a8b802b5613f4cb requests/sanity-I2.json
|
||||
c29b13e6e31b674ff733202b3c0b1f9410768b9ea9368a4a6a13430b3e011fb8 requests/sanity-I3.json
|
||||
780fbd1d02f119323cf50ac4dabcfffbd23dd84115efbe18152e7ee9112b041a requests/sanity-I4.json
|
||||
3c1e0718ad06e591e47cf8df9c36bab2d13332d44b05e440ec17d0402bf9facc requests/sanity-R1.json
|
||||
352e054c48e08e11d339d3a02a14f4f50125071710679b8d04b77aba711fbafc requests/sanity-R2.json
|
||||
36770434f1ea046ec84b32691d644e8416e89fbcbb04b8f6032da66016d78291 requests/sanity-R3.json
|
||||
a63df33c784732cc6b4d8a0f2258b7226c8df5a3b4f6818f9e886bbf09604999 requests/sanity-R4.json
|
||||
6b272ef684b43c1e0004a40b958c807b84dcd9de579d4b2d8819e39ae4196fb2 requests/sanity-T1.json
|
||||
8253ee4aa6b19e61e28d64406807218c9aa10cf2cf682101e8a2d6a50564de46 requests/sanity-T2.json
|
||||
aa2be8761793f51eab6ef04bc54c96e018f6981b7a6a26b9e50cc3c7ac856803 requests/sanity-T3.json
|
||||
04043bdec397df487bebd2999e280a9fae5925e5efec1a10fa96055ec3430faa requests/sanity-T4.json
|
||||
d3365c57db8be1fd0dd9771f74f264b1f6994a396e472c3e5fa4ede2641ffb5c requests/warmup.json
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,76 @@
|
||||
# <user> standard benchmark: NVIDIA GeForce RTX 3090
|
||||
|
||||
Protocol `strata-qwen38-thinking-v1-draft.1` (draft), run `20261027T035250Z-ae1396b5`, started 2026-10-27T03:52:50Z. **Status: nonconforming.**
|
||||
|
||||
Written by `<user> bench run`. The machine-readable result is [results.json](results.json); every request, its raw usage and timings and the client event offsets are in [outputs.jsonl](outputs.jsonl).
|
||||
|
||||
This run is not a complete standard result:
|
||||
|
||||
- no model file could be hashed
|
||||
|
||||
## Hardware and software
|
||||
|
||||
- GPU: NVIDIA GeForce RTX 3090 (24.0 GiB dedicated)
|
||||
- CPU: Intel(R) Core(TM) i9-14900KS; RAM: 110.0 GiB
|
||||
- Storage of the model files: not measured
|
||||
- OS: NixOS 26.05 (Yarara) 6.18.55; drivers: gpu-driver 595.71.05; runtime: cuda 13.2
|
||||
- <user> commit: `4b9e05a2a806a3fe24f7b2560d98626c37dade35`; engine 0.1.51; source build
|
||||
- Background work: not measured
|
||||
|
||||
## Model and configuration
|
||||
|
||||
- Model: qwen3.8-flash-next-iq3_s; quantization IQ3_S; revision not measured; source revision not measured
|
||||
- Files: not measured
|
||||
- Track: **custom** (power cap 250W rep 2, case closed)
|
||||
- Context 131072 tokens; server capacity 1 slot(s)
|
||||
- Sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0, repetition_penalty 1.0, reasoning_effort xhigh; thinking on, preserved
|
||||
- Engine options: `config.json`; chat template: `provenance/chat_template.jinja`
|
||||
|
||||
## Method
|
||||
|
||||
The frozen requests of `bench/prompts/standard-v1` (corpus `e75eff8b9feea04f...`) were sent over loopback HTTP, streaming, one at a time, in this order: 1 warm-up (excluded), 3 repetitions of the short, 4K and 16K prompts, and the twelve sanity checks. Prompt and decode speed are the engine's own `timings`; time to first token is the client's monotonic clock from dispatch to the first nonempty reasoning, answer or tool-call delta. No request was retried. Time after the model loaded: 1 min 34 s.
|
||||
|
||||
## Results
|
||||
|
||||
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| short | n/a | 116.3 (114.1-125.2) n=3 | 0.47 (0.38-0.47) n=3 | 1.05 (0.90-1.26) n=3 | 3.8 (2.9-4.2) n=3 | 0/3 |
|
||||
| prompt-4k | 1712.1 (1544.2-1749.7) n=3 | 122.1 (96.2-129.4) n=3 | 2.41 (2.36-2.67) n=3 | 4.46 (4.11-4.81) n=2 | 6.5 (6.4-8.0) n=3 | 3/3 |
|
||||
| prompt-16k | 2886.7 (2849.0-2887.3) n=3 | 106.8 (102.6-112.9) n=3 | 5.71 (5.70-5.78) n=3 | 9.39 (9.39-9.39) n=1 | 10.5 (10.2-10.8) n=3 | 3/3 |
|
||||
|
||||
Cells are median (min-max) n=samples. Short is a prompt of about 80 tokens: its prompt rate is dominated by the fixed cost of a request, so it is not shown; read prompt speed from prompt-4k and prompt-16k. Capped = requests that reached the 512-token cap, reasoning and answer together. A capped request's total time is the time to the cap, not to an answer, and it has no first-answer time when it was still reasoning.
|
||||
|
||||
| Phase | Planned | Completed | Truncated | Failed | Skipped |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| warmup | 1 | 0 | 1 | 0 | 0 |
|
||||
| performance | 9 | 3 | 6 | 0 | 0 |
|
||||
| sanity | 12 | 12 | 0 | 0 | 0 |
|
||||
| concurrency | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
Sanity checks: **12 of 12**.
|
||||
|
||||
| Case | Result | Reason |
|
||||
| --- | --- | --- |
|
||||
| I1 | pass | the expected JSON value |
|
||||
| I2 | pass | the expected JSON value |
|
||||
| I3 | pass | the expected JSON value |
|
||||
| I4 | pass | the exact expected text |
|
||||
| R1 | pass | the exact expected text |
|
||||
| R2 | pass | the exact expected text |
|
||||
| R3 | pass | the exact expected text |
|
||||
| R4 | pass | the expected JSON value |
|
||||
| T1 | pass | the expected tool call |
|
||||
| T2 | pass | the expected tool call |
|
||||
| T3 | pass | the expected tool call |
|
||||
| T4 | pass | the expected tool call |
|
||||
|
||||
## Failures, exclusions and limitations
|
||||
|
||||
- No request failed.
|
||||
- Excluded from speed summaries: performance-prompt-16k-r1: first_answer_s is not available for this request
|
||||
- Excluded from speed summaries: performance-prompt-16k-r2: first_answer_s is not available for this request
|
||||
- Excluded from speed summaries: performance-prompt-4k-r1: first_answer_s is not available for this request
|
||||
- Prompt-prefix reuse is the shared template prefix only; the cache was not emptied between requests, and the expert cache stays warm and evolves. These are not cold-model measurements.
|
||||
- Output caps include reasoning; a request that reaches its cap is `truncated` and fails its sanity check.
|
||||
- The runner cannot see other programs on the machine, the repeat-stop and loop-recovery switches of the server, or the quantization repository's revision; see `missing` in results.json for every value it could not read.
|
||||
- The server had answered no request before this run (a freshly loaded engine).
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"engine_options": null,
|
||||
"other_engine_options": {},
|
||||
"server_defaults": {
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
"top_k": 20,
|
||||
"n_predict": -1
|
||||
},
|
||||
"total_slots": 1,
|
||||
"context": 131072,
|
||||
"source": "the server was already running; its engine options are not exposed"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
+184
@@ -0,0 +1,184 @@
|
||||
{%- set image_count = namespace(value=0) %}
|
||||
{%- set video_count = namespace(value=0) %}
|
||||
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
||||
{%- if content is string %}
|
||||
{{- content }}
|
||||
{%- elif content is iterable and content is not mapping %}
|
||||
{%- for item in content %}
|
||||
{%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain images.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set image_count.value = image_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
||||
{%- elif 'video' in item or item.type == 'video' %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain videos.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set video_count.value = video_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
||||
{%- elif 'text' in item %}
|
||||
{{- item.text }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected item type in content.') }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- elif content is none or content is undefined %}
|
||||
{{- '' }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected content type.') }}
|
||||
{%- endif %}
|
||||
{%- endmacro %}
|
||||
{%- if not messages %}
|
||||
{{- raise_exception('No messages provided.') }}
|
||||
{%- endif %}
|
||||
{%- set sysns = namespace(count=0, text='') %}
|
||||
{%- for message in messages %}
|
||||
{%- if sysns.count == loop.index0 and (message.role == 'system' or message.role == 'developer') %}
|
||||
{%- set sys_content = render_content(message.content, false, true)|trim %}
|
||||
{%- if sys_content %}
|
||||
{%- set sysns.text = sysns.text + ('\n' if sysns.text else '') + sys_content %}
|
||||
{%- endif %}
|
||||
{%- set sysns.count = sysns.count + 1 %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set num_sys = sysns.count %}
|
||||
{%- set merged_system = sysns.text %}
|
||||
{%- set reasoning_instructions = '' %}
|
||||
{%- if enable_thinking is undefined or enable_thinking is true %}
|
||||
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
|
||||
{%- if resolved_reasoning_effort == 'high' %}
|
||||
{%- set resolved_reasoning_effort = 'xhigh' %}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
|
||||
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
|
||||
{%- endif %}
|
||||
{%- if resolved_reasoning_effort == 'xhigh' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
|
||||
{%- elif resolved_reasoning_effort == 'low' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if tools and tools is iterable and tools is not mapping %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if reasoning_instructions %}
|
||||
{{- reasoning_instructions + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
|
||||
{%- for tool in tools %}
|
||||
{{- "\n" }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- "\n</tools>" }}
|
||||
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
||||
{%- if merged_system %}
|
||||
{{- '\n\n' + merged_system }}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{%- if merged_system %}
|
||||
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + merged_system + '<|im_end|>\n' }}
|
||||
{%- elif reasoning_instructions %}
|
||||
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
||||
{%- for message in messages[::-1] %}
|
||||
{%- set index = (messages|length - 1) - loop.index0 %}
|
||||
{%- if ns.multi_step_tool and message.role == "user" %}
|
||||
{%- set content = render_content(message.content, false)|trim %}
|
||||
{%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
|
||||
{%- set ns.multi_step_tool = false %}
|
||||
{%- set ns.last_query_index = index %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- for message in messages %}
|
||||
{%- if loop.index0 >= num_sys %}
|
||||
{%- set content = render_content(message.content, true)|trim %}
|
||||
{%- if message.role == "system" or message.role == "developer" %}
|
||||
{{- raise_exception('System message must be at the beginning.') }}
|
||||
{%- elif message.role == "user" %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||
{%- elif message.role == "assistant" %}
|
||||
{%- set reasoning_content = '' %}
|
||||
{%- if message.reasoning_content is string %}
|
||||
{%- set reasoning_content = message.reasoning_content %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = reasoning_content|trim %}
|
||||
{%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
|
||||
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||
{%- endif %}
|
||||
{%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if tool_call.function is defined %}
|
||||
{%- set tool_call = tool_call.function %}
|
||||
{%- endif %}
|
||||
{%- if tool_call.name is not defined or tool_call.name is none %}
|
||||
{{- raise_exception('Tool call is missing a function name.') }}
|
||||
{%- endif %}
|
||||
{%- if loop.first %}
|
||||
{%- if content|trim %}
|
||||
{{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- else %}
|
||||
{{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- if tool_call.arguments is mapping %}
|
||||
{%- for args_name, args_value in tool_call.arguments|items %}
|
||||
{{- '<parameter=' + args_name + '>\n' }}
|
||||
{%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
|
||||
{{- args_value }}
|
||||
{{- '\n</parameter>\n' }}
|
||||
{%- endfor %}
|
||||
{%- elif tool_call.arguments is string %}
|
||||
{%- if tool_call.arguments|trim %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" were passed as a JSON string. Parse them into an object before calling apply_chat_template.') }}
|
||||
{%- endif %}
|
||||
{%- elif tool_call.arguments is defined and tool_call.arguments is not none %}
|
||||
{{- raise_exception('Tool call arguments for function "' + (tool_call.name | string) + '" must be an object/mapping or a JSON string.') }}
|
||||
{%- endif %}
|
||||
{{- '</function>\n</tool_call>' }}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == "tool" %}
|
||||
{%- if loop.previtem and loop.previtem.role != "tool" %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' }}
|
||||
{{- content }}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if not loop.last and loop.nextitem.role != "tool" %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif loop.last %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected message role.') }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- if enable_thinking is defined and enable_thinking is false %}
|
||||
{{- '<think>\n\n</think>\n\n' }}
|
||||
{%- else %}
|
||||
{{- '<think>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{#- Unsloth fixes - developer role, merged system messages, tool calling #}
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
{
|
||||
"protocol_id": "strata-qwen38-thinking-v1-draft.1",
|
||||
"bundle": "standard-v1",
|
||||
"profile": "qwen3.8-flash-next-thinking",
|
||||
"tokenizer_files": {
|
||||
"vocab.json": "4ba64f0332abcfb0b600b7df1537d1e836e68009d5fb7cdb77339738dc6365c4",
|
||||
"merges.txt": "080cf95432173729d160346b7882bd2a644b07f7a3813a3eb73f4e66787bfde9",
|
||||
"token_type.json": "5088c8c298fc06af8382ddb3b76c888703ac2634e82263b97eedcc2ad202738b"
|
||||
},
|
||||
"template": {
|
||||
"path": "serve/chat_template.jinja",
|
||||
"sha256": "ae9644304511ef71f28e036b6f65b4c709880bc5a23b7d0b42125ec63a8c70ca"
|
||||
},
|
||||
"template_kwargs": {
|
||||
"enable_thinking": true,
|
||||
"preserve_thinking": true,
|
||||
"reasoning_effort": "xhigh"
|
||||
},
|
||||
"record_format": "record{:06d}: alpha beta gamma delta epsilon zeta eta theta.",
|
||||
"short_task": "Explain how a hash table finds a value, including collisions and one small example.",
|
||||
"summary_question": "Summarize the structure of these records and explain how to look up a record by its ID.",
|
||||
"record_counts": {
|
||||
"prompt-4k": {
|
||||
"records": 211,
|
||||
"target_tokens": 4096
|
||||
},
|
||||
"prompt-16k": {
|
||||
"records": 858,
|
||||
"target_tokens": 16384
|
||||
}
|
||||
},
|
||||
"length_tolerance": 0.02,
|
||||
"requests": [
|
||||
{
|
||||
"id": "warmup",
|
||||
"input_tokens": 75,
|
||||
"max_shared_prefix_tokens": 47
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r1",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r1",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r1",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r2",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r2",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r2",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "performance-short-r3",
|
||||
"input_tokens": 77,
|
||||
"max_shared_prefix_tokens": 50
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-4k-r3",
|
||||
"input_tokens": 4093,
|
||||
"max_shared_prefix_tokens": 54
|
||||
},
|
||||
{
|
||||
"id": "performance-prompt-16k-r3",
|
||||
"input_tokens": 16387,
|
||||
"max_shared_prefix_tokens": 55
|
||||
},
|
||||
{
|
||||
"id": "sanity-I1",
|
||||
"input_tokens": 82,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I2",
|
||||
"input_tokens": 90,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I3",
|
||||
"input_tokens": 85,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-I4",
|
||||
"input_tokens": 71,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R1",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R2",
|
||||
"input_tokens": 4089,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R3",
|
||||
"input_tokens": 16382,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-R4",
|
||||
"input_tokens": 16390,
|
||||
"max_shared_prefix_tokens": 49
|
||||
},
|
||||
{
|
||||
"id": "sanity-T1",
|
||||
"input_tokens": 325,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T2",
|
||||
"input_tokens": 335,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "sanity-T3",
|
||||
"input_tokens": 352,
|
||||
"max_shared_prefix_tokens": 61
|
||||
},
|
||||
{
|
||||
"id": "sanity-T4",
|
||||
"input_tokens": 339,
|
||||
"max_shared_prefix_tokens": 62
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r1-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r2-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c1",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c2",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c3",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
},
|
||||
{
|
||||
"id": "concurrency-r3-c4",
|
||||
"input_tokens": 78,
|
||||
"max_shared_prefix_tokens": 51
|
||||
}
|
||||
],
|
||||
"note": "input_tokens counts the rendered thinking request with the generation prompt. max_shared_prefix_tokens is the longest token prefix this request shares with any other request of the bundle: cache reuse above it means the user payload was read from the cache."
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
bc6c1f81d86863c3dc36a31f58010701d8ab05180f69f86c6978214409e43af2 build.json
|
||||
f724855774dfbea89fade712a70d534a085c88cda6abc123db419fd79eefab49 requests/concurrency-r1-c1.json
|
||||
2c85f877abbf144ca065c58efc2a7568aaf2c8dc317ad8266e78c149f8245abb requests/concurrency-r1-c2.json
|
||||
264097f61645a0d5089562dc003b7748b1c20339c2df1290ad6664fc1834068a requests/concurrency-r1-c3.json
|
||||
65895c5a5fab53c06e9358d88d921f1bfc9dc7cc8b6e8a2521ac90de080ff470 requests/concurrency-r1-c4.json
|
||||
2c08d50e0c95e345512ec918f58fa716780b79900f09bbfabc37519c8d86543a requests/concurrency-r2-c1.json
|
||||
8aba49693701b9315b2bc5ad55e97a8b03496baedd05f370c49c2c77f45ce6cd requests/concurrency-r2-c2.json
|
||||
7cfb4a5a8fb5702244ce42e0dba9ccd3064d4023d700b409a1a472790a82809c requests/concurrency-r2-c3.json
|
||||
3ab56fbba5fd405945e53cff01b59c3ef966b0f08a0a1b0b50633d26e90cad62 requests/concurrency-r2-c4.json
|
||||
3adb684cf2cb84d4e514de30a4c6f8ccebdd8471160cfcdfc0cdb8ca19ac2137 requests/concurrency-r3-c1.json
|
||||
650379ef9a4da667f6581ae9573eb9af2db2a1fc617701e50536590b18f70077 requests/concurrency-r3-c2.json
|
||||
111415388c5e69ecbd9ab66efe85eb8f3db9142201e1548d03203d3acc05eca1 requests/concurrency-r3-c3.json
|
||||
691a8d27fd06e9beab57e6efb3d82c2a7ed9fc26635fd51b1b1d017117440c1e requests/concurrency-r3-c4.json
|
||||
06776ef38485dc4bb7e18809534fe16a57d421d163f81d4d1532c4b08225968e requests/performance-prompt-16k-r1.json
|
||||
4995c5f3c47c1682f4dc753a9f3a1fba6920fea14691e1ded2da6208ef5fe628 requests/performance-prompt-16k-r2.json
|
||||
7af4c49f99192007be27110296b5a0bcc2efbc91c7ff1c337a6cac6732050559 requests/performance-prompt-16k-r3.json
|
||||
bda0e60c2de0703899ac1bd5711d793b9b325c7f8ba6be6c967f22dd07d8dda8 requests/performance-prompt-4k-r1.json
|
||||
e898eec0ee22499473179a719505c9feb9d3772658833f607dffb34c3727150e requests/performance-prompt-4k-r2.json
|
||||
6d3b3300eace318574490f4379119409920cbe7e88fdaff28eb6f5c79430c3e3 requests/performance-prompt-4k-r3.json
|
||||
483b4f18fcc64ca802fde3377c59bf7b114ec5a5c96fe781bd2be4665ef7b898 requests/performance-short-r1.json
|
||||
b0ec85f40117303d19c8c17a4813625c8554f59caddbe69e7d8aafdcf51d501a requests/performance-short-r2.json
|
||||
e210da20b80c61147da5102fcba65131dafbbb4ceee952fa6b91ce2bb06581ed requests/performance-short-r3.json
|
||||
70ebcd31af5ae1bc7573136d9a257469d8ad9e85b5d0233980998e957c36d6c2 requests/sanity-I1.json
|
||||
1cc6e4bc56b5a22043b6e384b3036170dc6d3edeb403be391a8b802b5613f4cb requests/sanity-I2.json
|
||||
c29b13e6e31b674ff733202b3c0b1f9410768b9ea9368a4a6a13430b3e011fb8 requests/sanity-I3.json
|
||||
780fbd1d02f119323cf50ac4dabcfffbd23dd84115efbe18152e7ee9112b041a requests/sanity-I4.json
|
||||
3c1e0718ad06e591e47cf8df9c36bab2d13332d44b05e440ec17d0402bf9facc requests/sanity-R1.json
|
||||
352e054c48e08e11d339d3a02a14f4f50125071710679b8d04b77aba711fbafc requests/sanity-R2.json
|
||||
36770434f1ea046ec84b32691d644e8416e89fbcbb04b8f6032da66016d78291 requests/sanity-R3.json
|
||||
a63df33c784732cc6b4d8a0f2258b7226c8df5a3b4f6818f9e886bbf09604999 requests/sanity-R4.json
|
||||
6b272ef684b43c1e0004a40b958c807b84dcd9de579d4b2d8819e39ae4196fb2 requests/sanity-T1.json
|
||||
8253ee4aa6b19e61e28d64406807218c9aa10cf2cf682101e8a2d6a50564de46 requests/sanity-T2.json
|
||||
aa2be8761793f51eab6ef04bc54c96e018f6981b7a6a26b9e50cc3c7ac856803 requests/sanity-T3.json
|
||||
04043bdec397df487bebd2999e280a9fae5925e5efec1a10fa96055ec3430faa requests/sanity-T4.json
|
||||
d3365c57db8be1fd0dd9771f74f264b1f6994a396e472c3e5fa4ede2641ffb5c requests/warmup.json
|
||||
File diff suppressed because it is too large
Load Diff
@@ -43,10 +43,39 @@ so concurrent clients just queue.
|
||||
- Prompt speed is already above the community 2×3090 report (1,798 tok/s at 19.9K). The 915 tok/s in
|
||||
the earlier notes was a first cold prompt right after load, not a steady-state number.
|
||||
|
||||
## Power cap: 250 W vs 275 W
|
||||
|
||||
2026-10-27, setup defaults, **case closed**, caps set with `nvidia-smi -pl` at runtime. Runs alternated
|
||||
250 / 275 / 250 / 275, each after a fresh restart. GPU temp, power, SM clock and throttle reasons were
|
||||
sampled every 2 s.
|
||||
|
||||
| Cap | Folder | Prompt 4K tok/s | Prompt 16K tok/s | TTFT 16K | Decode short | Decode 4K | Decode 16K |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|
|
||||
| 250 W | `6a05efdc` | 1,719 | 2,897 | 5.69 s | 122.9 | 112.9 | 108.2 |
|
||||
| 275 W | `5c64d992` | 1,742 | 3,030 | 5.44 s | 125.3 | 115.3 | 116.1 |
|
||||
| 250 W | `ae1396b5` | 1,712 | 2,887 | 5.71 s | 116.3 | 122.1 | 106.8 |
|
||||
| 275 W | `16ea71d0` | 1,743 | 3,016 | 5.46 s | 114.2 | 111.6 | 111.0 |
|
||||
|
||||
While busy (samples above 150 W):
|
||||
|
||||
| Cap | GPU | Peak temp | Mean / peak W | Mean SM MHz | Power-throttled |
|
||||
|---|---|---:|---:|---:|---:|
|
||||
| 250 W | 0 | 51°C | 220 / 250 | 1,858 | 62 of 80 samples |
|
||||
| 250 W | 1 | 59°C | 223 / 249 | 1,807 | 82 of 86 |
|
||||
| 275 W | 0 | 53°C | 228 / 273 | 1,867 | 45 of 78 |
|
||||
| 275 W | 1 | 63°C | 239 / 272 | 1,839 | 80 of 87 |
|
||||
|
||||
- **275 W: +4.6% prompt speed at 16K** (2,892 → 3,023 tok/s, −0.25 s to first token), +1.6% at 4K. Both
|
||||
repeats agree to within 0.5%, so this is real and not noise.
|
||||
- Decode is unchanged within noise; decode isn't power-bound.
|
||||
- Cost: +2–4°C peak and +8–16 W mean per card while busy. All 12 sanity checks passed in every run.
|
||||
- GPU1 still hits its power limit most of the time at 275 W, so a higher cap would probably buy a
|
||||
little more prompt speed.
|
||||
|
||||
## Not tried yet
|
||||
|
||||
- Huge pages for the expert arena: the engine logs `MAP_HUGETLB unavailable (needed 23983 2 MiB pages,
|
||||
vm.nr_hugepages=0)`. That would mean reserving ~47 GB of huge pages in the guest (a NixOS change and a
|
||||
reboot).
|
||||
- Raising the power cap from 250 W toward 300–350 W for prompt speed (gpu-unlock GEMM was +40% at 350 W).
|
||||
- Power cap above 275 W (300–350 W). gpu-unlock GEMM was +40% at 350 W, but 275 W gave only +4.6% prompt speed.
|
||||
- Other quants.
|
||||
|
||||
@@ -2,6 +2,14 @@
|
||||
|
||||
Newest first. Each entry: what broke, the cause, and what fixed it.
|
||||
|
||||
## 2026-10-27: power cap 250 W vs 275 W
|
||||
|
||||
With the case closed, alternating runs (250/275/250/275, fresh restart each) gave +4.6% prompt speed at
|
||||
16K at 275 W (2,892 → 3,023 tok/s) and +1.6% at 4K; decode was unchanged within noise. Peak temps went
|
||||
from 51/59°C to 53/63°C. The cap is **runtime only** (`nvidia-smi -pl 275`), and
|
||||
`nvidia-power-limit.service` still sets 250 W at boot. Results:
|
||||
[benchmarks/standard](../benchmarks/standard/README.md#power-cap-250-w-vs-275-w).
|
||||
|
||||
## 2026-10-26 (night): first standard benchmark and tuning arms
|
||||
|
||||
Ran Strata's `standard-v1` suite on the idle server, then one arm per tuning flag with a fresh restart
|
||||
|
||||
Reference in New Issue
Block a user