Add passive Strata snapshot tooling and IQ3_S baseline

snapshot.sh reads /metrics and /v1/status only (no prompts), so it is
safe while the server is in use; summarize.py turns a snapshot into
Markdown and can compare two. First snapshot is the IQ3_S baseline
from 11.9 h of pi use.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

README.md

@@ -53,6 +53,7 @@ proxmox/      files from lovey, at their real paths (etc/..., usr/local/bin/...,
 strata/       strata-iq3_s.json, the engine config (API key removed)
 pi/           ~/.pi/agent/models.json and settings.json for the client (API key removed)
 docs/         per-layer notes, rebuild steps and the change log
+benchmarks/   live metric snapshots (passive) and later benchmark runs
 ```
 
 **Secrets are not in this repo** (it is public). The Strata API key lives only in

benchmarks/README.md

@@ -0,0 +1,29 @@
+# Benchmarks
+
+## Live snapshots (passive, safe while working)
+
+`./benchmarks/snapshot.sh <label>` reads Strata's `/metrics` and `/v1/status` and writes
+`live/<UTC time>-<label>/` with the raw JSON and a `summary.md`. It sends no prompts, so it doesn't
+load the GPUs or disturb a session. To compare two snapshots:
+
+```sh
+python3 -I benchmarks/summarize.py benchmarks/live/<new> benchmarks/live/<old>
+```
+
+Caveats: the counters cover everything since the engine started, so the mix of work changes the numbers.
+The detailed per-request list only holds the last 12 requests (`/api/requests` is off in this config).
+For a before/after on a tuning or quant change, snapshot just before the change, then again after a
+similar stretch of use. "Prompt tok/s (all reads)" mixes cold prompts with small cached tails; the
+"cold, >=4K" row only fills in when a recent request read at least 4K uncached tokens.
+
+## Standard benchmark (needs an idle server)
+
+`strata.py bench run --url http://127.0.0.1:8080` on the guest runs Strata's fixed protocol
+(`docs/BENCHMARK_PROTOCOL.md` in the Strata repo, 21 requests, ~2–4 min). This is the comparable
+number for tuning and quant changes. **Only run it when nothing else is using the server.** Not run yet.
+
+## Baseline
+
+`live/2026-10-26T2015Z-baseline-iq3_s/`: IQ3_S, engine 0.1.51 at `4b9e05a2`, 250 W cap, gpu-unlock,
+settings as in `strata/strata-iq3_s.json`. 758 requests over 11.9 h of pi use: decode 136.8 tok/s,
+draft acceptance 75.8%, 98% of prompt tokens served from cache, expert cache hit ~99%.

benchmarks/live/2026-10-26T2015Z-baseline-iq3_s/status.json

@@ -0,0 +1 @@
+{"service": "strata", "model": "qwen3.8-flash-next-iq3_s", "loaded": true, "auto_load": true, "structured_output": {"formats": ["json_object", "json_schema"], "method": "prompt_and_validate", "constrained_decoding": false, "stream_buffered": true}, "engine": "0.1.51", "started": 1793002959, "uptime_s": 42742, "cache_max_tokens": 131072, "context": {"native": 131072, "max_positions": 131072}, "concurrency": {"serving": 1, "requested": 1}, "dialects": ["/v1/chat/completions", "/v1/messages", "/v1/responses"], "vision": {"enabled": true, "available": true, "error": null}, "image_generation": {"enabled": false}, "activity": {"requests": 758, "in_flight": 0, "last_request_at": 1793045696}, "last_timings": {"cache_n": 57189, "prompt_n": 953, "prompt_ms": 1445.0, "prompt_per_token_ms": 1.516, "prompt_per_second": 659.5, "predicted_n": 333, "predicted_ms": 2499.9, "predicted_per_token_ms": 7.507, "predicted_per_second": 133.2, "draft_n": 305, "draft_n_accepted": 209, "at": 1793045696}, "vram": {"elastic": false}, "machine": {"at": 1793045701, "gpu": {"name": "NVIDIA GeForce RTX 3090 + NVIDIA GeForce RTX 3090", "used_mib": 48586.0, "total_mib": 49152.0, "util_pct": 0.0, "temp_c": 59, "power_w": 228.298}, "ram": {"used_gib": 77.3, "total_gib": 110.0}}}

benchmarks/live/2026-10-26T2015Z-baseline-iq3_s/summary.md

@@ -0,0 +1,64 @@
+# Strata live snapshot, 2026-10-26 20:15 UTC
+
+Model `qwen3.8-flash-next-iq3_s`, engine 0.1.51. Passive counters from normal use, not a benchmark: mixed workloads, so compare like with like.
+
+| Metric | Value |
+|---|---:|
+| Uptime (h) | 11.9 |
+| Requests | 758 |
+| Decode tok/s (all output) | 136.8 |
+| Prompt tok/s (all reads) | 1,349.8 |
+| Prompt tok/s (cold, >=4K, recent) | - |
+| Draft acceptance % | 75.8 |
+| Prompt tokens from cache % | 98.0 |
+| Expert cache hit % (recent) | 99.0 |
+| Experts over PCIe % (recent) | 0.1 |
+| Output tokens | 1,125,514 |
+| Prompt tokens read | 1,016,042 |
+
+## Recent requests
+
+| Prompt | Reused | Read | Prompt tok/s | Output | Decode tok/s | Draft acc % | Hit % | Finish |
+|---:|---:|---:|---:|---:|---:|---:|---:|---|
+| 58,142 | 57,189 | 953 | 659.5 | 333 | 133.2 | 68.5 | 99.6 | stop |
+| 56,910 | 56,644 | 266 | 382.6 | 279 | 112.2 | 56.2 | 99.5 | stop |
+| 56,145 | 55,930 | 215 | 374.4 | 498 | 110.9 | 57.1 | 99.5 | stop |
+| 55,813 | 55,789 | 24 | 193.1 | 115 | 137.3 | 77.8 | 99.2 | stop |
+| 55,202 | 54,807 | 395 | 490.1 | 585 | 137.9 | 75.4 | 98.8 | stop |
+| 54,511 | 54,478 | 33 | 190.4 | 294 | 117.7 | 62.9 | 99.1 | stop |
+| 54,361 | 54,326 | 35 | 204.8 | 116 | 125.2 | 70.3 | 99.4 | stop |
+| 53,592 | 53,146 | 446 | 631.4 | 733 | 134.1 | 70.0 | 99.0 | stop |
+| 53,041 | 52,819 | 222 | 451.9 | 106 | 146.7 | 75.7 | 97.2 | stop |
+| 52,684 | 52,479 | 205 | 344.4 | 135 | 132.2 | 75.7 | 98.9 | stop |
+| 50,103 | 50,086 | 17 | 184.4 | 2,376 | 128.7 | 68.3 | 99.2 | stop |
+| 48,862 | 48,273 | 589 | 667.1 | 1,225 | 130.1 | 70.9 | 98.9 | stop |
+
+## Engine settings
+
+| Setting | Value |
+|---|---|
+| `version` | 0.1.51 |
+| `max_context` | 131072 |
+| `kv` | int8 |
+| `kv_resident` | 32768 |
+| `expert_slots` | 17889 |
+| `expert_slots_primary` | 9508 |
+| `expert_cache_mib` | 34724 |
+| `expert_cache_primary_mib` | 17542 |
+| `arena_mib` | 47962 |
+| `spec` | 6 |
+| `mtp_max` | 4 |
+| `spec_min_p` | 0.50 |
+| `drafting` | mtp |
+| `prefill_cap` | 8192 |
+| `prefill_stages` | 2 |
+| `prefill_pipe` | 0 |
+| `pcie_frac` | 0.34 |
+| `conversation_cache_mib` | 8192 |
+| `vram_free_mib` | 455 |
+
+## Hardware at snapshot
+
+- GPU0: 54°C, 115 W, 23.7/24.0 GiB, util 0%
+- GPU1: 59°C, 113 W, 23.8/24.0 GiB, util 0%
+- Guest RAM: 77.3/110.0 GiB

benchmarks/snapshot.sh

@@ -0,0 +1,15 @@
+#!/bin/sh
+# Passive snapshot of Strata's live counters: GETs /metrics and /v1/status only, sends no prompts,
+# so it is safe to run while the server is in use. Writes benchmarks/live/<UTC time>-<label>/.
+# Usage: benchmarks/snapshot.sh [label]        (run from the workstation; needs ssh to llm-nixos)
+set -eu
+host=${STRATA_HOST:-jason@192.168.1.67}
+label=${1:-snapshot}
+dir="$(dirname "$0")/live/$(date -u +%Y-%m-%dT%H%MZ)-$label"
+mkdir -p "$dir"
+for ep in metrics v1/status; do
+  ssh "$host" 'curl -sf -H "Authorization: Bearer $(sudo cat /var/lib/strata/api-key)" http://127.0.0.1:8080/'"$ep" \
+    > "$dir/$(basename "$ep").json"
+done
+python3 -I "$(dirname "$0")/summarize.py" "$dir" > "$dir/summary.md"
+cat "$dir/summary.md"

benchmarks/summarize.py

@@ -0,0 +1,99 @@
+"""Summarize a Strata live snapshot (metrics.json + status.json) as Markdown.
+
+Usage: python3 summarize.py <snapshot dir> [<older snapshot dir>]   (second dir adds a comparison column)
+"""
+import json
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+ENGINE_KEYS = ["version", "max_context", "kv", "kv_resident", "expert_slots", "expert_slots_primary",
+               "expert_cache_mib", "expert_cache_primary_mib", "arena_mib", "spec", "mtp_max", "spec_min_p",
+               "drafting", "prefill_cap", "prefill_stages", "prefill_pipe", "pcie_frac", "conversation_cache_mib",
+               "vram_free_mib"]
+
+
+def load(d):
+    d = Path(d)
+    return json.loads((d / "metrics.json").read_text()), json.loads((d / "status.json").read_text())
+
+
+def stats(m):
+    t = m["totals"]
+    read = t["prompt_tokens"] - t["reused"]
+    out = {
+        "uptime_h": (m["time"] - t["since"]) / 3600,
+        "requests": t["requests"],
+        "decode_tok_s": t["output_tokens"] / (t["decode_ms"] / 1000) if t["decode_ms"] else None,
+        "prompt_read_tok_s": read / (t["prompt_ms"] / 1000) if t["prompt_ms"] else None,
+        "draft_accept_pct": 100 * t["drafts_accepted"] / t["drafts_offered"] if t["drafts_offered"] else None,
+        "prompt_reuse_pct": 100 * t["reused"] / t["prompt_tokens"] if t["prompt_tokens"] else None,
+        "output_tokens": t["output_tokens"],
+        "prompt_tokens_read": read,
+    }
+    # Cold prompts (little reuse, >= 4K read) show real prefill speed; the totals mix in tiny cached tails.
+    cold = [r for r in m["requests"] if r["prompt_read"] >= 4096 and r["prompt_ms"]]
+    if cold:
+        out["cold_prefill_tok_s"] = sum(r["prompt_read"] for r in cold) / (sum(r["prompt_ms"] for r in cold) / 1000)
+    recent = [r for r in m["requests"] if r["output_tokens"] >= 100]
+    if recent:
+        out["recent_hit_rate_pct"] = 100 * sum(r["hit_rate"] for r in recent) / len(recent)
+        out["recent_pcie_share_pct"] = 100 * sum(r["pcie_share"] for r in recent) / len(recent)
+    return out
+
+
+def fmt(v):
+    if v is None:
+        return "-"
+    if isinstance(v, float):
+        return f"{v:,.1f}"
+    return f"{v:,}"
+
+
+def main():
+    m, s = load(sys.argv[1])
+    old = load(sys.argv[2]) if len(sys.argv) > 2 else None
+    cur = stats(m)
+    prev = stats(old[0]) if old else {}
+    when = datetime.fromtimestamp(m["time"], timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
+    print(f"# Strata live snapshot, {when}\n")
+    print(f"Model `{s['model']}`, engine {s['engine']}. Passive counters from normal use, not a benchmark:"
+          " mixed workloads, so compare like with like.\n")
+    head = "| Metric | Value |" + (" Before |" if old else "")
+    print(head + "\n|---|---:|" + ("---:|" if old else ""))
+    labels = {
+        "uptime_h": "Uptime (h)", "requests": "Requests", "decode_tok_s": "Decode tok/s (all output)",
+        "prompt_read_tok_s": "Prompt tok/s (all reads)", "cold_prefill_tok_s": "Prompt tok/s (cold, >=4K, recent)",
+        "draft_accept_pct": "Draft acceptance %", "prompt_reuse_pct": "Prompt tokens from cache %",
+        "recent_hit_rate_pct": "Expert cache hit % (recent)", "recent_pcie_share_pct": "Experts over PCIe % (recent)",
+        "output_tokens": "Output tokens", "prompt_tokens_read": "Prompt tokens read",
+    }
+    for k, label in labels.items():
+        print(f"| {label} | {fmt(cur.get(k))} |" + (f" {fmt(prev.get(k))} |" if old else ""))
+
+    print("\n## Recent requests\n")
+    print("| Prompt | Reused | Read | Prompt tok/s | Output | Decode tok/s | Draft acc % | Hit % | Finish |")
+    print("|---:|---:|---:|---:|---:|---:|---:|---:|---|")
+    for r in m["requests"]:
+        ptps = r["prompt_read"] / (r["prompt_ms"] / 1000) if r["prompt_ms"] else None
+        acc = 100 * r["drafts_accepted"] / r["drafts_offered"] if r["drafts_offered"] else None
+        print(f"| {fmt(r['prompt_tokens'])} | {fmt(r['reused'])} | {fmt(r['prompt_read'])} | {fmt(ptps)} "
+              f"| {fmt(r['output_tokens'])} | {fmt(r['decode_tok_s'])} | {fmt(acc)} | {fmt(100 * r['hit_rate'])} "
+              f"| {r['finish']} |")
+
+    e = m["engine"]
+    print("\n## Engine settings\n")
+    print("| Setting | Value |\n|---|---|")
+    for k in ENGINE_KEYS:
+        if k in e:
+            print(f"| `{k}` | {e[k]} |")
+    g = m["hardware"]["gpus"]
+    print("\n## Hardware at snapshot\n")
+    for x in g:
+        print(f"- GPU{x['index']}: {x['temp']}°C, {x['power']:.0f} W, "
+              f"{x['mem_used'] / 2**30:.1f}/{x['mem_total'] / 2**30:.1f} GiB, util {x['util']}%")
+    print(f"- Guest RAM: {s['machine']['ram']['used_gib']}/{s['machine']['ram']['total_gib']} GiB")
+
+
+if __name__ == "__main__":
+    main()