lovey/docs/strata.md

Strata: Qwen3.8-Flash-Next IQ3_S

Niko1221/Strata (MIT) is an MoE expert-offload engine built on llama.cpp. It serves OpenAI- and Anthropic-compatible APIs and splits layers across both 3090s as a pipeline, so it needs no NVLink.

• API: http://192.168.1.67:8080/v1, API key required
• Model: Qwen3.8-Flash-Next, ISTA-DASLab GSQ-RCO IQ3_S, plus the BF16 mmproj for images and the MTP draft layer
• Engine: source build pinned at 4b9e05a2a806a3fe24f7b2560d98626c37dade35 (2026-10-25), CUDA 13.0 from the repo's flake

Decisions

• IQ3_S quant
• Source build at a reviewed commit. The prebuilt engine zip installs unverified if GitHub returns no digest, so we don't use it.
• Whole LAN with an API key (default would bind 127.0.0.1)
• Images on, GPU reader

Security review

Read statically, key points checked against the code:

• No telemetry and no auto-update. Model files are pinned to an HF commit and SHA-256 checked.
• Any client with the key can make the server read any local file as an "image", or fetch any URL. --offline blocks URL fetches, and the systemd sandbox (ProtectSystem=strict, ProtectHome, writes only to /var/lib/strata and /models/strata) limits which files it sees.

Layout

Path	What
/var/lib/strata/Strata	Source checkout, .venv, built engine, strata-iq3_s.json
/var/lib/strata/api-key	The API key (0600, user strata)
/models/strata/models/IQ3_S/	The two GGUF shards
/models/strata/models/mmproj-…-BF16.gguf	Vision projector
/models/strata/packs/iq3_s, /models/strata/mtp/rt	Expert pack and MTP draft layer

strata/strata-iq3_s.json is the config with the key removed. Key engine args: --max-context 131072, --kv int8, --kv-resident 32768, --spec 4 --spec-min-p 0.5 with MTP, --expert-cache auto, --prefill auto, --ple-io ram, --vision, --vram-reserve-mib 700.

The service is systemd.services.strata in nixos/configuration.nix: serve/server.py --engine strata --config strata-iq3_s.json --port 8080 --offline, STRATA_NO_ACQUIRE=1, LimitMEMLOCK=infinity, plus the sandboxing above.

Gotchas

• The engine's RUNPATH resolves libcuda.so.1 to the CUDA toolkit's stub. The unit sets LD_LIBRARY_PATH=/run/opengl-driver/lib so the real driver wins.
• The CUDA 13.0 toolkit only exists in the flake's dev shell. It is pinned by the GC root /nix/var/nix/gcroots/strata-cuda-13.0 → /nix/store/…-cuda-merged-13.0, so nix-collect-garbage can't break the engine. Recreate it after a rebuild (below).
• Running setup under systemd-run needs SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt, or it (correctly) rejects GitHub's certificate.
• Qwen thinks first (reasoning_content); a small max_tokens can end before any content.
• Thinking control: Strata reads chat_template_kwargs.enable_thinking and reasoning_effort. With neither, it thinks at full effort. Effort low/medium add a short system note before the answer (unknown values act as medium); high/xhigh or none mean unrestricted thinking.
• With thinking on and no explicit budget, Strata caps thinking at 75% of max_tokens ("answer_room"), and shortens max_tokens to whatever room the prompt leaves in the window rather than rejecting the request.

Benchmarks

2026-10-26, engine at 4b9e05a2, 250 W cap, gpu-unlock active:

Metric	Result
Load	~2 min: 46.8 GiB of experts into RAM at 1.2 GiB/s over virtiofs, 9,508 experts (17 GiB) in VRAM, 29 GB PLE table in RAM
Decode (code, 1,500 tokens, greedy)	127.5 tok/s, draft acceptance 83%
Decode (long agent turns, pi)	142–158 tok/s, expert cache ~99% hit
17K-token cold prompt	915 tok/s prompt, 113 tok/s output; repeats reuse the cached prefix
Community 2×3090 reference	~137 decode, 1,800–2,300 prompt tok/s with tuning

Standard benchmark (standard-v1, idle server, 2026-10-26): prompt 1,712–1,776 tok/s at 4K and 2,881–2,938 tok/s at 16K, decode ~110–131 tok/s at temperature 1.0, 12/12 sanity checks. The 915 tok/s above was a first cold prompt right after load. Tuning arms (--adapt-async 1, --pipeline-windows 2, --prefill auto:32768) gave no gain, and the 32K prefill chunk was 22% slower at 16K: see benchmarks/standard. The setup defaults stay.

Memory at idle with the model loaded: both GPUs ~23.8 of 24.5 GB used, guest ~76 GB RAM used with ~33 GB available.

Rebuild

As user strata on the guest (after the NixOS config is applied):

cd /var/lib/strata
git clone https://github.com/Niko1221/Strata.git && cd Strata
git checkout 4b9e05a2a806a3fe24f7b2560d98626c37dade35
nix develop -c ./setup.sh --build --cuda 13 --gpus 0,1 --family qwen --model IQ3_S --vision gpu \
  --data-dir /models/strata --host 0.0.0.0 --api-key "$(cat /var/lib/strata/api-key)" \
  --no-start --no-browser --yes

Generate a new key first (openssl rand -hex 24 > /var/lib/strata/api-key; chmod 600 …) if the old one is lost. If /models/strata survived (it's on the host's NVMe), setup reuses the downloaded files.

Then pin the toolkit and start the service:

cuda=$(sudo -u strata -H nix develop /var/lib/strata/Strata -c sh -c 'dirname $(dirname $(command -v nvcc))')
echo "$cuda"    # today: /nix/store/7mi3cxyj9rgx76yxr6jg4z8i3cvb3dmz-cuda-merged-13.0
sudo nix-store --add-root /nix/var/nix/gcroots/strata-cuda-13.0 -r "$cuda"
sudo systemctl enable --now strata
journalctl -u strata -f    # wait for READY, ~2 min

Compare the resulting strata-iq3_s.json with strata/strata-iq3_s.json.