lovey/docs/strata.md

# Strata: Qwen3.8-Flash-Next IQ3_S

[Niko1221/Strata](https://github.com/Niko1221/Strata) (MIT) is an MoE expert-offload engine built on
llama.cpp. It serves OpenAI- and Anthropic-compatible APIs and splits layers across both 3090s as a
pipeline, so it needs no NVLink.

- API: `http://192.168.1.67:8080/v1`, API key required
- Model: Qwen3.8-Flash-Next, ISTA-DASLab GSQ-RCO **IQ3_S**, plus the BF16 mmproj for images and the
  MTP draft layer
- Engine: **source build pinned at `4b9e05a2a806a3fe24f7b2560d98626c37dade35`** (2026-10-25), CUDA 13.0
  from the repo's flake

## Decisions

- IQ3_S quant
- Source build at a reviewed commit. The prebuilt engine zip installs unverified if GitHub returns no
  digest, so we don't use it.
- Whole LAN with an API key (default would bind 127.0.0.1)
- Images on, GPU reader

## Security review

Read statically, key points checked against the code:

- No telemetry and no auto-update. Model files are pinned to an HF commit and SHA-256 checked.
- Any client with the key can make the server read **any local file** as an "image", or fetch any
  URL. `--offline` blocks URL fetches, and the systemd sandbox (`ProtectSystem=strict`,
  `ProtectHome`, writes only to `/var/lib/strata` and `/models/strata`) limits which files it sees.

## Layout

| Path | What |
|---|---|
| `/var/lib/strata/Strata` | Source checkout, `.venv`, built engine, `strata-iq3_s.json` |
| `/var/lib/strata/api-key` | The API key (0600, user `strata`) |
| `/models/strata/models/IQ3_S/` | The two GGUF shards |
| `/models/strata/models/mmproj-…-BF16.gguf` | Vision projector |
| `/models/strata/packs/iq3_s`, `/models/strata/mtp/rt` | Expert pack and MTP draft layer |

[`strata/strata-iq3_s.json`](../strata/strata-iq3_s.json) is the config with the key removed. Key
engine args: `--max-context 131072`, `--kv int8`, `--kv-resident 32768`, `--spec 4 --spec-min-p 0.5`
with MTP, `--expert-cache auto`, `--prefill auto`, `--ple-io ram`, `--vision`, `--vram-reserve-mib 700`.

The service is `systemd.services.strata` in [`nixos/configuration.nix`](../nixos/configuration.nix):
`serve/server.py --engine strata --config strata-iq3_s.json --port 8080 --offline`,
`STRATA_NO_ACQUIRE=1`, `LimitMEMLOCK=infinity`, plus the sandboxing above.

## Gotchas

- The engine's RUNPATH resolves `libcuda.so.1` to the CUDA toolkit's **stub**. The unit sets
  `LD_LIBRARY_PATH=/run/opengl-driver/lib` so the real driver wins.
- The CUDA 13.0 toolkit only exists in the flake's dev shell. It is pinned by the GC root
  `/nix/var/nix/gcroots/strata-cuda-13.0` → `/nix/store/…-cuda-merged-13.0`, so `nix-collect-garbage`
  can't break the engine. Recreate it after a rebuild (below).
- Running setup under `systemd-run` needs `SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt`, or it
  (correctly) rejects GitHub's certificate.
- Qwen thinks first (`reasoning_content`); a small `max_tokens` can end before any `content`.
- Thinking control: Strata reads `chat_template_kwargs.enable_thinking` and `reasoning_effort`. With
  neither, it thinks at full effort. Effort `low`/`medium` add a short system note before the answer
  (unknown values act as medium); `high`/`xhigh` or none mean unrestricted thinking.
- With thinking on and no explicit budget, Strata caps thinking at 75% of `max_tokens`
  ("answer_room"), and shortens `max_tokens` to whatever room the prompt leaves in the window rather
  than rejecting the request.

## Benchmarks

2026-10-26, engine at `4b9e05a2`, 250 W cap, gpu-unlock active:

| Metric | Result |
|---|---|
| Load | ~2 min: 46.8 GiB of experts into RAM at 1.2 GiB/s over virtiofs, 9,508 experts (17 GiB) in VRAM, 29 GB PLE table in RAM |
| Decode (code, 1,500 tokens, greedy) | 127.5 tok/s, draft acceptance 83% |
| Decode (long agent turns, pi) | 142–158 tok/s, expert cache ~99% hit |
| 17K-token cold prompt | 915 tok/s prompt, 113 tok/s output; repeats reuse the cached prefix |
| Community 2×3090 reference | ~137 decode, 1,800–2,300 prompt tok/s with tuning |

**Standard benchmark** (`standard-v1`, idle server, 2026-10-26): prompt 1,712–1,776 tok/s at 4K and
2,881–2,938 tok/s at 16K, decode ~110–131 tok/s at temperature 1.0, 12/12 sanity checks. The 915 tok/s
above was a first cold prompt right after load. Tuning arms (`--adapt-async 1`, `--pipeline-windows 2`,
`--prefill auto:32768`) gave no gain, and the 32K prefill chunk was 22% slower at 16K: see
[benchmarks/standard](../benchmarks/standard/README.md). The setup defaults stay.

Memory at idle with the model loaded: both GPUs ~23.8 of 24.5 GB used, guest ~76 GB RAM used with
~33 GB available.

## Rebuild

As user `strata` on the guest (after the NixOS config is applied):

```sh
cd /var/lib/strata
git clone https://github.com/Niko1221/Strata.git && cd Strata
git checkout 4b9e05a2a806a3fe24f7b2560d98626c37dade35
nix develop -c ./setup.sh --build --cuda 13 --gpus 0,1 --family qwen --model IQ3_S --vision gpu \
  --data-dir /models/strata --host 0.0.0.0 --api-key "$(cat /var/lib/strata/api-key)" \
  --no-start --no-browser --yes
```

Generate a new key first (`openssl rand -hex 24 > /var/lib/strata/api-key; chmod 600 …`) if the old
one is lost. If `/models/strata` survived (it's on the host's NVMe), setup reuses the downloaded files.

Then pin the toolkit and start the service:

```sh
cuda=$(sudo -u strata -H nix develop /var/lib/strata/Strata -c sh -c 'dirname $(dirname $(command -v nvcc))')
echo "$cuda"    # today: /nix/store/7mi3cxyj9rgx76yxr6jg4z8i3cvb3dmz-cuda-merged-13.0
sudo nix-store --add-root /nix/var/nix/gcroots/strata-cuda-13.0 -r "$cuda"
sudo systemctl enable --now strata
journalctl -u strata -f    # wait for READY, ~2 min
```

Compare the resulting `strata-iq3_s.json` with [`strata/strata-iq3_s.json`](../strata/strata-iq3_s.json).