Track Strata configs per model; add UD-Q4_K_XL config
strata/README.md indexes the configs and which one is active; configuration.nix selects it with strataConfig. UD-Q4_K_XL downloaded and prepared, not active yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Jason Witty
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
cff8cf640e
commit
08c79d84bc
@@ -50,14 +50,14 @@ Current numbers (standard benchmark, idle): prompt ~1,750 tok/s at 4K and ~2,900
|
||||
```
|
||||
nixos/ /etc/nixos on llm-nixos (configuration.nix, gpu-unlock.nix, hardware-configuration.nix)
|
||||
proxmox/ files from lovey, at their real paths (etc/..., usr/local/bin/..., root/...)
|
||||
strata/ strata-iq3_s.json, the engine config (API key removed)
|
||||
strata/ one engine config per model/quant (API key removed); README.md says which is active
|
||||
pi/ ~/.pi/agent/models.json, settings.json and extensions/ for the client (API key removed)
|
||||
docs/ per-layer notes, rebuild steps and the change log
|
||||
benchmarks/ live metric snapshots (passive) and later benchmark runs
|
||||
```
|
||||
|
||||
**Secrets are not in this repo** (it is public). The Strata API key lives only in
|
||||
`/var/lib/strata/api-key` and `strata-iq3_s.json` on the guest; placeholders are marked `<...>`.
|
||||
`/var/lib/strata/api-key` and the `strata-*.json` configs on the guest; placeholders are marked `<...>`.
|
||||
|
||||
## Rebuild order
|
||||
|
||||
|
||||
@@ -2,6 +2,18 @@
|
||||
|
||||
Newest first. Each entry: what broke, the cause, and what fixed it.
|
||||
|
||||
## 2026-10-27: UD-Q4_K_XL installed, configs tracked per model
|
||||
|
||||
Downloaded Unsloth's UD-Q4_K_XL (111 GB, SHA-256 checked by setup) to test a bigger quant than IQ3_S: Strata's
|
||||
own yardstick puts it at 93.1% same-top-token as Q8_0 vs 87.7% for IQ3_S (mean KL 0.056 vs 0.144). Setup's
|
||||
check says all 77 GB of its experts fit in the guest's 110 GB. `setup.sh --no-start` wrote
|
||||
`strata-unsloth-ud-q4_k_xl.json` beside `strata-iq3_s.json`, which it left byte-identical (compared with
|
||||
the backup in `/var/lib/strata/backup-2026-10-27/`).
|
||||
|
||||
Configs are now tracked one file per model in [`strata/`](../strata/README.md), which records which is
|
||||
active. `nixos/configuration.nix` picks it with `strataConfig`; pi's `models.json` lists both model ids.
|
||||
Still serving IQ3_S until the switch.
|
||||
|
||||
## 2026-10-27: power cap 250 W vs 275 W
|
||||
|
||||
With the case closed, alternating runs (250/275/250/275, fresh restart each) gave +4.6% prompt speed at
|
||||
|
||||
@@ -114,10 +114,10 @@
|
||||
nix.settings.trusted-users = [ "root" "jason" ];
|
||||
systemd.tmpfiles.rules = [ "d /models/strata 0750 strata strata -" ];
|
||||
|
||||
# The API server (setup's run-iq3_s.sh) with offline mode and no downloads. The API key and engine args live in
|
||||
# strata-iq3_s.json, which setup wrote; host 0.0.0.0, port 8080.
|
||||
systemd.services.strata = {
|
||||
description = "Strata inference server (Qwen3.8-Flash-Next IQ3_S)";
|
||||
# The API server (setup's run-<model>.sh) with offline mode and no downloads. The API key and engine args live in
|
||||
# the run config setup wrote; host 0.0.0.0, port 8080. strataConfig picks the active one (repo: strata/README.md).
|
||||
systemd.services.strata = let strataConfig = "strata-iq3_s.json"; in {
|
||||
description = "Strata inference server (${strataConfig})";
|
||||
after = [ "network-online.target" "nvidia-persistenced.service" "nvidia-power-limit.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
requires = [ "nvidia-persistenced.service" ];
|
||||
@@ -134,7 +134,7 @@
|
||||
User = "strata";
|
||||
Group = "strata";
|
||||
WorkingDirectory = "/var/lib/strata/Strata";
|
||||
ExecStart = "/var/lib/strata/Strata/.venv/bin/python /var/lib/strata/Strata/serve/server.py --engine strata --config /var/lib/strata/Strata/strata-iq3_s.json --port 8080 --offline";
|
||||
ExecStart = "/var/lib/strata/Strata/.venv/bin/python /var/lib/strata/Strata/serve/server.py --engine strata --config /var/lib/strata/Strata/${strataConfig} --port 8080 --offline";
|
||||
Restart = "on-failure";
|
||||
RestartSec = 10;
|
||||
LimitMEMLOCK = "infinity"; # the expert arena and KV are pinned host memory
|
||||
|
||||
@@ -36,6 +36,23 @@
|
||||
"cacheRead": 0,
|
||||
"cacheWrite": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "qwen3.8-flash-next-unsloth-ud-q4_k_xl",
|
||||
"name": "Qwen3.8-Flash-Next UD-Q4_K_XL (lovey, Strata)",
|
||||
"reasoning": true,
|
||||
"input": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"contextWindow": 131072,
|
||||
"maxTokens": 32768,
|
||||
"cost": {
|
||||
"input": 0,
|
||||
"output": 0,
|
||||
"cacheRead": 0,
|
||||
"cacheWrite": 0
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
# Strata configurations
|
||||
|
||||
One file per configuration: the run config `setup.sh` wrote on the guest
|
||||
(`/var/lib/strata/Strata/strata-<model>.json`), with the API key replaced by a placeholder. The guest keeps
|
||||
every config it has run; the service starts whichever one `strataConfig` names in
|
||||
[`nixos/configuration.nix`](../nixos/configuration.nix).
|
||||
|
||||
**Active:** `strata-iq3_s.json`
|
||||
|
||||
| Config | Model / quant | Experts in RAM | Status | Benchmarks |
|
||||
|---|---|---|---|---|
|
||||
| [`strata-iq3_s.json`](strata-iq3_s.json) | Qwen3.8-Flash-Next, ISTA-DASLab GSQ-RCO IQ3_S | 47 GB arena + 29 GB PLE table | active since 2026-10-26 | [standard](../benchmarks/standard/README.md), husky-bench `*iq3_s*` |
|
||||
| [`strata-unsloth-ud-q4_k_xl.json`](strata-unsloth-ud-q4_k_xl.json) | Qwen3.8-Flash-Next, Unsloth UD-Q4_K_XL (pinned rev `38bb39e`), env `STRATA_HC_Q8=1`, `STRATA_ROUTE_TAIL_SKIP=0` | all 77 GB (setup's estimate) + 29 GB PLE table | installed 2026-10-27, testing | not run yet |
|
||||
|
||||
## Switching
|
||||
|
||||
1. Set `strataConfig` in `nixos/configuration.nix`, copy it to `/etc/nixos/`, run
|
||||
`sudo nixos-rebuild switch` (restarts `strata`; load takes ~2 min, watch `journalctl -u strata -f`).
|
||||
2. Add the config's `model_name` to pi's [`models.json`](../pi/models.json) if it's new; the API serves
|
||||
only the loaded model.
|
||||
3. Update **Active** above, the status column, and `docs/changelog.md`.
|
||||
|
||||
## Adding a configuration
|
||||
|
||||
Run `setup.sh` with `--no-start` (see [docs/strata.md](../docs/strata.md#rebuild)); it writes
|
||||
`strata-<model>.json` and `run-<model>.sh` next to the existing ones without touching them. Copy the
|
||||
JSON here, replace `api_key` with `<API_KEY: see /var/lib/strata/api-key>`, and add a row.
|
||||
|
||||
Name benchmark runs after the config so results stay attributable: husky-bench folders as
|
||||
`<model_name>_<power cap>_<effort>`, standard runs with the config in the README table.
|
||||
@@ -0,0 +1,89 @@
|
||||
{
|
||||
"exe": "/var/lib/strata/Strata/engine/strata",
|
||||
"args": [
|
||||
"--pack",
|
||||
"/models/strata/packs/unsloth-ud-q4_k_xl",
|
||||
"--native",
|
||||
"/models/strata/models/unsloth-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf",
|
||||
"--expert-profile",
|
||||
"/var/lib/strata/Strata/data/expert-profile.bin",
|
||||
"--expert-cache",
|
||||
"auto",
|
||||
"--prefill",
|
||||
"auto",
|
||||
"--spec",
|
||||
"4",
|
||||
"--spec-min-p",
|
||||
"0.5",
|
||||
"--mtp",
|
||||
"/models/strata/mtp/rt",
|
||||
"--max-context",
|
||||
"131072",
|
||||
"--kv",
|
||||
"int8",
|
||||
"--ple-io",
|
||||
"ram",
|
||||
"--kv-resident",
|
||||
"32768",
|
||||
"--vision",
|
||||
"--vram-reserve-mib",
|
||||
"700"
|
||||
],
|
||||
"cwd": "/var/lib/strata/Strata",
|
||||
"tokenizer": "/models/strata/packs/unsloth-ud-q4_k_xl/tokenizer",
|
||||
"model_name": "qwen3.8-flash-next-unsloth-ud-q4_k_xl",
|
||||
"log": "/var/lib/strata/Strata/strata-unsloth-ud-q4_k_xl.log",
|
||||
"lib_dirs": [
|
||||
"/nix/store/7mi3cxyj9rgx76yxr6jg4z8i3cvb3dmz-cuda-merged-13.0/bin"
|
||||
],
|
||||
"port": 8080,
|
||||
"env": {
|
||||
"STRATA_HC_Q8": "1",
|
||||
"STRATA_ROUTE_TAIL_SKIP": "0"
|
||||
},
|
||||
"gpu": [
|
||||
0,
|
||||
1
|
||||
],
|
||||
"gpus_asked": true,
|
||||
"layer_split": "auto",
|
||||
"host": "0.0.0.0",
|
||||
"api_key": "<API_KEY: see /var/lib/strata/api-key>",
|
||||
"open_browser": false,
|
||||
"vision": {
|
||||
"exe": "/var/lib/strata/Strata/engine/strata-vision",
|
||||
"mmproj": "/models/strata/models/mmproj-Qwen3.8-Flash-Next-BF16.gguf",
|
||||
"model": "/models/strata/models/unsloth-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf",
|
||||
"gpu": true,
|
||||
"max_tokens": 1024
|
||||
},
|
||||
"args_written": [
|
||||
"--pack",
|
||||
"/models/strata/packs/unsloth-ud-q4_k_xl",
|
||||
"--native",
|
||||
"/models/strata/models/unsloth-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf",
|
||||
"--expert-profile",
|
||||
"/var/lib/strata/Strata/data/expert-profile.bin",
|
||||
"--expert-cache",
|
||||
"auto",
|
||||
"--prefill",
|
||||
"auto",
|
||||
"--spec",
|
||||
"4",
|
||||
"--spec-min-p",
|
||||
"0.5",
|
||||
"--mtp",
|
||||
"/models/strata/mtp/rt",
|
||||
"--max-context",
|
||||
"131072",
|
||||
"--kv",
|
||||
"int8",
|
||||
"--ple-io",
|
||||
"ram",
|
||||
"--kv-resident",
|
||||
"32768",
|
||||
"--vision",
|
||||
"--vram-reserve-mib",
|
||||
"700"
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user