Document lovey setup: host, NixOS guest, Strata, pi client
README with overview, hardware, stack and rebuild order. Configs copied from the live machines at their real paths (proxmox/, nixos/), Strata's engine config and the pi client config with API keys removed. docs/ covers each layer, rebuild steps, benchmarks and a change log with the memory-instability and pi compaction-loop fixes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jason Witty
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
82e456892b
commit
e59662b873
@@ -1,3 +1,79 @@
|
|||||||
# lovey
|
# lovey
|
||||||
|
|
||||||
lovey is my inference server running a 2 3090s. this repo is to store and document different configurations, models and benchmarks.
|
lovey is my inference server running 2× RTX 3090s. This repo stores and documents its configuration,
|
||||||
|
models and benchmarks, and holds everything needed to rebuild the current setup from bare metal.
|
||||||
|
|
||||||
|
**Current state (2026-10-26):** Proxmox host → one NixOS VM with both GPUs passed through → Strata
|
||||||
|
serving Qwen3.8-Flash-Next IQ3_S on an OpenAI/Anthropic-compatible API, used from
|
||||||
|
[pi](https://pi.dev) on my workstation.
|
||||||
|
|
||||||
|
```
|
||||||
|
cosmic (workstation) lovey 192.168.1.53 (Proxmox VE 9.2)
|
||||||
|
┌──────────────────┐ LAN ┌──────────────────────────────────────────────────────┐
|
||||||
|
│ pi coding agent │──────────────▶│ VM 200 llm-nixos 192.168.1.67 (NixOS 26.05) │
|
||||||
|
│ ~/.pi/agent │ :8080 + key │ strata.service ── Qwen3.8-Flash-Next IQ3_S │
|
||||||
|
└──────────────────┘ │ 2× RTX 3090 (VFIO), gpu-unlock, 250 W cap │
|
||||||
|
│ /models ◀── virtiofs ── /mnt/models (NVMe XFS) │
|
||||||
|
│ gpu-temp-push ──UDP :7777──▶ lovey-fans.service │
|
||||||
|
└──────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Hardware
|
||||||
|
|
||||||
|
| Part | Detail |
|
||||||
|
|---|---|
|
||||||
|
| Board | ASUS ROG Maximus Z790 Hero |
|
||||||
|
| CPU | Intel i9-14900KS, microcode 0x133, 200 W PL1 / 253 W PL2, Intel default profile |
|
||||||
|
| RAM | 4× 32 GB = 128 GB DDR5, **run at 4800 MT/s JEDEC** (unstable at 5200+ with 4 DIMMs) |
|
||||||
|
| GPUs | 2× Zotac RTX 3090 24 GB at `01:00` and `02:00`, PCIe 4.0 x8 each, no NVLink |
|
||||||
|
| Boot disk | 250 GB SATA SSD (`sda`): Proxmox only |
|
||||||
|
| Data disk | MSI M470 2 TB NVMe, 4K LBA: model store + VM disks |
|
||||||
|
| Cooling | Liquid: radiator fans on nct6798 fan2–5 (driven by `lovey-fans`), pump on fan7, no coolant sensor |
|
||||||
|
| NIC | Onboard Intel 2.5G `eno2`, bridged as `vmbr0` |
|
||||||
|
|
||||||
|
## Software stack
|
||||||
|
|
||||||
|
| Layer | What | Docs |
|
||||||
|
|---|---|---|
|
||||||
|
| Host | Proxmox VE 9.2.21, kernel 7.0.14-pve, pve-no-subscription repos | [docs/proxmox-host.md](docs/proxmox-host.md) |
|
||||||
|
| VM 200 | `llm-nixos`: NixOS 26.05, kernel 6.18, NVIDIA open 595.71, CUDA 13.2 | [docs/nixos-guest.md](docs/nixos-guest.md) |
|
||||||
|
| GPU tuning | 250 W power cap + [gpu-unlock](https://github.com/Zagitta/gpu-unlock) (full-rate FP16/BF16) | [docs/nixos-guest.md](docs/nixos-guest.md#gpu-tuning) |
|
||||||
|
| Inference | [Strata](https://github.com/Niko1221/Strata) @ `4b9e05a2`, Qwen3.8-Flash-Next IQ3_S | [docs/strata.md](docs/strata.md) |
|
||||||
|
| Client | pi coding agent 1.4.0 | [docs/pi-client.md](docs/pi-client.md) |
|
||||||
|
| Fans | `lovey-fans.py` on the host, fed GPU temps by the guest | [docs/proxmox-host.md](docs/proxmox-host.md#fan-control) |
|
||||||
|
|
||||||
|
Current numbers: decode ~127–145 tok/s (MTP draft acceptance ~83%), cold prompt ~915 tok/s on 17K
|
||||||
|
tokens, model load ~2 min. See [docs/strata.md](docs/strata.md#benchmarks).
|
||||||
|
|
||||||
|
## Repo layout
|
||||||
|
|
||||||
|
```
|
||||||
|
nixos/ /etc/nixos on llm-nixos (configuration.nix, gpu-unlock.nix, hardware-configuration.nix)
|
||||||
|
proxmox/ files from lovey, at their real paths (etc/..., usr/local/bin/..., root/...)
|
||||||
|
strata/ strata-iq3_s.json, the engine config (API key removed)
|
||||||
|
pi/ ~/.pi/agent/models.json and settings.json for the client (API key removed)
|
||||||
|
docs/ per-layer notes, rebuild steps and the change log
|
||||||
|
```
|
||||||
|
|
||||||
|
**Secrets are not in this repo** (it is public). The Strata API key lives only in
|
||||||
|
`/var/lib/strata/api-key` and `strata-iq3_s.json` on the guest; placeholders are marked `<...>`.
|
||||||
|
|
||||||
|
## Rebuild order
|
||||||
|
|
||||||
|
1. BIOS: memory at 4800 JEDEC, CPU 200/253 W, VT-d on. See [proxmox-host.md](docs/proxmox-host.md#bios).
|
||||||
|
2. Proxmox host: repos, NVMe layout, VFIO, Gluster, fans. [proxmox-host.md](docs/proxmox-host.md#rebuild)
|
||||||
|
3. VM 200 from `proxmox/etc/pve/qemu-server/200.conf`, then NixOS from `nixos/`. [nixos-guest.md](docs/nixos-guest.md#rebuild)
|
||||||
|
4. Strata source build + model download. [strata.md](docs/strata.md#rebuild)
|
||||||
|
5. pi on the workstation. [pi-client.md](docs/pi-client.md)
|
||||||
|
|
||||||
|
## History
|
||||||
|
|
||||||
|
Everything that went wrong and what fixed it is in [docs/changelog.md](docs/changelog.md): the Z690
|
||||||
|
memory-channel fault, the 4-DIMM instability on the Z790, and the pi auto-compaction loop.
|
||||||
|
|
||||||
|
## Next up
|
||||||
|
|
||||||
|
- Strata tuning: `--prefill auto:32768`, `--pipeline-windows 2`, `--adapt-async 1` (community 2×3090
|
||||||
|
reports ~1,800–2,300 prompt tok/s vs our 915).
|
||||||
|
- Try other quants.
|
||||||
|
- Planned services as OCI LXCs on the host: SearXNG + degoog, Qdrant, Open WebUI, Tika, Whisper.
|
||||||
|
|||||||
@@ -0,0 +1,57 @@
|
|||||||
|
# Change log and fixes
|
||||||
|
|
||||||
|
Newest first. Each entry: what broke, the cause, and what fixed it.
|
||||||
|
|
||||||
|
## 2026-10-26: pi auto-compaction loop
|
||||||
|
|
||||||
|
**Symptom:** on long pi sessions, "Auto-compaction failed: Summarization failed: generation hit the
|
||||||
|
token cap and the summary is incomplete", retried every ~2 minutes, forever. `/thinking off` +
|
||||||
|
`/compact` also failed.
|
||||||
|
|
||||||
|
**Cause:**
|
||||||
|
1. Pi caps the summary request at the model's `maxTokens` (was 16384). Strata gave thinking 75% of
|
||||||
|
that (12288), leaving ~4K for the summary, and every attempt ran to 16384 tokens (`length`).
|
||||||
|
2. Pi's thinking level never reached Strata: the compat had `supportsReasoningEffort: false` and no
|
||||||
|
`thinkingFormat`, so pi sent no thinking control and Strata thought at full effort on every
|
||||||
|
request. That's why the thinking-off workaround did nothing.
|
||||||
|
|
||||||
|
**Fix** (in [`pi/`](../pi)):
|
||||||
|
- `maxTokens` 16384 → 32768
|
||||||
|
- `compaction: { reserveTokens: 32768, keepRecentTokens: 20000 }` (compact at ~98K instead of ~115K)
|
||||||
|
- compat `thinkingFormat: "chat-template"` + `chatTemplateKwargs` for `enable_thinking` and
|
||||||
|
`reasoning_effort`
|
||||||
|
|
||||||
|
Verified: `pi -p --thinking off` produced no thinking; `--thinking medium` got a 24,576-token
|
||||||
|
budget; the stuck session compacted with thinking off.
|
||||||
|
|
||||||
|
## 2026-10-26: Strata, firewall, fans, GPU tuning
|
||||||
|
|
||||||
|
- Strata installed and serving IQ3_S ([strata.md](strata.md)). NixOS firewall turned on (22, 8080).
|
||||||
|
- `lovey-fans` fan control on the host, `gpu-temp-push` in the guest.
|
||||||
|
- 250 W cap and gpu-unlock in the guest ([nixos-guest.md](nixos-guest.md#gpu-tuning)).
|
||||||
|
`configuration.nix.pre-unlock` on the guest is the config from before; generation 1 is plain.
|
||||||
|
- Guest switched to static IP, NetworkManager removed.
|
||||||
|
|
||||||
|
## 2026-10-25: board swap and 4-DIMM instability
|
||||||
|
|
||||||
|
Moved to ASUS Maximus Z790 Hero + i9-14900KS, 4× 32 GB, both 3090s.
|
||||||
|
|
||||||
|
| Memory speed | Result |
|
||||||
|
|---|---|
|
||||||
|
| 5400 MT/s | Bootloop: kernel oopses at `RIP 0x0` in `pve-startall`, `qm`/`pveversion` segfaults |
|
||||||
|
| 5200 MT/s | Random `pvesm status` segfaults, host dropped off the network within minutes |
|
||||||
|
| **4800 JEDEC, 1.1 V** | Stable: 15-min soak ([`root/stab.sh`](../proxmox/root/stab.sh)) with 0 failures |
|
||||||
|
|
||||||
|
- Every PVE/Perl file checksum-verified clean: the corruption was in RAM, not on disk. Cause is
|
||||||
|
4-DIMM speed, not CPU degradation.
|
||||||
|
- Cheap tell: "Initramfs unpacking failed: ZSTD-compressed data is corrupt" at boot means memory is
|
||||||
|
unstable on this box. It disappears at 4800.
|
||||||
|
- `vmbr0` moved to `eno2` (Proxmox pins NIC names to MACs).
|
||||||
|
- NixOS 26.05 installed in VM 200, which got both GPUs and 112 GB.
|
||||||
|
|
||||||
|
## 2026-10-12: Z690 memory channel A fault
|
||||||
|
|
||||||
|
On the original MSI Z690 Carbon, memory channel A was contact-marginal: under load the box hard-froze
|
||||||
|
(purple/static on the iGPU, no logs). Memtest froze with 4 DIMMs or one DIMM in A after ~4 min with
|
||||||
|
0 errors; channel B alone was stable. CPU and RAM were cleared on another board. Resolved by moving
|
||||||
|
to the Z790 Hero. Crash-capture sysctls from this debugging are still on the host.
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
# NixOS guest (VM 200 `llm-nixos`, 192.168.1.67)
|
||||||
|
|
||||||
|
NixOS 26.05, systemd-boot, kernel 6.18, NVIDIA open driver 595.71 (CUDA 13.2). The complete config
|
||||||
|
is in [`nixos/`](../nixos) and is the source of truth: `/etc/nixos` on the guest should match it.
|
||||||
|
|
||||||
|
## What the config sets up
|
||||||
|
|
||||||
|
- **Network:** static 192.168.1.67/24 on `ens18`, gateway .1, DNS 1.1.1.1/8.8.8.8 (NetworkManager
|
||||||
|
removed). Static because `lovey-fans` only trusts GPU temps from this address. Firewall on: TCP 22
|
||||||
|
(SSH) and 8080 (Strata).
|
||||||
|
- **Users:** `jason` and root are SSH-key only; sudo needs no password.
|
||||||
|
- **NVIDIA:** open kernel module, headless, `nvidia-persistenced`. `nvidia-container-toolkit` +
|
||||||
|
podman (`--device nvidia.com/gpu=all`).
|
||||||
|
- **CUDA tooling:** `cuda_nvcc` and the cuda-maintainers cachix substituter, so nothing CUDA-heavy
|
||||||
|
compiles locally. `programs.nix-ld` with `config.hardware.nvidia.package` in its libraries:
|
||||||
|
without that, pip/uv torch and vLLM can't find `libcuda` and `torch.cuda.is_available()` is False.
|
||||||
|
- **Memory:** memlock unlimited (PAM and systemd default), needed for pinned host memory.
|
||||||
|
- **/models:** virtiofs from lovey's `/mnt/models`, `nofail`.
|
||||||
|
- **Services:** `nvidia-power-limit`, `gpu-temp-push`, `strata` (see [strata.md](strata.md)).
|
||||||
|
|
||||||
|
## GPU tuning
|
||||||
|
|
||||||
|
- **Power cap:** `nvidia-power-limit` runs `nvidia-smi -pl 250` after persistenced on every boot
|
||||||
|
(stock is 350 W).
|
||||||
|
- **gpu-unlock** ([`nixos/gpu-unlock.nix`](../nixos/gpu-unlock.nix)): the
|
||||||
|
[Zagitta/gpu-unlock](https://github.com/Zagitta/gpu-unlock) kernel driver, pinned at `af28cbb`,
|
||||||
|
enables full-rate FP16/BF16/TF32 tensor math on GA102. It runs inside the guest and flashes nothing.
|
||||||
|
The Nix file builds the module for the guest kernel and generates its firmware from pinned NVIDIA
|
||||||
|
610.43.03 inputs (all SHA-256 checked), loads `gpu_unlock` from the initrd before `nvidia`, and sets
|
||||||
|
`gpu_unlock.targets=all`. To disable it, drop that kernel parameter.
|
||||||
|
- Check: `journalctl -k -b | grep -E "gpu_unlock|entry:"`. `unlocked` appears only on the first boot
|
||||||
|
after lovey powers off. After a VM reboot it logs `unlock failed (-114)` (-EALREADY): the cards
|
||||||
|
are still unlocked, which is harmless.
|
||||||
|
- The unlock survives VM reboots and host bus resets; only a host power-off clears it.
|
||||||
|
|
||||||
|
Dense 8192³ GEMM, torch 2.14 / cu130:
|
||||||
|
|
||||||
|
| Power limit | FP16 | BF16 |
|
||||||
|
|---|---|---|
|
||||||
|
| 250 W | 77–81 TFLOPS | 80–86 TFLOPS |
|
||||||
|
| 350 W | 112–114 TFLOPS | 115–116 TFLOPS |
|
||||||
|
|
||||||
|
A stock 3090 peaks at about 71 TFLOPS. The 250 W cap costs ~30% of matmul (prefill) throughput;
|
||||||
|
decode is bandwidth-bound and barely changes.
|
||||||
|
|
||||||
|
## Temps
|
||||||
|
|
||||||
|
4-minute dual-GPU BF16 burn at 250 W: GPU0 34→45°C, GPU1 44→55°C (still creeping), fans ~36%.
|
||||||
|
GPU1 runs ~10°C warmer than GPU0 (loop order or block contact). Plenty of headroom.
|
||||||
|
|
||||||
|
## Rebuild
|
||||||
|
|
||||||
|
1. Create VM 200 ([proxmox-host.md](proxmox-host.md#vm-200-llm-nixos)) and boot the NixOS 26.05
|
||||||
|
minimal ISO.
|
||||||
|
2. Partition: 1 GB ESP (vfat) + ext4 root on the 100 GB disk; mount at `/mnt` and `/mnt/boot`.
|
||||||
|
3. `nixos-generate-config --root /mnt`, then copy `configuration.nix` and `gpu-unlock.nix` from this
|
||||||
|
repo. Keep the **generated** `hardware-configuration.nix` (its disk UUIDs are new) but add the
|
||||||
|
`/models` virtiofs entry from the copy here.
|
||||||
|
4. `nixos-install`, reboot, detach the ISO.
|
||||||
|
5. Check: `nvidia-smi` shows both 3090s at 250 W, `ls /models`, `systemctl status gpu-temp-push`
|
||||||
|
and `journalctl -u lovey-fans` on the host shows a GPU temperature.
|
||||||
|
6. Install Strata: [strata.md](strata.md#rebuild).
|
||||||
|
|
||||||
|
Applying a config change: edit here, copy to `/etc/nixos/`, `sudo nixos-rebuild switch`, commit.
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
# pi client (cosmic)
|
||||||
|
|
||||||
|
[pi](https://pi.dev) coding agent 1.4.0 on the workstation, pointed at Strata. Config lives in
|
||||||
|
`~/.pi/agent/`; the copies in [`pi/`](../pi) have the API key replaced with `<STRATA_API_KEY>`.
|
||||||
|
|
||||||
|
## models.json
|
||||||
|
|
||||||
|
[`pi/models.json`](../pi/models.json) defines provider `strata`:
|
||||||
|
|
||||||
|
- `baseUrl: http://192.168.1.67:8080/v1`, `api: openai-completions`
|
||||||
|
- `contextWindow: 131072` (matches Strata's `--max-context`)
|
||||||
|
- `maxTokens: 32768`
|
||||||
|
- `reasoning: true`, `input: text + image`
|
||||||
|
- `compat`:
|
||||||
|
- `supportsDeveloperRole: false`, `supportsReasoningEffort: false`
|
||||||
|
- `thinkingFormat: "chat-template"` with `chatTemplateKwargs`
|
||||||
|
`{ enable_thinking: {$var: thinking.enabled}, reasoning_effort: {$var: thinking.effort, omitWhenOff: true} }`.
|
||||||
|
**Without this, pi's thinking level never reaches Strata** and every request thinks at full effort.
|
||||||
|
|
||||||
|
## settings.json
|
||||||
|
|
||||||
|
[`pi/settings.json`](../pi/settings.json): default provider/model, thinking `medium`, and
|
||||||
|
|
||||||
|
```json
|
||||||
|
"compaction": { "reserveTokens": 32768, "keepRecentTokens": 20000 }
|
||||||
|
```
|
||||||
|
|
||||||
|
Pi compacts once context exceeds `contextWindow - reserveTokens`, so this starts at ~98K. Keep
|
||||||
|
`reserveTokens` ≥ `maxTokens`.
|
||||||
|
|
||||||
|
## Thinking levels
|
||||||
|
|
||||||
|
| pi level | What Strata does |
|
||||||
|
|---|---|
|
||||||
|
| off | No thinking |
|
||||||
|
| low | System note asking for brief thinking |
|
||||||
|
| minimal / medium | System note: think as much as the task needs |
|
||||||
|
| high / xhigh / max | Unrestricted thinking |
|
||||||
|
|
||||||
|
Use medium day to day, high for hard debugging or planning. Compaction reuses the session's level.
|
||||||
|
|
||||||
|
## How compaction works (and why it failed)
|
||||||
|
|
||||||
|
Pi sends one summary request containing everything except the last `keepRecentTokens`. Its output is
|
||||||
|
capped at the model's `maxTokens` and it uses the current thinking level. If the reply ends with
|
||||||
|
`stopReason: "length"`, pi refuses to save the partial summary ("generation hit the token cap and the
|
||||||
|
summary is incomplete") and tries again on the next turn. See the
|
||||||
|
[2026-10-26 changelog entry](changelog.md#2026-10-26-pi-auto-compaction-loop).
|
||||||
|
|
||||||
|
If it ever happens again: `/thinking off`, `/compact`, then set thinking back.
|
||||||
|
|
||||||
|
`maxTokens` costs no VRAM. Strata sizes the KV cache from `--max-context` / `--kv-resident` at engine
|
||||||
|
start; `max_tokens` is only a per-request stop, and Strata shortens it to the room left in the window.
|
||||||
@@ -0,0 +1,98 @@
|
|||||||
|
# Proxmox host (lovey, 192.168.1.53)
|
||||||
|
|
||||||
|
Proxmox VE 9.2.21 on Debian trixie, kernel 7.0.14-pve. Web UI at https://192.168.1.53:8006.
|
||||||
|
All files referenced here are in [`proxmox/`](../proxmox) at their real paths.
|
||||||
|
|
||||||
|
## BIOS
|
||||||
|
|
||||||
|
- **Memory: 4800 MT/s JEDEC, 1.1 V.** With 4 DIMMs, 5400 bootlooped and 5200 corrupted memory (see
|
||||||
|
[changelog](changelog.md#2026-10-25-board-swap-and-4-dimm-instability)). 5000 is untested; soak with
|
||||||
|
`systemd-run --unit=stab /root/stab.sh` before trusting it.
|
||||||
|
- CPU: Intel default profile, PL1 200 W / PL2 253 W. Set it in the BIOS; a RAPL limit set from Linux
|
||||||
|
does not survive a reboot.
|
||||||
|
- VT-d on (needed for passthrough).
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
|
||||||
|
| Device | Use |
|
||||||
|
|---|---|
|
||||||
|
| `sda` 250 GB | Proxmox only: VG `pve` (`root`, `swap`, plus an unused `data` thin pool that could be removed to grow root) |
|
||||||
|
| NVMe 2 TB | One GPT partition, VG `nvme` |
|
||||||
|
| ↳ LV `models` 900 GB XFS | `/mnt/models` (fstab, `noatime`), shared into VMs as virtiofs mapping `models` |
|
||||||
|
| ↳ thin pool `guests` 700 GB | Storage `nvme-thin` for VM disks |
|
||||||
|
| ↳ ~262 GB free | Left unallocated on purpose: neither XFS nor a thin pool can shrink |
|
||||||
|
| Gluster `gvolume0` | `odroid1:/gvolume0` at `/mnt/gluster`, storage `gluster` (dir, `is_mountpoint`) for ISOs and backups. PVE 9 dropped the native Gluster plugin, so it is a plain fuse mount |
|
||||||
|
|
||||||
|
Files: `etc/fstab`, `etc/pve/storage.cfg`, `etc/pve/mapping/directory.cfg`.
|
||||||
|
|
||||||
|
## GPU passthrough (VFIO)
|
||||||
|
|
||||||
|
- GRUB: `intel_iommu=on iommu=pt` (`etc/default/grub.cmdline`).
|
||||||
|
- `etc/modprobe.d/vfio.conf` binds `10de:2204` (3090) and `10de:1aef` (its HDMI audio) to vfio-pci,
|
||||||
|
which covers both cards. nouveau is blacklisted (`blacklist-nouveau.conf`).
|
||||||
|
- The cards are alone in IOMMU groups 21 (`01:00`) and 22 (`02:00`) on the Z790 Hero.
|
||||||
|
- QEMU logs "PCI peer-to-peer transactions on BARs are not supported". That is expected: GPU-to-GPU
|
||||||
|
traffic goes through host memory and NCCL falls back to SHM.
|
||||||
|
|
||||||
|
## VM 200 `llm-nixos`
|
||||||
|
|
||||||
|
[`etc/pve/qemu-server/200.conf`](../proxmox/etc/pve/qemu-server/200.conf):
|
||||||
|
|
||||||
|
- q35 / OVMF, `cpu: host`, 16 vCPU pinned to threads 0–15 (the P-cores; 16–31 are E-cores)
|
||||||
|
- 112 GB RAM, balloon 0. Passthrough pins all guest RAM, so leave ~10 GB for the host and LXCs
|
||||||
|
- `hostpci0: 01:00`, `hostpci1: 02:00` (both `pcie=1`)
|
||||||
|
- 100 GB disk on `nvme-thin`, virtiofs `models`, qemu guest agent
|
||||||
|
|
||||||
|
## Network
|
||||||
|
|
||||||
|
`etc/network/interfaces`: `vmbr0` on `eno2`, static 192.168.1.53/24, gateway .1.
|
||||||
|
|
||||||
|
Gotcha: the Proxmox installer pins NIC names to MACs in `/usr/local/lib/systemd/network/50-pmx-nic*.link`.
|
||||||
|
The three files still there match the old Z690's MACs and are harmless (not copied here). On a board
|
||||||
|
swap, either re-pin before swapping or fix `vmbr0` from the local console.
|
||||||
|
|
||||||
|
## Fan control
|
||||||
|
|
||||||
|
The GPUs belong to the VM, so the BIOS can't see their temperature. `lovey-fans.service` runs
|
||||||
|
[`usr/local/bin/lovey-fans.py`](../proxmox/usr/local/bin/lovey-fans.py), which sets fan2–5 to
|
||||||
|
max(CPU curve, GPU curve) every 2 s.
|
||||||
|
|
||||||
|
| Header | What | Control |
|
||||||
|
|---|---|---|
|
||||||
|
| fan2–fan5 | Radiator fans, ~1700 rpm max | `lovey-fans` |
|
||||||
|
| fan6 | Fan-like, BIOS-pinned at 100%, probably the AIO_PUMP header | BIOS |
|
||||||
|
| fan7 | Pump, ~3180 rpm, ignores PWM | BIOS |
|
||||||
|
|
||||||
|
- Curves: CPU 50°C→30%, 70°C→50%, 85°C→100%; GPU 45°C→30%, 60°C→40%, 75°C→100%. EMA-smoothed,
|
||||||
|
instant ramp-up, at most 2% down per tick.
|
||||||
|
- The guest's `gpu-temp-push` sends its hottest GPU temp as UDP to `192.168.1.53:7777`. Only packets
|
||||||
|
from `192.168.1.67` count: **if the guest IP changes, update `GPU_SOURCE`**, otherwise the fans sit at 85%.
|
||||||
|
- Fail-safes (tested): feed stale >15 s while VM 200 runs → 85%; VM off → GPUs ignored; daemon stops
|
||||||
|
or dies → pwm2–5 return to BIOS SmartFan (`pwm_enable=5`, also in `ExecStopPost`).
|
||||||
|
- The `nct6775` driver loads from `etc/modules-load.d/nct6775.conf`. Logs: `journalctl -u lovey-fans`
|
||||||
|
(one line a minute).
|
||||||
|
|
||||||
|
## Crash capture
|
||||||
|
|
||||||
|
Left on from the Z690 debugging, harmless: `etc/sysctl.d/90-crash-capture.conf` (lockup → panic →
|
||||||
|
reboot after 10 s, dump to EFI pstore) and `etc/systemd/journald.conf.d/90-sync.conf` (`SyncIntervalSec=15s`).
|
||||||
|
|
||||||
|
## Rebuild
|
||||||
|
|
||||||
|
1. Install Proxmox VE 9 on `sda`.
|
||||||
|
2. Repos: copy `etc/apt/sources.list.d/*` (enterprise disabled, pve-no-subscription on, Ceph over
|
||||||
|
http because https failed the cert check), then `apt update && apt full-upgrade`.
|
||||||
|
3. NVMe: reformat to 4K LBA (`nvme format --lbaf=<4K index>`), one GPT partition, then
|
||||||
|
```
|
||||||
|
pvcreate /dev/nvme0n1p1 && vgcreate nvme /dev/nvme0n1p1
|
||||||
|
lvcreate -L 900G -n models nvme && mkfs.xfs /dev/nvme/models
|
||||||
|
lvcreate -L 700G --thinpool guests nvme
|
||||||
|
```
|
||||||
|
Copy `etc/fstab` lines, `etc/pve/storage.cfg` and `etc/pve/mapping/directory.cfg`.
|
||||||
|
4. VFIO: GRUB cmdline from `etc/default/grub.cmdline`, `update-grub`, copy `etc/modprobe.d/*`,
|
||||||
|
`update-initramfs -u -k all`, reboot, check `lspci -nnk -s 01:00` shows `vfio-pci`.
|
||||||
|
5. Gluster: `apt install glusterfs-client`, fstab line, `mkdir -p /mnt/gluster/proxmox`.
|
||||||
|
6. Fans: copy `usr/local/bin/lovey-fans.py`, `etc/systemd/system/lovey-fans.service`,
|
||||||
|
`etc/modules-load.d/nct6775.conf`, then `systemctl enable --now lovey-fans`.
|
||||||
|
7. Create VM 200 from `etc/pve/qemu-server/200.conf` (new disks on `nvme-thin`) and install NixOS:
|
||||||
|
[nixos-guest.md](nixos-guest.md).
|
||||||
+109
@@ -0,0 +1,109 @@
|
|||||||
|
# Strata: Qwen3.8-Flash-Next IQ3_S
|
||||||
|
|
||||||
|
[Niko1221/Strata](https://github.com/Niko1221/Strata) (MIT) is an MoE expert-offload engine built on
|
||||||
|
llama.cpp. It serves OpenAI- and Anthropic-compatible APIs and splits layers across both 3090s as a
|
||||||
|
pipeline, so it needs no NVLink.
|
||||||
|
|
||||||
|
- API: `http://192.168.1.67:8080/v1`, API key required
|
||||||
|
- Model: Qwen3.8-Flash-Next, ISTA-DASLab GSQ-RCO **IQ3_S**, plus the BF16 mmproj for images and the
|
||||||
|
MTP draft layer
|
||||||
|
- Engine: **source build pinned at `4b9e05a2a806a3fe24f7b2560d98626c37dade35`** (2026-10-25), CUDA 13.0
|
||||||
|
from the repo's flake
|
||||||
|
|
||||||
|
## Decisions
|
||||||
|
|
||||||
|
- IQ3_S quant
|
||||||
|
- Source build at a reviewed commit. The prebuilt engine zip installs unverified if GitHub returns no
|
||||||
|
digest, so we don't use it.
|
||||||
|
- Whole LAN with an API key (default would bind 127.0.0.1)
|
||||||
|
- Images on, GPU reader
|
||||||
|
|
||||||
|
## Security review
|
||||||
|
|
||||||
|
Read statically, key points checked against the code:
|
||||||
|
|
||||||
|
- No telemetry and no auto-update. Model files are pinned to an HF commit and SHA-256 checked.
|
||||||
|
- Any client with the key can make the server read **any local file** as an "image", or fetch any
|
||||||
|
URL. `--offline` blocks URL fetches, and the systemd sandbox (`ProtectSystem=strict`,
|
||||||
|
`ProtectHome`, writes only to `/var/lib/strata` and `/models/strata`) limits which files it sees.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
| Path | What |
|
||||||
|
|---|---|
|
||||||
|
| `/var/lib/strata/Strata` | Source checkout, `.venv`, built engine, `strata-iq3_s.json` |
|
||||||
|
| `/var/lib/strata/api-key` | The API key (0600, user `strata`) |
|
||||||
|
| `/models/strata/models/IQ3_S/` | The two GGUF shards |
|
||||||
|
| `/models/strata/models/mmproj-…-BF16.gguf` | Vision projector |
|
||||||
|
| `/models/strata/packs/iq3_s`, `/models/strata/mtp/rt` | Expert pack and MTP draft layer |
|
||||||
|
|
||||||
|
[`strata/strata-iq3_s.json`](../strata/strata-iq3_s.json) is the config with the key removed. Key
|
||||||
|
engine args: `--max-context 131072`, `--kv int8`, `--kv-resident 32768`, `--spec 4 --spec-min-p 0.5`
|
||||||
|
with MTP, `--expert-cache auto`, `--prefill auto`, `--ple-io ram`, `--vision`, `--vram-reserve-mib 700`.
|
||||||
|
|
||||||
|
The service is `systemd.services.strata` in [`nixos/configuration.nix`](../nixos/configuration.nix):
|
||||||
|
`serve/server.py --engine strata --config strata-iq3_s.json --port 8080 --offline`,
|
||||||
|
`STRATA_NO_ACQUIRE=1`, `LimitMEMLOCK=infinity`, plus the sandboxing above.
|
||||||
|
|
||||||
|
## Gotchas
|
||||||
|
|
||||||
|
- The engine's RUNPATH resolves `libcuda.so.1` to the CUDA toolkit's **stub**. The unit sets
|
||||||
|
`LD_LIBRARY_PATH=/run/opengl-driver/lib` so the real driver wins.
|
||||||
|
- The CUDA 13.0 toolkit only exists in the flake's dev shell. It is pinned by the GC root
|
||||||
|
`/nix/var/nix/gcroots/strata-cuda-13.0` → `/nix/store/…-cuda-merged-13.0`, so `nix-collect-garbage`
|
||||||
|
can't break the engine. Recreate it after a rebuild (below).
|
||||||
|
- Running setup under `systemd-run` needs `SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt`, or it
|
||||||
|
(correctly) rejects GitHub's certificate.
|
||||||
|
- Qwen thinks first (`reasoning_content`); a small `max_tokens` can end before any `content`.
|
||||||
|
- Thinking control: Strata reads `chat_template_kwargs.enable_thinking` and `reasoning_effort`. With
|
||||||
|
neither, it thinks at full effort. Effort `low`/`medium` add a short system note before the answer
|
||||||
|
(unknown values act as medium); `high`/`xhigh` or none mean unrestricted thinking.
|
||||||
|
- With thinking on and no explicit budget, Strata caps thinking at 75% of `max_tokens`
|
||||||
|
("answer_room"), and shortens `max_tokens` to whatever room the prompt leaves in the window rather
|
||||||
|
than rejecting the request.
|
||||||
|
|
||||||
|
## Benchmarks
|
||||||
|
|
||||||
|
2026-10-26, engine at `4b9e05a2`, 250 W cap, gpu-unlock active:
|
||||||
|
|
||||||
|
| Metric | Result |
|
||||||
|
|---|---|
|
||||||
|
| Load | ~2 min: 46.8 GiB of experts into RAM at 1.2 GiB/s over virtiofs, 9,508 experts (17 GiB) in VRAM, 29 GB PLE table in RAM |
|
||||||
|
| Decode (code, 1,500 tokens, greedy) | 127.5 tok/s, draft acceptance 83% |
|
||||||
|
| Decode (long agent turns, pi) | 142–158 tok/s, expert cache ~99% hit |
|
||||||
|
| 17K-token cold prompt | 915 tok/s prompt, 113 tok/s output; repeats reuse the cached prefix |
|
||||||
|
| Community 2×3090 reference | ~137 decode, 1,800–2,300 prompt tok/s with tuning |
|
||||||
|
|
||||||
|
Untried tuning: `--prefill auto:32768` (setup's tip, +21–35% prompt), `--pipeline-windows 2`,
|
||||||
|
`--adapt-async 1`.
|
||||||
|
|
||||||
|
Memory at idle with the model loaded: both GPUs ~23.8 of 24.5 GB used, guest ~76 GB RAM used with
|
||||||
|
~33 GB available.
|
||||||
|
|
||||||
|
## Rebuild
|
||||||
|
|
||||||
|
As user `strata` on the guest (after the NixOS config is applied):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd /var/lib/strata
|
||||||
|
git clone https://github.com/Niko1221/Strata.git && cd Strata
|
||||||
|
git checkout 4b9e05a2a806a3fe24f7b2560d98626c37dade35
|
||||||
|
nix develop -c ./setup.sh --build --cuda 13 --gpus 0,1 --family qwen --model IQ3_S --vision gpu \
|
||||||
|
--data-dir /models/strata --host 0.0.0.0 --api-key "$(cat /var/lib/strata/api-key)" \
|
||||||
|
--no-start --no-browser --yes
|
||||||
|
```
|
||||||
|
|
||||||
|
Generate a new key first (`openssl rand -hex 24 > /var/lib/strata/api-key; chmod 600 …`) if the old
|
||||||
|
one is lost. If `/models/strata` survived (it's on the host's NVMe), setup reuses the downloaded files.
|
||||||
|
|
||||||
|
Then pin the toolkit and start the service:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cuda=$(sudo -u strata -H nix develop /var/lib/strata/Strata -c sh -c 'dirname $(dirname $(command -v nvcc))')
|
||||||
|
echo "$cuda" # today: /nix/store/7mi3cxyj9rgx76yxr6jg4z8i3cvb3dmz-cuda-merged-13.0
|
||||||
|
sudo nix-store --add-root /nix/var/nix/gcroots/strata-cuda-13.0 -r "$cuda"
|
||||||
|
sudo systemctl enable --now strata
|
||||||
|
journalctl -u strata -f # wait for READY, ~2 min
|
||||||
|
```
|
||||||
|
|
||||||
|
Compare the resulting `strata-iq3_s.json` with [`strata/strata-iq3_s.json`](../strata/strata-iq3_s.json).
|
||||||
@@ -0,0 +1,164 @@
|
|||||||
|
# llm-nixos: VM 200 on lovey. 2x RTX 3090 via VFIO, model weights on virtiofs /models.
|
||||||
|
{ config, lib, pkgs, ... }:
|
||||||
|
|
||||||
|
{
|
||||||
|
imports = [ ./hardware-configuration.nix ./gpu-unlock.nix ];
|
||||||
|
|
||||||
|
boot.loader.systemd-boot.enable = true;
|
||||||
|
boot.loader.efi.canTouchEfiVariables = true;
|
||||||
|
boot.loader.systemd-boot.configurationLimit = 5; # initrds carry ~84 MB of gpu-unlock firmware
|
||||||
|
boot.kernelPackages = pkgs.linuxPackages; # LTS-ish default, keeps the NVIDIA module happy
|
||||||
|
|
||||||
|
networking.hostName = "llm-nixos";
|
||||||
|
# Static: lovey-fans only accepts GPU temps from this address
|
||||||
|
networking.useDHCP = false;
|
||||||
|
networking.interfaces.ens18.ipv4.addresses = [ { address = "192.168.1.67"; prefixLength = 24; } ];
|
||||||
|
networking.defaultGateway = { address = "192.168.1.1"; interface = "ens18"; };
|
||||||
|
networking.nameservers = [ "1.1.1.1" "8.8.8.8" ];
|
||||||
|
networking.firewall.enable = true;
|
||||||
|
networking.firewall.allowedTCPPorts = [ 22 8080 ]; # SSH, Strata API (key required)
|
||||||
|
|
||||||
|
time.timeZone = "America/Los_Angeles";
|
||||||
|
i18n.defaultLocale = "en_US.UTF-8";
|
||||||
|
|
||||||
|
nixpkgs.config.allowUnfree = true;
|
||||||
|
nix.settings = {
|
||||||
|
experimental-features = [ "nix-command" "flakes" ];
|
||||||
|
# prebuilt CUDA packages, so nothing CUDA-heavy compiles locally
|
||||||
|
substituters = [ "https://cache.nixos.org" "https://cuda-maintainers.cachix.org" ];
|
||||||
|
trusted-public-keys = [
|
||||||
|
"cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
|
||||||
|
"cuda-maintainers.cachix.org-1:0dq3bujKpuEPMCX6U4WylrUDZ9JyUG0VpVZa7CNfq5E="
|
||||||
|
];
|
||||||
|
};
|
||||||
|
|
||||||
|
# NVIDIA: open kernel module (Ampere is supported), headless, persistence daemon
|
||||||
|
services.xserver.videoDrivers = [ "nvidia" ];
|
||||||
|
hardware.graphics.enable = true;
|
||||||
|
hardware.nvidia = {
|
||||||
|
open = true;
|
||||||
|
package = config.boot.kernelPackages.nvidiaPackages.stable;
|
||||||
|
nvidiaPersistenced = true;
|
||||||
|
nvidiaSettings = false;
|
||||||
|
};
|
||||||
|
# Cap both 3090s at 250 W (default 350 W); reapplied on every boot
|
||||||
|
systemd.services.nvidia-power-limit = {
|
||||||
|
description = "Set NVIDIA GPU power limit";
|
||||||
|
after = [ "nvidia-persistenced.service" ];
|
||||||
|
requires = [ "nvidia-persistenced.service" ];
|
||||||
|
wantedBy = [ "multi-user.target" ];
|
||||||
|
serviceConfig = {
|
||||||
|
Type = "oneshot";
|
||||||
|
RemainAfterExit = true;
|
||||||
|
ExecStart = "${config.hardware.nvidia.package.bin}/bin/nvidia-smi -pl 250";
|
||||||
|
};
|
||||||
|
};
|
||||||
|
# lovey's fan daemon (lovey-fans) can't see passed-through GPUs; send it the hottest one every 2 s
|
||||||
|
systemd.services.gpu-temp-push = {
|
||||||
|
description = "Push hottest GPU temp to lovey fan control";
|
||||||
|
after = [ "nvidia-persistenced.service" "network-online.target" ];
|
||||||
|
wants = [ "network-online.target" ];
|
||||||
|
requires = [ "nvidia-persistenced.service" ];
|
||||||
|
wantedBy = [ "multi-user.target" ];
|
||||||
|
path = [ config.hardware.nvidia.package.bin pkgs.coreutils ];
|
||||||
|
script = ''
|
||||||
|
while true; do
|
||||||
|
t=$(nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader,nounits | sort -n | tail -1)
|
||||||
|
echo "$t" > /dev/udp/192.168.1.53/7777 || true
|
||||||
|
sleep 2
|
||||||
|
done
|
||||||
|
'';
|
||||||
|
serviceConfig.Restart = "always";
|
||||||
|
};
|
||||||
|
hardware.nvidia-container-toolkit.enable = true; # GPUs inside podman containers (--device nvidia.com/gpu=all)
|
||||||
|
virtualisation.podman.enable = true;
|
||||||
|
|
||||||
|
# Model weights shared from the host (lovey:/mnt/models); don't block boot if it's missing
|
||||||
|
fileSystems."/models".options = [ "nofail" ];
|
||||||
|
|
||||||
|
# Pinned/locked memory for CUDA, NCCL and vLLM
|
||||||
|
security.pam.loginLimits = [
|
||||||
|
{ domain = "*"; type = "-"; item = "memlock"; value = "unlimited"; }
|
||||||
|
];
|
||||||
|
systemd.settings.Manager.DefaultLimitMEMLOCK = "infinity";
|
||||||
|
|
||||||
|
# Prebuilt Python wheels (pip/uv vLLM, torch) need a normal dynamic loader
|
||||||
|
programs.nix-ld.enable = true;
|
||||||
|
# libcuda/libnvidia-ml for pip/uv wheels (torch, vLLM); on NixOS they live in /run/opengl-driver/lib
|
||||||
|
programs.nix-ld.libraries = [ config.hardware.nvidia.package pkgs.stdenv.cc.cc.lib pkgs.zlib ];
|
||||||
|
|
||||||
|
services.qemuGuest.enable = true;
|
||||||
|
services.openssh = {
|
||||||
|
enable = true;
|
||||||
|
settings.PasswordAuthentication = false;
|
||||||
|
settings.PermitRootLogin = "prohibit-password";
|
||||||
|
};
|
||||||
|
|
||||||
|
users.users.jason = {
|
||||||
|
isNormalUser = true;
|
||||||
|
extraGroups = [ "wheel" "video" ];
|
||||||
|
openssh.authorizedKeys.keys = [
|
||||||
|
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINZMA1RumOjm198i2Jdaa1lw7vTWnnZKRuYwYhLp6YKx jason@cosmic"
|
||||||
|
];
|
||||||
|
};
|
||||||
|
# Strata inference engine: source checkout + venv in /var/lib/strata, model data on /models/strata
|
||||||
|
users.groups.strata = { };
|
||||||
|
users.users.strata = {
|
||||||
|
isSystemUser = true;
|
||||||
|
group = "strata";
|
||||||
|
extraGroups = [ "video" ];
|
||||||
|
home = "/var/lib/strata";
|
||||||
|
createHome = true;
|
||||||
|
shell = pkgs.bashInteractive; # setup runs as this user
|
||||||
|
};
|
||||||
|
nix.settings.trusted-users = [ "root" "jason" ];
|
||||||
|
systemd.tmpfiles.rules = [ "d /models/strata 0750 strata strata -" ];
|
||||||
|
|
||||||
|
# The API server (setup's run-iq3_s.sh) with offline mode and no downloads. The API key and engine args live in
|
||||||
|
# strata-iq3_s.json, which setup wrote; host 0.0.0.0, port 8080.
|
||||||
|
systemd.services.strata = {
|
||||||
|
description = "Strata inference server (Qwen3.8-Flash-Next IQ3_S)";
|
||||||
|
after = [ "network-online.target" "nvidia-persistenced.service" "nvidia-power-limit.service" ];
|
||||||
|
wants = [ "network-online.target" ];
|
||||||
|
requires = [ "nvidia-persistenced.service" ];
|
||||||
|
unitConfig.RequiresMountsFor = [ "/models" ];
|
||||||
|
wantedBy = [ "multi-user.target" ];
|
||||||
|
environment = {
|
||||||
|
HOME = "/var/lib/strata";
|
||||||
|
# The engine's RUNPATH points libcuda at the CUDA toolkit's stub; the real driver must come first.
|
||||||
|
LD_LIBRARY_PATH = "/run/opengl-driver/lib";
|
||||||
|
SSL_CERT_FILE = "/etc/ssl/certs/ca-certificates.crt";
|
||||||
|
STRATA_NO_ACQUIRE = "1";
|
||||||
|
};
|
||||||
|
serviceConfig = {
|
||||||
|
User = "strata";
|
||||||
|
Group = "strata";
|
||||||
|
WorkingDirectory = "/var/lib/strata/Strata";
|
||||||
|
ExecStart = "/var/lib/strata/Strata/.venv/bin/python /var/lib/strata/Strata/serve/server.py --engine strata --config /var/lib/strata/Strata/strata-iq3_s.json --port 8080 --offline";
|
||||||
|
Restart = "on-failure";
|
||||||
|
RestartSec = 10;
|
||||||
|
LimitMEMLOCK = "infinity"; # the expert arena and KV are pinned host memory
|
||||||
|
# Hardening: image paths can name local files, so the server sees only its own state and the models
|
||||||
|
NoNewPrivileges = true;
|
||||||
|
ProtectSystem = "strict";
|
||||||
|
ProtectHome = true;
|
||||||
|
ReadWritePaths = [ "/var/lib/strata" "/models/strata" ];
|
||||||
|
PrivateTmp = true;
|
||||||
|
ProtectKernelTunables = true;
|
||||||
|
ProtectKernelModules = true;
|
||||||
|
ProtectControlGroups = true;
|
||||||
|
RestrictSUIDSGID = true;
|
||||||
|
LockPersonality = true;
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
users.users.root.openssh.authorizedKeys.keys = config.users.users.jason.openssh.authorizedKeys.keys;
|
||||||
|
security.sudo.wheelNeedsPassword = false; # key-only login; set a password with `passwd` for console use
|
||||||
|
|
||||||
|
environment.systemPackages = with pkgs; [
|
||||||
|
git vim htop btop nvtopPackages.nvidia pciutils tmux uv python3 wget curl
|
||||||
|
cudaPackages.cuda_nvcc # nvcc for building custom kernels; runtime libs come from the wheels
|
||||||
|
];
|
||||||
|
|
||||||
|
system.stateVersion = "26.05";
|
||||||
|
}
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
# gpu-unlock (github.com/Zagitta/gpu-unlock): full-rate FP16/BF16 on GA102.
|
||||||
|
# Loads from the initrd before nvidia binds; nothing is flashed, a power-off undoes it.
|
||||||
|
# Disable by removing gpu_unlock.targets from boot.kernelParams.
|
||||||
|
{ config, lib, pkgs, ... }:
|
||||||
|
|
||||||
|
let
|
||||||
|
nv = "610.43.03";
|
||||||
|
kernel = config.boot.kernelPackages.kernel;
|
||||||
|
|
||||||
|
src = pkgs.fetchFromGitHub {
|
||||||
|
owner = "Zagitta";
|
||||||
|
repo = "gpu-unlock";
|
||||||
|
rev = "af28cbbfd3d922a659227ecfb7369adfd924cf03"; # 1.0.0, 2026-10-24
|
||||||
|
hash = "sha256-6FaDuPojB707ds7gkQY/g/+m4Tnz1xHZ95UjNfFxBJc=";
|
||||||
|
};
|
||||||
|
|
||||||
|
nvSource = pkgs.fetchurl {
|
||||||
|
url = "https://download.nvidia.com/XFree86/NVIDIA-kernel-module-source/NVIDIA-kernel-module-source-${nv}.tar.xz";
|
||||||
|
hash = "sha256-fhGJI8eiPtw2EU1jJzpG4+BOmvmGlaQiA+esLf6fwdw=";
|
||||||
|
};
|
||||||
|
nvUtils = pkgs.fetchurl {
|
||||||
|
url = "https://archive.cachyos.org/archive/cachyos-v3/nvidia-utils-${nv}-1-x86_64_v3.pkg.tar.zst";
|
||||||
|
hash = "sha256-4X58ku7/oXiw/h9A6xekcJX4/Dzkp3vxFc9b1aTfhuk=";
|
||||||
|
};
|
||||||
|
|
||||||
|
# The generators check every input and output against pinned SHA-256s.
|
||||||
|
firmware = pkgs.stdenvNoCC.mkDerivation {
|
||||||
|
pname = "gpu-unlock-firmware";
|
||||||
|
version = "1.0.0";
|
||||||
|
inherit src;
|
||||||
|
nativeBuildInputs = [ pkgs.python3 pkgs.zstd pkgs.xz ];
|
||||||
|
dontConfigure = true;
|
||||||
|
buildPhase = ''
|
||||||
|
mkdir nvsrc assets
|
||||||
|
tar -xf ${nvSource} -C nvsrc
|
||||||
|
python3 firmware/generate_v67.py ${nvUtils} assets
|
||||||
|
python3 firmware/extract_boot_assets.py nvsrc/NVIDIA-kernel-module-source-${nv} assets
|
||||||
|
'';
|
||||||
|
installPhase = ''
|
||||||
|
install -Dm644 -t $out/lib/firmware/gpu-unlock/${nv} \
|
||||||
|
assets/booter_load_ga102.bin assets/gsp_ga10x.bin assets/gsp_boot_ga102.bin \
|
||||||
|
assets/gsp_boot_desc_ga102.bin assets/v67-ga102-${nv}.bin
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
module = pkgs.stdenv.mkDerivation {
|
||||||
|
pname = "gpu-unlock";
|
||||||
|
version = "1.0.0-${kernel.modDirVersion}";
|
||||||
|
inherit src;
|
||||||
|
nativeBuildInputs = kernel.moduleBuildDependencies;
|
||||||
|
hardeningDisable = [ "pic" "format" ];
|
||||||
|
buildPhase = ''
|
||||||
|
make -C ${kernel.dev}/lib/modules/${kernel.modDirVersion}/build \
|
||||||
|
M=$PWD/kernel/gpu_unlock ARCH=${pkgs.stdenv.hostPlatform.linuxArch} modules
|
||||||
|
'';
|
||||||
|
installPhase = ''
|
||||||
|
install -Dm644 kernel/gpu_unlock/gpu_unlock.ko \
|
||||||
|
$out/lib/modules/${kernel.modDirVersion}/extra/gpu_unlock.ko
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
in
|
||||||
|
{
|
||||||
|
boot.extraModulePackages = [ module ];
|
||||||
|
hardware.firmware = [ firmware ];
|
||||||
|
# Must run before nvidia binds the cards, so it lives in the initrd with its firmware.
|
||||||
|
boot.initrd.kernelModules = lib.mkBefore [ "gpu_unlock" ];
|
||||||
|
boot.initrd.extraFirmwarePaths = map (f: "gpu-unlock/${nv}/${f}") [
|
||||||
|
"booter_load_ga102.bin" "gsp_ga10x.bin" "gsp_boot_ga102.bin"
|
||||||
|
"gsp_boot_desc_ga102.bin" "v67-ga102-${nv}.bin"
|
||||||
|
];
|
||||||
|
boot.kernelParams = [ "gpu_unlock.targets=all" ];
|
||||||
|
}
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Do not modify this file! It was generated by ‘nixos-generate-config’
|
||||||
|
# and may be overwritten by future invocations. Please make changes
|
||||||
|
# to /etc/nixos/configuration.nix instead.
|
||||||
|
{ config, lib, pkgs, modulesPath, ... }:
|
||||||
|
|
||||||
|
{
|
||||||
|
imports =
|
||||||
|
[ (modulesPath + "/profiles/qemu-guest.nix")
|
||||||
|
];
|
||||||
|
|
||||||
|
boot.initrd.availableKernelModules = [ "virtio_pci" "uhci_hcd" "ehci_pci" "ahci" "virtio_scsi" "sd_mod" "sr_mod" ];
|
||||||
|
boot.initrd.kernelModules = [ ];
|
||||||
|
boot.kernelModules = [ "kvm-intel" ];
|
||||||
|
boot.extraModulePackages = [ ];
|
||||||
|
|
||||||
|
fileSystems."/" =
|
||||||
|
{ device = "/dev/disk/by-uuid/b87439f1-ecb7-4c47-89d5-f58fb75a965b";
|
||||||
|
fsType = "ext4";
|
||||||
|
};
|
||||||
|
|
||||||
|
fileSystems."/boot" =
|
||||||
|
{ device = "/dev/disk/by-uuid/8070-3FEA";
|
||||||
|
fsType = "vfat";
|
||||||
|
options = [ "fmask=0077" "dmask=0077" ];
|
||||||
|
};
|
||||||
|
|
||||||
|
fileSystems."/models" =
|
||||||
|
{ device = "models";
|
||||||
|
fsType = "virtiofs";
|
||||||
|
};
|
||||||
|
|
||||||
|
swapDevices = [ ];
|
||||||
|
|
||||||
|
nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
|
||||||
|
}
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
{
|
||||||
|
"$schema": "https://pi.dev/schemas/models.schema.json",
|
||||||
|
"providers": {
|
||||||
|
"strata": {
|
||||||
|
"baseUrl": "http://192.168.1.67:8080/v1",
|
||||||
|
"api": "openai-completions",
|
||||||
|
"apiKey": "<STRATA_API_KEY>",
|
||||||
|
"compat": {
|
||||||
|
"supportsDeveloperRole": false,
|
||||||
|
"supportsReasoningEffort": false,
|
||||||
|
"thinkingFormat": "chat-template",
|
||||||
|
"chatTemplateKwargs": {
|
||||||
|
"enable_thinking": {
|
||||||
|
"$var": "thinking.enabled"
|
||||||
|
},
|
||||||
|
"reasoning_effort": {
|
||||||
|
"$var": "thinking.effort",
|
||||||
|
"omitWhenOff": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"models": [
|
||||||
|
{
|
||||||
|
"id": "qwen3.8-flash-next-iq3_s",
|
||||||
|
"name": "Qwen3.8-Flash-Next (lovey, Strata)",
|
||||||
|
"reasoning": true,
|
||||||
|
"input": [
|
||||||
|
"text",
|
||||||
|
"image"
|
||||||
|
],
|
||||||
|
"contextWindow": 131072,
|
||||||
|
"maxTokens": 32768,
|
||||||
|
"cost": {
|
||||||
|
"input": 0,
|
||||||
|
"output": 0,
|
||||||
|
"cacheRead": 0,
|
||||||
|
"cacheWrite": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
{
|
||||||
|
"defaultProvider": "strata",
|
||||||
|
"defaultModel": "qwen3.8-flash-next-iq3_s",
|
||||||
|
"defaultThinkingLevel": "medium",
|
||||||
|
"compaction": {
|
||||||
|
"reserveTokens": 32768,
|
||||||
|
"keepRecentTokens": 20000
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
Types: deb
|
||||||
|
URIs: http://download.proxmox.com/debian/ceph-squid
|
||||||
|
Suites: trixie
|
||||||
|
Components: no-subscription
|
||||||
|
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
Types: deb
|
||||||
|
URIs: http://deb.debian.org/debian/
|
||||||
|
Suites: trixie trixie-updates
|
||||||
|
Components: main contrib non-free-firmware
|
||||||
|
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
|
||||||
|
|
||||||
|
Types: deb
|
||||||
|
URIs: http://security.debian.org/debian-security/
|
||||||
|
Suites: trixie-security
|
||||||
|
Components: main contrib non-free-firmware
|
||||||
|
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
Types: deb
|
||||||
|
URIs: http://download.proxmox.com/debian/pve
|
||||||
|
Suites: trixie
|
||||||
|
Components: pve-no-subscription
|
||||||
|
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
Types: deb
|
||||||
|
URIs: https://enterprise.proxmox.com/debian/pve
|
||||||
|
Suites: trixie
|
||||||
|
Components: pve-enterprise
|
||||||
|
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
|
||||||
|
Enabled: false
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
|
||||||
|
GRUB_CMDLINE_LINUX=""
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
# <file system> <mount point> <type> <options> <dump> <pass>
|
||||||
|
/dev/pve/root / ext4 errors=remount-ro 0 1
|
||||||
|
UUID=CE38-9BFE /boot/efi vfat defaults 0 1
|
||||||
|
/dev/pve/swap none swap sw 0 0
|
||||||
|
proc /proc proc defaults 0 0
|
||||||
|
odroid1:/gvolume0 /mnt/gluster glusterfs defaults,_netdev,backup-volfile-servers=odroid3:odroid5 0 0
|
||||||
|
/dev/nvme/models /mnt/models xfs defaults,noatime 0 2
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
blacklist nouveau
|
||||||
|
options nouveau modeset=0
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# RTX 3090 GPU + HDMI audio -> vfio-pci (matches the second 3090 too, same IDs)
|
||||||
|
options vfio-pci ids=10de:2204,10de:1aef disable_vga=1
|
||||||
|
softdep nouveau pre: vfio-pci
|
||||||
|
softdep nvidia pre: vfio-pci
|
||||||
|
softdep snd_hda_intel pre: vfio-pci
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
nct6775
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
auto lo
|
||||||
|
iface lo inet loopback
|
||||||
|
|
||||||
|
iface eno2 inet manual
|
||||||
|
|
||||||
|
auto vmbr0
|
||||||
|
iface vmbr0 inet static
|
||||||
|
address 192.168.1.53/24
|
||||||
|
gateway 192.168.1.1
|
||||||
|
bridge-ports eno2
|
||||||
|
bridge-stp off
|
||||||
|
bridge-fd 0
|
||||||
|
|
||||||
|
iface nic2 inet manual
|
||||||
|
|
||||||
|
iface nic1 inet manual
|
||||||
|
|
||||||
|
|
||||||
|
source /etc/network/interfaces.d/*
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
models
|
||||||
|
map node=lovey,path=/mnt/models
|
||||||
|
description Shared model weights
|
||||||
|
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
affinity: 0-15
|
||||||
|
agent: enabled=1
|
||||||
|
balloon: 0
|
||||||
|
bios: ovmf
|
||||||
|
boot: order=scsi0
|
||||||
|
cores: 16
|
||||||
|
cpu: host
|
||||||
|
efidisk0: nvme-thin:vm-200-disk-0,efitype=4m,pre-enrolled-keys=0,size=4M
|
||||||
|
hostpci0: 0000:01:00,pcie=1
|
||||||
|
hostpci1: 0000:02:00,pcie=1
|
||||||
|
ide2: none,media=cdrom
|
||||||
|
machine: q35
|
||||||
|
memory: 114688
|
||||||
|
meta: creation-qemu=11.0.3,ctime=1791860572
|
||||||
|
name: llm-nixos
|
||||||
|
net0: virtio=BC:24:11:DB:23:10,bridge=vmbr0,firewall=1
|
||||||
|
numa: 0
|
||||||
|
ostype: l26
|
||||||
|
scsi0: nvme-thin:vm-200-disk-1,discard=on,iothread=1,size=100G,ssd=1
|
||||||
|
scsihw: virtio-scsi-single
|
||||||
|
serial0: socket
|
||||||
|
smbios1: uuid=d317306e-ae51-4a39-ad97-c93f3ef7f2a1
|
||||||
|
sockets: 1
|
||||||
|
vga: std
|
||||||
|
virtiofs0: dirid=models,cache=auto
|
||||||
|
vmgenid: f1fe44ef-3edf-4e48-89e3-8ce1d516c1df
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
dir: local
|
||||||
|
path /var/lib/vz
|
||||||
|
content iso,backup,import,vztmpl
|
||||||
|
|
||||||
|
lvmthin: local-lvm
|
||||||
|
thinpool data
|
||||||
|
vgname pve
|
||||||
|
content rootdir,images
|
||||||
|
|
||||||
|
dir: gluster
|
||||||
|
path /mnt/gluster/proxmox
|
||||||
|
content vztmpl,backup,iso
|
||||||
|
is_mountpoint /mnt/gluster
|
||||||
|
|
||||||
|
lvmthin: nvme-thin
|
||||||
|
thinpool guests
|
||||||
|
vgname nvme
|
||||||
|
content rootdir,images
|
||||||
|
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# lockups -> panic -> dump to EFI pstore -> auto-reboot in 10s
|
||||||
|
kernel.softlockup_panic = 1
|
||||||
|
kernel.hardlockup_panic = 1
|
||||||
|
kernel.panic_on_oops = 1
|
||||||
|
kernel.panic = 10
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
[Journal]
|
||||||
|
SyncIntervalSec=15s
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Radiator fans follow CPU and GPU temps
|
||||||
|
After=network-online.target
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
ExecStartPre=/usr/sbin/modprobe nct6775
|
||||||
|
ExecStart=/usr/bin/python3 /usr/local/bin/lovey-fans.py
|
||||||
|
# Belt and braces: hand fans back to the BIOS even if the daemon was killed hard
|
||||||
|
ExecStopPost=/bin/sh -c 'for p in /sys/class/hwmon/hwmon*/pwm[2-5]_enable; do grep -q nct6798 "$(dirname $p)/name" && echo 5 > "$p"; done'
|
||||||
|
Restart=always
|
||||||
|
RestartSec=5
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
Executable
+17
@@ -0,0 +1,17 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# stability soak: PVE CLI calls + zstd round-trips + all-core load
|
||||||
|
log=/root/stab.log; : > $log
|
||||||
|
stress-ng --cpu 24 --cpu-method matrixprod --vm 4 --vm-bytes 60% --vm-method all --verify -t 900s --metrics-brief >> /root/stress.log 2>&1 &
|
||||||
|
head -c 2G /dev/urandom > /root/rand.bin; ref=$(sha256sum < /root/rand.bin)
|
||||||
|
end=$((SECONDS+900)); i=0; bad=0
|
||||||
|
while [ $SECONDS -lt $end ]; do
|
||||||
|
i=$((i+1))
|
||||||
|
pvesm status >/dev/null 2>&1 || { bad=$((bad+1)); echo "$(date +%T) pvesm fail #$i" >> $log; }
|
||||||
|
qm config 200 >/dev/null 2>&1 || { bad=$((bad+1)); echo "$(date +%T) qm fail #$i" >> $log; }
|
||||||
|
if [ $((i % 10)) -eq 0 ]; then
|
||||||
|
s=$(zstd -q -T8 -c /root/rand.bin | zstd -q -d -c | sha256sum)
|
||||||
|
[ "$s" = "$ref" ] || { bad=$((bad+1)); echo "$(date +%T) zstd MISMATCH #$i" >> $log; }
|
||||||
|
fi
|
||||||
|
echo "$(date +%T) iter=$i bad=$bad" > /root/stab.status
|
||||||
|
done
|
||||||
|
wait; echo "DONE iter=$i bad=$bad" >> $log; rm -f /root/rand.bin
|
||||||
Executable
+133
@@ -0,0 +1,133 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Radiator fan control for lovey: fans follow max(CPU curve, GPU curve).
|
||||||
|
|
||||||
|
CPU temp comes from coretemp. The GPUs are passed through to VM 200, so the
|
||||||
|
guest pushes its hottest GPU temp here over UDP every 2 s (gpu-temp-push in
|
||||||
|
the guest's NixOS config). On exit the fans go back to BIOS control.
|
||||||
|
"""
|
||||||
|
import glob, os, signal, socket, sys, time
|
||||||
|
|
||||||
|
FANS = [2, 3, 4, 5] # nct6798 headers with radiator fans; fan6 (AIO_PUMP) and fan7 (pump) stay on BIOS
|
||||||
|
LISTEN = ("192.168.1.53", 7777)
|
||||||
|
GPU_SOURCE = "192.168.1.67" # llm-nixos; only accept temps from it
|
||||||
|
VMID = 200
|
||||||
|
INTERVAL = 2.0
|
||||||
|
STALE = 15.0 # seconds without a GPU packet while the VM runs -> failsafe
|
||||||
|
FAILSAFE = 85 # % when GPU temp is unknown but the VM is up
|
||||||
|
MIN_PCT = 30
|
||||||
|
RAMP_DOWN = 2 # max % drop per interval (100 -> 30 in ~70 s)
|
||||||
|
ALPHA = 0.3 # temperature smoothing per interval (~6 s time constant)
|
||||||
|
|
||||||
|
# (temp C, fan %) points, linearly interpolated, flat outside the range
|
||||||
|
CPU_CURVE = [(50, 30), (70, 50), (85, 100)]
|
||||||
|
GPU_CURVE = [(45, 30), (60, 40), (75, 100)]
|
||||||
|
|
||||||
|
|
||||||
|
def hwmon(name):
|
||||||
|
for d in glob.glob("/sys/class/hwmon/hwmon*"):
|
||||||
|
with open(f"{d}/name") as f:
|
||||||
|
if f.read().strip() == name:
|
||||||
|
return d
|
||||||
|
sys.exit(f"hwmon {name} not found")
|
||||||
|
|
||||||
|
|
||||||
|
def read_int(path):
|
||||||
|
with open(path) as f:
|
||||||
|
return int(f.read())
|
||||||
|
|
||||||
|
|
||||||
|
def write(path, value):
|
||||||
|
with open(path, "w") as f:
|
||||||
|
f.write(str(value))
|
||||||
|
|
||||||
|
|
||||||
|
def curve(points, t):
|
||||||
|
if t <= points[0][0]:
|
||||||
|
return points[0][1]
|
||||||
|
for (t0, p0), (t1, p1) in zip(points, points[1:]):
|
||||||
|
if t <= t1:
|
||||||
|
return p0 + (p1 - p0) * (t - t0) / (t1 - t0)
|
||||||
|
return points[-1][1]
|
||||||
|
|
||||||
|
|
||||||
|
def vm_running():
|
||||||
|
try:
|
||||||
|
with open(f"/var/run/qemu-server/{VMID}.pid") as f:
|
||||||
|
os.kill(int(f.read()), 0)
|
||||||
|
return True
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
NCT = hwmon("nct6798")
|
||||||
|
CORETEMP = hwmon("coretemp")
|
||||||
|
CPU_TEMP = next(p.replace("_label", "_input") for p in glob.glob(f"{CORETEMP}/temp*_label")
|
||||||
|
if open(p).read().startswith("Package"))
|
||||||
|
|
||||||
|
|
||||||
|
def restore(*_):
|
||||||
|
for n in FANS:
|
||||||
|
try:
|
||||||
|
write(f"{NCT}/pwm{n}_enable", 5) # 5 = SmartFan IV, the BIOS curve
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
sys.exit(0)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
signal.signal(signal.SIGTERM, restore)
|
||||||
|
signal.signal(signal.SIGINT, restore)
|
||||||
|
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
|
||||||
|
sock.bind(LISTEN)
|
||||||
|
sock.setblocking(False)
|
||||||
|
for n in FANS:
|
||||||
|
write(f"{NCT}/pwm{n}_enable", 1)
|
||||||
|
|
||||||
|
cpu_s = gpu_s = None
|
||||||
|
gpu_at = 0.0
|
||||||
|
pct = 100.0
|
||||||
|
last_log = 0.0
|
||||||
|
try:
|
||||||
|
while True:
|
||||||
|
while True: # drain queued GPU packets, keep the newest
|
||||||
|
try:
|
||||||
|
data, (addr, _) = sock.recvfrom(64)
|
||||||
|
except BlockingIOError:
|
||||||
|
break
|
||||||
|
if addr == GPU_SOURCE:
|
||||||
|
try:
|
||||||
|
g = float(data.decode().strip())
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
gpu_s = g if gpu_s is None else gpu_s + ALPHA * (g - gpu_s)
|
||||||
|
gpu_at = time.monotonic()
|
||||||
|
|
||||||
|
c = read_int(CPU_TEMP) / 1000
|
||||||
|
cpu_s = c if cpu_s is None else cpu_s + ALPHA * (c - cpu_s)
|
||||||
|
|
||||||
|
want = curve(CPU_CURVE, cpu_s)
|
||||||
|
if time.monotonic() - gpu_at < STALE:
|
||||||
|
want = max(want, curve(GPU_CURVE, gpu_s))
|
||||||
|
gpu_state = f"{gpu_s:.0f}C"
|
||||||
|
elif vm_running():
|
||||||
|
want = max(want, FAILSAFE)
|
||||||
|
gpu_state = "stale"
|
||||||
|
else:
|
||||||
|
gpu_state = "vm-off"
|
||||||
|
want = max(want, MIN_PCT)
|
||||||
|
pct = want if want > pct else max(want, pct - RAMP_DOWN)
|
||||||
|
|
||||||
|
for n in FANS:
|
||||||
|
write(f"{NCT}/pwm{n}", round(pct * 255 / 100))
|
||||||
|
|
||||||
|
if time.monotonic() - last_log >= 60:
|
||||||
|
rpm = " ".join(str(read_int(f"{NCT}/fan{n}_input")) for n in FANS)
|
||||||
|
print(f"cpu={cpu_s:.0f}C gpu={gpu_state} fans={pct:.0f}% rpm={rpm}", flush=True)
|
||||||
|
last_log = time.monotonic()
|
||||||
|
time.sleep(INTERVAL)
|
||||||
|
finally:
|
||||||
|
restore()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
{
|
||||||
|
"exe": "/var/lib/strata/Strata/engine/strata",
|
||||||
|
"args": [
|
||||||
|
"--pack",
|
||||||
|
"/models/strata/packs/iq3_s",
|
||||||
|
"--native",
|
||||||
|
"/models/strata/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",
|
||||||
|
"--ple-gguf",
|
||||||
|
"/models/strata/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf",
|
||||||
|
"--expert-profile",
|
||||||
|
"/var/lib/strata/Strata/data/expert-profile.bin",
|
||||||
|
"--expert-cache",
|
||||||
|
"auto",
|
||||||
|
"--prefill",
|
||||||
|
"auto",
|
||||||
|
"--spec",
|
||||||
|
"4",
|
||||||
|
"--spec-min-p",
|
||||||
|
"0.5",
|
||||||
|
"--mtp",
|
||||||
|
"/models/strata/mtp/rt",
|
||||||
|
"--max-context",
|
||||||
|
"131072",
|
||||||
|
"--kv",
|
||||||
|
"int8",
|
||||||
|
"--ple-io",
|
||||||
|
"ram",
|
||||||
|
"--kv-resident",
|
||||||
|
"32768",
|
||||||
|
"--vision",
|
||||||
|
"--vram-reserve-mib",
|
||||||
|
"700"
|
||||||
|
],
|
||||||
|
"cwd": "/var/lib/strata/Strata",
|
||||||
|
"tokenizer": "/models/strata/packs/iq3_s/tokenizer",
|
||||||
|
"model_name": "qwen3.8-flash-next-iq3_s",
|
||||||
|
"log": "/var/lib/strata/Strata/strata-iq3_s.log",
|
||||||
|
"lib_dirs": [
|
||||||
|
"/nix/store/7mi3cxyj9rgx76yxr6jg4z8i3cvb3dmz-cuda-merged-13.0/bin"
|
||||||
|
],
|
||||||
|
"port": 8080,
|
||||||
|
"gpu": [
|
||||||
|
0,
|
||||||
|
1
|
||||||
|
],
|
||||||
|
"gpus_asked": true,
|
||||||
|
"layer_split": "auto",
|
||||||
|
"host": "0.0.0.0",
|
||||||
|
"api_key": "<API_KEY: see /var/lib/strata/api-key>",
|
||||||
|
"open_browser": false,
|
||||||
|
"vision": {
|
||||||
|
"exe": "/var/lib/strata/Strata/engine/strata-vision",
|
||||||
|
"mmproj": "/models/strata/models/mmproj-Qwen3.8-Flash-Next-BF16.gguf",
|
||||||
|
"model": "/models/strata/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",
|
||||||
|
"gpu": true,
|
||||||
|
"max_tokens": 1024
|
||||||
|
},
|
||||||
|
"args_written": [
|
||||||
|
"--pack",
|
||||||
|
"/models/strata/packs/iq3_s",
|
||||||
|
"--native",
|
||||||
|
"/models/strata/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",
|
||||||
|
"--ple-gguf",
|
||||||
|
"/models/strata/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf",
|
||||||
|
"--expert-profile",
|
||||||
|
"/var/lib/strata/Strata/data/expert-profile.bin",
|
||||||
|
"--expert-cache",
|
||||||
|
"auto",
|
||||||
|
"--prefill",
|
||||||
|
"auto",
|
||||||
|
"--spec",
|
||||||
|
"4",
|
||||||
|
"--spec-min-p",
|
||||||
|
"0.5",
|
||||||
|
"--mtp",
|
||||||
|
"/models/strata/mtp/rt",
|
||||||
|
"--max-context",
|
||||||
|
"131072",
|
||||||
|
"--kv",
|
||||||
|
"int8",
|
||||||
|
"--ple-io",
|
||||||
|
"ram",
|
||||||
|
"--kv-resident",
|
||||||
|
"32768",
|
||||||
|
"--vision",
|
||||||
|
"--vram-reserve-mib",
|
||||||
|
"700"
|
||||||
|
]
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user