Jason WittyandClaude Opus 5.5
cff8cf640e
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
81 lines
4.8 KiB
Markdown
81 lines
4.8 KiB
Markdown
# lovey
|
||
|
||
lovey is my inference server running 2× RTX 3090s. This repo stores and documents its configuration,
|
||
models and benchmarks, and holds everything needed to rebuild the current setup from bare metal.
|
||
|
||
**Current state (2026-10-26):** Proxmox host → one NixOS VM with both GPUs passed through → Strata
|
||
serving Qwen3.8-Flash-Next IQ3_S on an OpenAI/Anthropic-compatible API, used from
|
||
[pi](https://pi.dev) on my workstation.
|
||
|
||
```
|
||
cosmic (workstation) lovey 192.168.1.53 (Proxmox VE 9.2)
|
||
┌──────────────────┐ LAN ┌──────────────────────────────────────────────────────┐
|
||
│ pi coding agent │──────────────▶│ VM 200 llm-nixos 192.168.1.67 (NixOS 26.05) │
|
||
│ ~/.pi/agent │ :8080 + key │ strata.service ── Qwen3.8-Flash-Next IQ3_S │
|
||
└──────────────────┘ │ 2× RTX 3090 (VFIO), gpu-unlock, 275 W cap │
|
||
│ /models ◀── virtiofs ── /mnt/models (NVMe XFS) │
|
||
│ gpu-temp-push ──UDP :7777──▶ lovey-fans.service │
|
||
└──────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Hardware
|
||
|
||
| Part | Detail |
|
||
|---|---|
|
||
| Board | ASUS ROG Maximus Z790 Hero |
|
||
| CPU | Intel i9-14900KS, microcode 0x133, 200 W PL1 / 253 W PL2, Intel default profile |
|
||
| RAM | 4× 32 GB = 128 GB DDR5, **run at 4800 MT/s JEDEC** (unstable at 5200+ with 4 DIMMs) |
|
||
| GPUs | 2× Zotac RTX 3090 24 GB at `01:00` and `02:00`, PCIe 4.0 x8 each, no NVLink |
|
||
| Boot disk | 250 GB SATA SSD (`sda`): Proxmox only |
|
||
| Data disk | MSI M470 2 TB NVMe, 4K LBA: model store + VM disks |
|
||
| Cooling | Liquid: radiator fans on nct6798 fan2–5 (driven by `lovey-fans`), pump on fan7, no coolant sensor |
|
||
| NIC | Onboard Intel 2.5G `eno2`, bridged as `vmbr0` |
|
||
|
||
## Software stack
|
||
|
||
| Layer | What | Docs |
|
||
|---|---|---|
|
||
| Host | Proxmox VE 9.2.21, kernel 7.0.14-pve, pve-no-subscription repos | [docs/proxmox-host.md](docs/proxmox-host.md) |
|
||
| VM 200 | `llm-nixos`: NixOS 26.05, kernel 6.18, NVIDIA open 595.71, CUDA 13.2 | [docs/nixos-guest.md](docs/nixos-guest.md) |
|
||
| GPU tuning | 275 W power cap + [gpu-unlock](https://github.com/Zagitta/gpu-unlock) (full-rate FP16/BF16) | [docs/nixos-guest.md](docs/nixos-guest.md#gpu-tuning) |
|
||
| Inference | [Strata](https://github.com/Niko1221/Strata) @ `4b9e05a2`, Qwen3.8-Flash-Next IQ3_S | [docs/strata.md](docs/strata.md) |
|
||
| Client | pi coding agent 1.4.0 | [docs/pi-client.md](docs/pi-client.md) |
|
||
| Fans | `lovey-fans.py` on the host, fed GPU temps by the guest | [docs/proxmox-host.md](docs/proxmox-host.md#fan-control) |
|
||
|
||
Current numbers (standard benchmark, idle): prompt ~1,750 tok/s at 4K and ~2,900 tok/s at 16K, decode
|
||
~110–131 tok/s, model load ~2 min. See [benchmarks/standard](benchmarks/standard/README.md).
|
||
|
||
## Repo layout
|
||
|
||
```
|
||
nixos/ /etc/nixos on llm-nixos (configuration.nix, gpu-unlock.nix, hardware-configuration.nix)
|
||
proxmox/ files from lovey, at their real paths (etc/..., usr/local/bin/..., root/...)
|
||
strata/ strata-iq3_s.json, the engine config (API key removed)
|
||
pi/ ~/.pi/agent/models.json, settings.json and extensions/ for the client (API key removed)
|
||
docs/ per-layer notes, rebuild steps and the change log
|
||
benchmarks/ live metric snapshots (passive) and later benchmark runs
|
||
```
|
||
|
||
**Secrets are not in this repo** (it is public). The Strata API key lives only in
|
||
`/var/lib/strata/api-key` and `strata-iq3_s.json` on the guest; placeholders are marked `<...>`.
|
||
|
||
## Rebuild order
|
||
|
||
1. BIOS: memory at 4800 JEDEC, CPU 200/253 W, VT-d on. See [proxmox-host.md](docs/proxmox-host.md#bios).
|
||
2. Proxmox host: repos, NVMe layout, VFIO, Gluster, fans. [proxmox-host.md](docs/proxmox-host.md#rebuild)
|
||
3. VM 200 from `proxmox/etc/pve/qemu-server/200.conf`, then NixOS from `nixos/`. [nixos-guest.md](docs/nixos-guest.md#rebuild)
|
||
4. Strata source build + model download. [strata.md](docs/strata.md#rebuild)
|
||
5. pi on the workstation. [pi-client.md](docs/pi-client.md)
|
||
|
||
## History
|
||
|
||
Everything that went wrong and what fixed it is in [docs/changelog.md](docs/changelog.md): the Z690
|
||
memory-channel fault, the 4-DIMM instability on the Z790, and the pi auto-compaction loop.
|
||
|
||
## Next up
|
||
|
||
- Strata tuning: the three documented flags gave nothing ([results](benchmarks/standard/README.md)).
|
||
Next levers: huge pages for the expert arena, a higher power cap for prompt speed.
|
||
- Try other quants.
|
||
- Planned services as OCI LXCs on the host: SearXNG + degoog, Qdrant, Open WebUI, Tika, Whisper.
|