Jason WittyandClaude Opus 5.5
cff8cf640e
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
70 lines
3.9 KiB
Markdown
70 lines
3.9 KiB
Markdown
# NixOS guest (VM 200 `llm-nixos`, 192.168.1.67)
|
||
|
||
NixOS 26.05, systemd-boot, kernel 6.18, NVIDIA open driver 595.71 (CUDA 13.2). The complete config
|
||
is in [`nixos/`](../nixos) and is the source of truth: `/etc/nixos` on the guest should match it.
|
||
|
||
## What the config sets up
|
||
|
||
- **Network:** static 192.168.1.67/24 on `ens18`, gateway .1, DNS 1.1.1.1/8.8.8.8 (NetworkManager
|
||
removed). Static because `lovey-fans` only trusts GPU temps from this address. Firewall on: TCP 22
|
||
(SSH) and 8080 (Strata).
|
||
- **Users:** `jason` and root are SSH-key only; sudo needs no password.
|
||
- **NVIDIA:** open kernel module, headless, `nvidia-persistenced`. `nvidia-container-toolkit` +
|
||
podman (`--device nvidia.com/gpu=all`).
|
||
- **CUDA tooling:** `cuda_nvcc` and the cuda-maintainers cachix substituter, so nothing CUDA-heavy
|
||
compiles locally. `programs.nix-ld` with `config.hardware.nvidia.package` in its libraries:
|
||
without that, pip/uv torch and vLLM can't find `libcuda` and `torch.cuda.is_available()` is False.
|
||
- **Memory:** memlock unlimited (PAM and systemd default), needed for pinned host memory.
|
||
- **/models:** virtiofs from lovey's `/mnt/models`, `nofail`.
|
||
- **Services:** `nvidia-power-limit`, `gpu-temp-push`, `strata` (see [strata.md](strata.md)).
|
||
|
||
## GPU tuning
|
||
|
||
- **Power cap:** `nvidia-power-limit` runs `nvidia-smi -pl 250` after persistenced on every boot
|
||
(stock is 350 W).
|
||
- **gpu-unlock** ([`nixos/gpu-unlock.nix`](../nixos/gpu-unlock.nix)): the
|
||
[Zagitta/gpu-unlock](https://github.com/Zagitta/gpu-unlock) kernel driver, pinned at `af28cbb`,
|
||
enables full-rate FP16/BF16/TF32 tensor math on GA102. It runs inside the guest and flashes nothing.
|
||
The Nix file builds the module for the guest kernel and generates its firmware from pinned NVIDIA
|
||
610.43.03 inputs (all SHA-256 checked), loads `gpu_unlock` from the initrd before `nvidia`, and sets
|
||
`gpu_unlock.targets=all`. To disable it, drop that kernel parameter.
|
||
- Check: `journalctl -k -b | grep -E "gpu_unlock|entry:"`. `unlocked` appears only on the first boot
|
||
after lovey powers off. After a VM reboot it logs `unlock failed (-114)` (-EALREADY): the cards
|
||
are still unlocked, which is harmless.
|
||
- The unlock survives VM reboots and host bus resets; only a host power-off clears it.
|
||
|
||
Dense 8192³ GEMM, torch 2.14 / cu130:
|
||
|
||
| Power limit | FP16 | BF16 |
|
||
|---|---|---|
|
||
| 250 W | 77–81 TFLOPS | 80–86 TFLOPS |
|
||
| 350 W | 112–114 TFLOPS | 115–116 TFLOPS |
|
||
|
||
A stock 3090 peaks at about 71 TFLOPS. The 250 W cap costs ~30% of matmul (prefill) throughput;
|
||
decode is bandwidth-bound and barely changes.
|
||
|
||
The cap is **275 W** since 2026-10-27. In Strata's benchmark, 275 W vs 250 W gave +4.6% prompt tok/s at
|
||
16K, unchanged decode, and +2–4°C (see
|
||
[benchmarks/standard](../benchmarks/standard/README.md#power-cap-250-w-vs-275-w)). Strata's real gain is far smaller
|
||
than the GEMM numbers above suggest, because much of each request isn't matmul-bound.
|
||
|
||
## Temps
|
||
|
||
4-minute dual-GPU BF16 burn at 250 W: GPU0 34→45°C, GPU1 44→55°C (still creeping), fans ~36%.
|
||
GPU1 runs ~10°C warmer than GPU0 (loop order or block contact). Plenty of headroom.
|
||
|
||
## Rebuild
|
||
|
||
1. Create VM 200 ([proxmox-host.md](proxmox-host.md#vm-200-llm-nixos)) and boot the NixOS 26.05
|
||
minimal ISO.
|
||
2. Partition: 1 GB ESP (vfat) + ext4 root on the 100 GB disk; mount at `/mnt` and `/mnt/boot`.
|
||
3. `nixos-generate-config --root /mnt`, then copy `configuration.nix` and `gpu-unlock.nix` from this
|
||
repo. Keep the **generated** `hardware-configuration.nix` (its disk UUIDs are new) but add the
|
||
`/models` virtiofs entry from the copy here.
|
||
4. `nixos-install`, reboot, detach the ISO.
|
||
5. Check: `nvidia-smi` shows both 3090s at 275 W, `ls /models`, `systemctl status gpu-temp-push`
|
||
and `journalctl -u lovey-fans` on the host shows a GPU temperature.
|
||
6. Install Strata: [strata.md](strata.md#rebuild).
|
||
|
||
Applying a config change: edit here, copy to `/etc/nixos/`, `sudo nixos-rebuild switch`, commit.
|