lovey/docs/proxmox-host.md

Proxmox host (lovey, 192.168.1.53)

Proxmox VE 9.2.21 on Debian trixie, kernel 7.0.14-pve. Web UI at https://192.168.1.53:8006. All files referenced here are in proxmox/ at their real paths.

BIOS

• Memory: 4800 MT/s JEDEC, 1.1 V. With 4 DIMMs, 5400 bootlooped and 5200 corrupted memory (see changelog). 5000 is untested; soak with systemd-run --unit=stab /root/stab.sh before trusting it.
• CPU: Intel default profile, PL1 200 W / PL2 253 W. Set it in the BIOS; a RAPL limit set from Linux does not survive a reboot.
• VT-d on (needed for passthrough).

Storage

Device	Use
sda 250 GB	Proxmox only: VG pve (root, swap, plus an unused data thin pool that could be removed to grow root)
NVMe 2 TB	One GPT partition, VG nvme
↳ LV models 900 GB XFS	/mnt/models (fstab, noatime), shared into VMs as virtiofs mapping models
↳ thin pool guests 700 GB	Storage nvme-thin for VM disks
↳ ~262 GB free	Left unallocated on purpose: neither XFS nor a thin pool can shrink
Gluster gvolume0	odroid1:/gvolume0 at /mnt/gluster, storage gluster (dir, is_mountpoint) for ISOs and backups. PVE 9 dropped the native Gluster plugin, so it is a plain fuse mount

Files: etc/fstab, etc/pve/storage.cfg, etc/pve/mapping/directory.cfg.

GPU passthrough (VFIO)

• GRUB: intel_iommu=on iommu=pt (etc/default/grub.cmdline).
• etc/modprobe.d/vfio.conf binds 10de:2204 (3090) and 10de:1aef (its HDMI audio) to vfio-pci, which covers both cards. nouveau is blacklisted (blacklist-nouveau.conf).
• The cards are alone in IOMMU groups 21 (01:00) and 22 (02:00) on the Z790 Hero.
• QEMU logs "PCI peer-to-peer transactions on BARs are not supported". That is expected: GPU-to-GPU traffic goes through host memory and NCCL falls back to SHM.

VM 200 llm-nixos

etc/pve/qemu-server/200.conf:

• q35 / OVMF, cpu: host, 16 vCPU pinned to threads 0–15 (the P-cores; 16–31 are E-cores)
• 112 GB RAM, balloon 0. Passthrough pins all guest RAM, so leave ~10 GB for the host and LXCs
• hostpci0: 01:00, hostpci1: 02:00 (both pcie=1)
• 100 GB disk on nvme-thin, virtiofs models, qemu guest agent

Network

etc/network/interfaces: vmbr0 on eno2, static 192.168.1.53/24, gateway .1.

Gotcha: the Proxmox installer pins NIC names to MACs in /usr/local/lib/systemd/network/50-pmx-nic*.link. The three files still there match the old Z690's MACs and are harmless (not copied here). On a board swap, either re-pin before swapping or fix vmbr0 from the local console.

Fan control

The GPUs belong to the VM, so the BIOS can't see their temperature. lovey-fans.service runs usr/local/bin/lovey-fans.py, which sets fan2–5 to max(CPU curve, GPU curve) every 2 s.

Header	What	Control
fan2–fan5	Radiator fans, ~1700 rpm max	lovey-fans
fan6	Fan-like, BIOS-pinned at 100%, probably the AIO_PUMP header	BIOS
fan7	Pump, ~3180 rpm, ignores PWM	BIOS

• Curves: CPU 50°C→30%, 70°C→50%, 85°C→100%; GPU 45°C→30%, 60°C→40%, 75°C→100%. EMA-smoothed, instant ramp-up, at most 2% down per tick.
• The guest's gpu-temp-push sends its hottest GPU temp as UDP to 192.168.1.53:7777. Only packets from 192.168.1.67 count: if the guest IP changes, update GPU_SOURCE, otherwise the fans sit at 85%.
• Fail-safes (tested): feed stale >15 s while VM 200 runs → 85%; VM off → GPUs ignored; daemon stops or dies → pwm2–5 return to BIOS SmartFan (pwm_enable=5, also in ExecStopPost).
• The nct6775 driver loads from etc/modules-load.d/nct6775.conf. Logs: journalctl -u lovey-fans (one line a minute).

Crash capture

Left on from the Z690 debugging, harmless: etc/sysctl.d/90-crash-capture.conf (lockup → panic → reboot after 10 s, dump to EFI pstore) and etc/systemd/journald.conf.d/90-sync.conf (SyncIntervalSec=15s).

Rebuild

1. Install Proxmox VE 9 on sda.
2. Repos: copy etc/apt/sources.list.d/* (enterprise disabled, pve-no-subscription on, Ceph over http because https failed the cert check), then apt update && apt full-upgrade.
3. NVMe: reformat to 4K LBA (nvme format --lbaf=<4K index>), one GPT partition, then

   pvcreate /dev/nvme0n1p1 && vgcreate nvme /dev/nvme0n1p1
   lvcreate -L 900G -n models nvme && mkfs.xfs /dev/nvme/models
   lvcreate -L 700G --thinpool guests nvme
   

   Copy etc/fstab lines, etc/pve/storage.cfg and etc/pve/mapping/directory.cfg.
4. VFIO: GRUB cmdline from etc/default/grub.cmdline, update-grub, copy etc/modprobe.d/*, update-initramfs -u -k all, reboot, check lspci -nnk -s 01:00 shows vfio-pci.
5. Gluster: apt install glusterfs-client, fstab line, mkdir -p /mnt/gluster/proxmox.
6. Fans: copy usr/local/bin/lovey-fans.py, etc/systemd/system/lovey-fans.service, etc/modules-load.d/nct6775.conf, then systemctl enable --now lovey-fans.
7. Create VM 200 from etc/pve/qemu-server/200.conf (new disks on nvme-thin) and install NixOS: nixos-guest.md.