homelab-k3s

k3s deployments and change management for the homelab Raspberry Pi cluster (rpi-master + rpi-worker-1/2/3, k3s v1.30.3, flannel/traefik/klipper-lb defaults).

Manifests were reverse-engineered from the live cluster on 2026-07-26 and cleaned of runtime fields. Except where a file header says DIVERGENCE, they are faithful to what was running.

Layout

Each app under apps/ is a self-contained kustomization (namespace + PV/PVC + deployment + service + ingress):

App	Namespace	Hostname(s)
vaultwarden	bitwarden	pm.wittyoneoff.com
searxng	default	searxng.wittyoneoff.com, origin-searxng.wittyoneoff.com
home-assistant	home-assistant	ha.wittyoneoff.com, ha-origin.wittyoneoff.com
nginx	nginx	www.wittyoneoff.com, www.ramonaajj.com
pihole (+unbound)	pihole	admin on :8000 via klipper-lb, DNS on :53
unified-streaming	unified-streaming	unified.wittyoneoff.com

Not in this repo: socktop (own repo + pipeline), rancher/monitoring/logging (rancher-managed), traefik (k3s packaged component).

Applying

kubectl apply -k .                      # everything
kubectl apply -k apps/home-assistant   # one app

One-time secret bootstrap (per cluster)

Three apps read secrets that are NOT in git. Each app dir has a secret.example.yaml; copy to secret.yaml (gitignored), fill in, apply.

Secret	Namespace	Used for
pihole-admin	pihole	pi-hole admin WEBPASSWORD
usp-license	unified-streaming	USP_LICENSE_KEY
searxng-1723974683-config	default	searxng settings.yml

One-time migration (pihole + unified-streaming): the live cluster (as of 2026-07-26) has WEBPASSWORD / USP_LICENSE_KEY as inline plaintext values. kubectl apply (client- or server-side) cannot patch an env var from value to valueFrom — the API rejects the merged object. Validated procedure:

# 1. create the two secrets (see secret.example.yaml in each app dir)
# 2. one-time replace (restarts the pods):
kubectl replace -f apps/pihole/deployment-pihole.yaml
kubectl replace -f apps/unified-streaming/deployment.yaml

After that, kubectl apply -k . works cleanly for everything (verified with --dry-run=server against the live cluster, 2026-07-26 — zero immutable-field conflicts).

Cluster assumptions (state that lives outside these manifests)

• Node labels — scheduling relies on them: workers carry cpu=arm (all nodes do), the control-plane node carries controller=true (most apps use controller NotIn (true) to stay off the master), pis carry model=raspi (pihole/unbound prefer it). Re-label with: kubectl label node <n> cpu=arm model=raspi / kubectl label node rpi-master controller=true.
• GlusterFS — every PV is hostPath under /mnt/gvolume0/…, which is the gluster volume mounted identically on all nodes. The mount must exist before pods schedule; PVs bind by explicit volumeName + storageClassName: "".
• Ingress/LB — traefik (k3s default) terminates all HTTP on 80/443 via svclb on every node. Because of that, no other LoadBalancer service can claim host port 80 (that's why unifiedstreaming-svc is ClusterIP).
• Home Assistant config — lives on the PV, not in git. http.trusted_proxies must include 10.42.0.0/16 (the cluster pod CIDR) or HA 400s everything the moment traefik reschedules to another node.

Known quirks (faithful to live state, fix at leisure)

• Most images are :latest/:stable — deploys are not reproducible until pinned. vaultwarden is pinned (1.35.4).
• searxng keeps its helm-generated name suffix (searxng-1723974683) because deployment selectors are immutable; its helm release record still exists in the cluster — don't run helm upgrade/uninstall on it, this repo is the source of truth now.
• searxng's TZ=America/Los Angeles (missing underscore) is invalid tzdata — container falls back to UTC.
• vaultwarden and home-assistant carry helm-chart-era label/name shapes but their release records are gone; they are plain manifests now.
• pihole runs privileged: true with PIHOLE_UID=0.

History

• 2026-07-25: cluster outage (rpi-master undervoltage → CNI cache corruption). Recovery runbook in the Notes vault: rpi-master_undervoltage_CNI_corruption_incident-2026-07-25.md.
• 2026-07-26: initial import of manifests from live cluster; HA ingress defaultBackend fixed; unifiedstreaming-svc LoadBalancer→ClusterIP; pihole/USP secrets moved to secretKeyRef (pending first apply).