homelab-k3s

k3s deployments and change management for the homelab Raspberry Pi cluster (rpi-master + rpi-worker-1/2/3, k3s v1.30.3, flannel/traefik/klipper-lb defaults).

Manifests were reverse-engineered from the live cluster on 2026-07-26 and cleaned of runtime fields. Except where a file header says DIVERGENCE, they are faithful to what was running.

Layout

Each app under apps/ is a self-contained kustomization (namespace + PV/PVC + deployment + service + ingress):

App	Namespace	Hostname(s)
vaultwarden	bitwarden	pm.wittyoneoff.com
searxng	default	searxng.wittyoneoff.com, origin-searxng.wittyoneoff.com
home-assistant	home-assistant	ha.wittyoneoff.com, ha-origin.wittyoneoff.com
nginx	nginx	www.wittyoneoff.com, www.ramonaajj.com
pihole	pihole	admin on :8000 via klipper-lb, DNS on :53
unbound	pihole	pi-hole's upstream recursive resolver (ClusterIP only)
unified-streaming	unified-streaming	unified.wittyoneoff.com

Not in this repo: socktop (own repo + pipeline), rancher/monitoring/logging (rancher-managed), traefik (k3s packaged component).

Applying

kubectl apply -k .                      # everything
kubectl apply -k apps/home-assistant   # one app

One-time secret bootstrap (per cluster)

Three apps read secrets that are NOT in git. Each app dir has a secret.example.yaml; copy to secret.yaml (gitignored), fill in, apply.

Secret	Namespace	Used for
pihole-admin	pihole	pi-hole admin WEBPASSWORD
usp-license	unified-streaming	USP_LICENSE_KEY
searxng-1723974683-config	default	searxng settings.yml

One-time migration (pihole + unified-streaming) — DONE 2026-07-26: secrets pihole-admin and usp-license created in-cluster and both deployments switched to secretKeyRef via kubectl replace (a plain kubectl apply cannot patch an env var from value to valueFrom). kubectl apply -k . now converges cleanly (verified --dry-run=server).

CI/CD (Gitea Actions)

.gitea/workflows/deploy.yaml, modeled on socktop-webterm's pipeline:

• PRs → kubectl apply -k . --dry-run=server (validation only)
• push to main → kubectl apply -k . + kubectl rollout status on all seven deployments

Repo Actions secret required: KUBECONFIG — base64-encoded kubeconfig for the gitea-deployer ServiceAccount (same identity socktop-webterm deploys with; regenerate anytime from default/gitea-deployer-token). Its RBAC is rbac/gitea-deployer.yaml — admin-applied once, deliberately outside the root kustomization so the pipeline token cannot escalate itself; it has no access to secrets and no delete verbs.

Cluster assumptions (state that lives outside these manifests)

• Node labels — scheduling relies on them: workers carry cpu=arm (all nodes do), the control-plane node carries controller=true (most apps use controller NotIn (true) to stay off the master), pis carry model=raspi (pihole/unbound prefer it). Re-label with: kubectl label node <n> cpu=arm model=raspi / kubectl label node rpi-master controller=true.
• GlusterFS — every PV is hostPath under /mnt/gvolume0/…, which is the gluster volume mounted identically on all nodes. The mount must exist before pods schedule; PVs bind by explicit volumeName + storageClassName: "".
• Ingress/LB — traefik (k3s default) terminates all HTTP on 80/443 via svclb on every node. Because of that, no other LoadBalancer service can claim host port 80 (that's why unifiedstreaming-svc is ClusterIP).
• Home Assistant config — lives on the PV, not in git. http.trusted_proxies must include 10.42.0.0/16 (the cluster pod CIDR) or HA 400s everything the moment traefik reschedules to another node.

Known quirks (faithful to live state, fix at leisure)

• Most images are :latest/:stable — deploys are not reproducible until pinned. vaultwarden is pinned (1.35.4); pihole is pinned by digest (Core v6.2.2) after :latest silently jumped v5→v6 on a fresh node pull (2026-07-26) and broke DNS for cross-node sources (v6 defaults to listeningMode=LOCAL) — fixed via FTLCONF_dns_listeningMode=ALL.
• searxng keeps its helm-generated name suffix (searxng-1723974683) because deployment selectors are immutable; its helm release record still exists in the cluster — don't run helm upgrade/uninstall on it, this repo is the source of truth now.
• searxng's TZ=America/Los Angeles (missing underscore) is invalid tzdata — container falls back to UTC. Fixed 2026-08-05: corrected to America/Los_Angeles in the manifest and deployed via the pipeline.
• vaultwarden and home-assistant carry helm-chart-era label/name shapes but their release records are gone; they are plain manifests now.
• pihole runs privileged: true with PIHOLE_UID=0.

History

• 2026-07-25: cluster outage (rpi-master undervoltage → CNI cache corruption). Recovery runbook in the Notes vault: rpi-master_undervoltage_CNI_corruption_incident-2026-07-25.md.
• 2026-07-26: initial import of manifests from live cluster; HA ingress defaultBackend fixed; unifiedstreaming-svc LoadBalancer→ClusterIP; pihole/USP secrets migrated to secretKeyRef in-cluster.
• 2026-07-26: pihole surprise v5→v6 upgrade via :latest on a fresh node pull; image pinned by digest, FTLCONF_dns_listeningMode=ALL + FTLCONF_webserver_api_password (v6 ignores WEBPASSWORD), readiness probe on :53 added. Gitea Actions pipeline + deployer RBAC added.