jason/homelab-k3s
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
8afac39d03
homelab-k3s/README.md
jasonwitty 8afac39d03 Import manifests reverse-engineered from live cluster 
Captured 2026-07-26 from rpi-master (k3s v1.30.3) and cleaned of runtime
fields. Six apps as per-app kustomizations: vaultwarden, searxng,
home-assistant, nginx, pihole(+unbound), unified-streaming.

Intentional divergences from live state:
- pihole WEBPASSWORD and USP_LICENSE_KEY moved from inline plaintext env
  to secretKeyRef (secrets gitignored; templates in secret.example.yaml)
- HA ingress defaultBackend fixed (pointed at nonexistent service)
- unifiedstreaming-svc kept as ClusterIP (LoadBalancer could never bind
  port 80 behind svclb-traefik)

Validated against the live cluster with kubectl apply --dry-run=server:
no immutable-field conflicts; one-time kubectl replace procedure for the
two env->secretKeyRef migrations documented in README.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 02:13:56 -07:00

4.5 KiB
Raw Blame History

homelab-k3s

k3s deployments and change management for the homelab Raspberry Pi cluster (rpi-master + rpi-worker-1/2/3, k3s v1.30.3, flannel/traefik/klipper-lb defaults).

Manifests were reverse-engineered from the live cluster on 2026-07-26 and cleaned of runtime fields. Except where a file header says DIVERGENCE, they are faithful to what was running.

Layout

Each app under apps/ is a self-contained kustomization (namespace + PV/PVC + deployment + service + ingress):

App Namespace Hostname(s)
vaultwarden bitwarden pm.wittyoneoff.com
searxng default searxng.wittyoneoff.com, origin-searxng.wittyoneoff.com
home-assistant home-assistant ha.wittyoneoff.com, ha-origin.wittyoneoff.com
nginx nginx www.wittyoneoff.com, www.ramonaajj.com
pihole (+unbound) pihole admin on :8000 via klipper-lb, DNS on :53
unified-streaming unified-streaming unified.wittyoneoff.com

Not in this repo: socktop (own repo + pipeline), rancher/monitoring/logging (rancher-managed), traefik (k3s packaged component).

Applying

kubectl apply -k .                      # everything
kubectl apply -k apps/home-assistant   # one app

One-time secret bootstrap (per cluster)

Three apps read secrets that are NOT in git. Each app dir has a secret.example.yaml; copy to secret.yaml (gitignored), fill in, apply.

Secret Namespace Used for
pihole-admin pihole pi-hole admin WEBPASSWORD
usp-license unified-streaming USP_LICENSE_KEY
searxng-1723974683-config default searxng settings.yml

One-time migration (pihole + unified-streaming): the live cluster (as of 2026-07-26) has WEBPASSWORD / USP_LICENSE_KEY as inline plaintext values. kubectl apply (client- or server-side) cannot patch an env var from value to valueFrom — the API rejects the merged object. Validated procedure:

# 1. create the two secrets (see secret.example.yaml in each app dir)
# 2. one-time replace (restarts the pods):
kubectl replace -f apps/pihole/deployment-pihole.yaml
kubectl replace -f apps/unified-streaming/deployment.yaml

After that, kubectl apply -k . works cleanly for everything (verified with --dry-run=server against the live cluster, 2026-07-26 — zero immutable-field conflicts).

Cluster assumptions (state that lives outside these manifests)

  • Node labels — scheduling relies on them: workers carry cpu=arm (all nodes do), the control-plane node carries controller=true (most apps use controller NotIn (true) to stay off the master), pis carry model=raspi (pihole/unbound prefer it). Re-label with: kubectl label node <n> cpu=arm model=raspi / kubectl label node rpi-master controller=true.
  • GlusterFS — every PV is hostPath under /mnt/gvolume0/…, which is the gluster volume mounted identically on all nodes. The mount must exist before pods schedule; PVs bind by explicit volumeName + storageClassName: "".
  • Ingress/LB — traefik (k3s default) terminates all HTTP on 80/443 via svclb on every node. Because of that, no other LoadBalancer service can claim host port 80 (that's why unifiedstreaming-svc is ClusterIP).
  • Home Assistant config — lives on the PV, not in git. http.trusted_proxies must include 10.42.0.0/16 (the cluster pod CIDR) or HA 400s everything the moment traefik reschedules to another node.

Known quirks (faithful to live state, fix at leisure)

  • Most images are :latest/:stable — deploys are not reproducible until pinned. vaultwarden is pinned (1.35.4).
  • searxng keeps its helm-generated name suffix (searxng-1723974683) because deployment selectors are immutable; its helm release record still exists in the cluster — don't run helm upgrade/uninstall on it, this repo is the source of truth now.
  • searxng's TZ=America/Los Angeles (missing underscore) is invalid tzdata — container falls back to UTC.
  • vaultwarden and home-assistant carry helm-chart-era label/name shapes but their release records are gone; they are plain manifests now.
  • pihole runs privileged: true with PIHOLE_UID=0.

History

  • 2026-07-25: cluster outage (rpi-master undervoltage → CNI cache corruption). Recovery runbook in the Notes vault: rpi-master_undervoltage_CNI_corruption_incident-2026-07-25.md.
  • 2026-07-26: initial import of manifests from live cluster; HA ingress defaultBackend fixed; unifiedstreaming-svc LoadBalancer→ClusterIP; pihole/USP secrets moved to secretKeyRef (pending first apply).