homelab-k3s/README.md

# homelab-k3s

k3s deployments and change management for the homelab Raspberry Pi cluster
(rpi-master + rpi-worker-1/2/3, k3s v1.30.3, flannel/traefik/klipper-lb
defaults).

Manifests were reverse-engineered from the live cluster on 2026-07-26 and
cleaned of runtime fields. Except where a file header says **DIVERGENCE**,
they are faithful to what was running.

## Layout

Each app under `apps/` is a self-contained kustomization
(namespace + PV/PVC + deployment + service + ingress):

| App | Namespace | Hostname(s) |
|---|---|---|
| vaultwarden | bitwarden | pm.wittyoneoff.com |
| searxng | default | searxng.wittyoneoff.com, origin-searxng.wittyoneoff.com |
| home-assistant | home-assistant | ha.wittyoneoff.com, ha-origin.wittyoneoff.com |
| nginx | nginx | www.wittyoneoff.com, www.ramonaajj.com |
| pihole | pihole | admin on :8000 via klipper-lb, DNS on :53 |
| unbound | pihole | pi-hole's upstream recursive resolver (ClusterIP only) |
| unified-streaming | unified-streaming | unified.wittyoneoff.com |

Not in this repo: socktop (own repo + pipeline), rancher/monitoring/logging
(rancher-managed), traefik (k3s packaged component).

## Applying

```sh
kubectl apply -k .                      # everything
kubectl apply -k apps/home-assistant   # one app
```

### One-time secret bootstrap (per cluster)

Three apps read secrets that are NOT in git. Each app dir has a
`secret.example.yaml`; copy to `secret.yaml` (gitignored), fill in, apply.

| Secret | Namespace | Used for |
|---|---|---|
| `pihole-admin` | pihole | pi-hole admin `WEBPASSWORD` |
| `usp-license` | unified-streaming | `USP_LICENSE_KEY` |
| `searxng-1723974683-config` | default | searxng `settings.yml` |

~~**One-time migration (pihole + unified-streaming)**~~ — DONE 2026-07-26:
secrets `pihole-admin` and `usp-license` created in-cluster and both
deployments switched to `secretKeyRef` via `kubectl replace` (a plain
`kubectl apply` cannot patch an env var from `value` to `valueFrom`).
`kubectl apply -k .` now converges cleanly (verified `--dry-run=server`).

## CI/CD (Gitea Actions)

`.gitea/workflows/deploy.yaml`, modeled on socktop-webterm's pipeline:

- **PRs** → `kubectl apply -k . --dry-run=server` (validation only)
- **push to main** → `kubectl apply -k .` + `kubectl rollout status` on all
  seven deployments

Repo Actions secret required: `KUBECONFIG` — base64-encoded kubeconfig for the
`gitea-deployer` ServiceAccount (same identity socktop-webterm deploys with;
regenerate anytime from `default/gitea-deployer-token`). Its RBAC is
`rbac/gitea-deployer.yaml` — admin-applied once, deliberately outside the root
kustomization so the pipeline token cannot escalate itself; it has no access
to secrets and no delete verbs.

## Cluster assumptions (state that lives outside these manifests)

- **Node labels** — scheduling relies on them: workers carry `cpu=arm`
  (all nodes do), the control-plane node carries `controller=true`
  (most apps use `controller NotIn (true)` to stay off the master), pis carry
  `model=raspi` (pihole/unbound prefer it). Re-label with:
  `kubectl label node <n> cpu=arm model=raspi` /
  `kubectl label node rpi-master controller=true`.
- **GlusterFS** — every PV is `hostPath` under `/mnt/gvolume0/…`, which is the
  gluster volume mounted identically on all nodes. The mount must exist before
  pods schedule; PVs bind by explicit `volumeName` + `storageClassName: ""`.
- **Ingress/LB** — traefik (k3s default) terminates all HTTP on 80/443 via
  svclb on every node. Because of that, no other LoadBalancer service can
  claim host port 80 (that's why unifiedstreaming-svc is ClusterIP).
- **Home Assistant config** — lives on the PV, not in git.
  `http.trusted_proxies` must include `10.42.0.0/16` (the cluster pod CIDR)
  or HA 400s everything the moment traefik reschedules to another node.

## Known quirks (faithful to live state, fix at leisure)

- Most images are `:latest`/`:stable` — deploys are not reproducible until
  pinned. vaultwarden is pinned (1.35.4); pihole is pinned by digest (Core
  v6.2.2) after `:latest` silently jumped v5→v6 on a fresh node pull
  (2026-07-26) and broke DNS for cross-node sources (v6 defaults to
  `listeningMode=LOCAL`) — fixed via `FTLCONF_dns_listeningMode=ALL`.
- searxng keeps its helm-generated name suffix (`searxng-1723974683`) because
  deployment selectors are immutable; its helm release record still exists in
  the cluster — don't run `helm upgrade/uninstall` on it, this repo is the
  source of truth now.
- searxng's `TZ=America/Los Angeles` (missing underscore) is invalid tzdata —
  container falls back to UTC.
- vaultwarden and home-assistant carry helm-chart-era label/name shapes but
  their release records are gone; they are plain manifests now.
- pihole runs `privileged: true` with `PIHOLE_UID=0`.

## History

- 2026-07-25: cluster outage (rpi-master undervoltage → CNI cache corruption).
  Recovery runbook in the Notes vault:
  `rpi-master_undervoltage_CNI_corruption_incident-2026-07-25.md`.
- 2026-07-26: initial import of manifests from live cluster; HA ingress
  defaultBackend fixed; unifiedstreaming-svc LoadBalancer→ClusterIP;
  pihole/USP secrets migrated to secretKeyRef in-cluster.
- 2026-07-26: pihole surprise v5→v6 upgrade via `:latest` on a fresh node
  pull; image pinned by digest, `FTLCONF_dns_listeningMode=ALL` +
  `FTLCONF_webserver_api_password` (v6 ignores WEBPASSWORD), readiness probe
  on :53 added. Gitea Actions pipeline + deployer RBAC added.