2026-07-26 08:51:54 +00:00
|
|
|
# homelab-k3s
|
|
|
|
|
|
2026-07-26 09:13:56 +00:00
|
|
|
k3s deployments and change management for the homelab Raspberry Pi cluster
|
|
|
|
|
(rpi-master + rpi-worker-1/2/3, k3s v1.30.3, flannel/traefik/klipper-lb
|
|
|
|
|
defaults).
|
|
|
|
|
|
|
|
|
|
Manifests were reverse-engineered from the live cluster on 2026-07-26 and
|
|
|
|
|
cleaned of runtime fields. Except where a file header says **DIVERGENCE**,
|
|
|
|
|
they are faithful to what was running.
|
|
|
|
|
|
|
|
|
|
## Layout
|
|
|
|
|
|
|
|
|
|
Each app under `apps/` is a self-contained kustomization
|
|
|
|
|
(namespace + PV/PVC + deployment + service + ingress):
|
|
|
|
|
|
|
|
|
|
| App | Namespace | Hostname(s) |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| vaultwarden | bitwarden | pm.wittyoneoff.com |
|
|
|
|
|
| searxng | default | searxng.wittyoneoff.com, origin-searxng.wittyoneoff.com |
|
|
|
|
|
| home-assistant | home-assistant | ha.wittyoneoff.com, ha-origin.wittyoneoff.com |
|
|
|
|
|
| nginx | nginx | www.wittyoneoff.com, www.ramonaajj.com |
|
|
|
|
|
| pihole (+unbound) | pihole | admin on :8000 via klipper-lb, DNS on :53 |
|
|
|
|
|
| unified-streaming | unified-streaming | unified.wittyoneoff.com |
|
|
|
|
|
|
|
|
|
|
Not in this repo: socktop (own repo + pipeline), rancher/monitoring/logging
|
|
|
|
|
(rancher-managed), traefik (k3s packaged component).
|
|
|
|
|
|
|
|
|
|
## Applying
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
kubectl apply -k . # everything
|
|
|
|
|
kubectl apply -k apps/home-assistant # one app
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### One-time secret bootstrap (per cluster)
|
|
|
|
|
|
|
|
|
|
Three apps read secrets that are NOT in git. Each app dir has a
|
|
|
|
|
`secret.example.yaml`; copy to `secret.yaml` (gitignored), fill in, apply.
|
|
|
|
|
|
|
|
|
|
| Secret | Namespace | Used for |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| `pihole-admin` | pihole | pi-hole admin `WEBPASSWORD` |
|
|
|
|
|
| `usp-license` | unified-streaming | `USP_LICENSE_KEY` |
|
|
|
|
|
| `searxng-1723974683-config` | default | searxng `settings.yml` |
|
|
|
|
|
|
|
|
|
|
**One-time migration (pihole + unified-streaming):** the live cluster (as of
|
|
|
|
|
2026-07-26) has WEBPASSWORD / USP_LICENSE_KEY as inline plaintext `value`s.
|
|
|
|
|
`kubectl apply` (client- or server-side) cannot patch an env var from `value`
|
|
|
|
|
to `valueFrom` — the API rejects the merged object. Validated procedure:
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
# 1. create the two secrets (see secret.example.yaml in each app dir)
|
|
|
|
|
# 2. one-time replace (restarts the pods):
|
|
|
|
|
kubectl replace -f apps/pihole/deployment-pihole.yaml
|
|
|
|
|
kubectl replace -f apps/unified-streaming/deployment.yaml
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
After that, `kubectl apply -k .` works cleanly for everything (verified with
|
|
|
|
|
`--dry-run=server` against the live cluster, 2026-07-26 — zero immutable-field
|
|
|
|
|
conflicts).
|
|
|
|
|
|
|
|
|
|
## Cluster assumptions (state that lives outside these manifests)
|
|
|
|
|
|
|
|
|
|
- **Node labels** — scheduling relies on them: workers carry `cpu=arm`
|
|
|
|
|
(all nodes do), the control-plane node carries `controller=true`
|
|
|
|
|
(most apps use `controller NotIn (true)` to stay off the master), pis carry
|
|
|
|
|
`model=raspi` (pihole/unbound prefer it). Re-label with:
|
|
|
|
|
`kubectl label node <n> cpu=arm model=raspi` /
|
|
|
|
|
`kubectl label node rpi-master controller=true`.
|
|
|
|
|
- **GlusterFS** — every PV is `hostPath` under `/mnt/gvolume0/…`, which is the
|
|
|
|
|
gluster volume mounted identically on all nodes. The mount must exist before
|
|
|
|
|
pods schedule; PVs bind by explicit `volumeName` + `storageClassName: ""`.
|
|
|
|
|
- **Ingress/LB** — traefik (k3s default) terminates all HTTP on 80/443 via
|
|
|
|
|
svclb on every node. Because of that, no other LoadBalancer service can
|
|
|
|
|
claim host port 80 (that's why unifiedstreaming-svc is ClusterIP).
|
|
|
|
|
- **Home Assistant config** — lives on the PV, not in git.
|
|
|
|
|
`http.trusted_proxies` must include `10.42.0.0/16` (the cluster pod CIDR)
|
|
|
|
|
or HA 400s everything the moment traefik reschedules to another node.
|
|
|
|
|
|
|
|
|
|
## Known quirks (faithful to live state, fix at leisure)
|
|
|
|
|
|
|
|
|
|
- Most images are `:latest`/`:stable` — deploys are not reproducible until
|
|
|
|
|
pinned. vaultwarden is pinned (1.35.4).
|
|
|
|
|
- searxng keeps its helm-generated name suffix (`searxng-1723974683`) because
|
|
|
|
|
deployment selectors are immutable; its helm release record still exists in
|
|
|
|
|
the cluster — don't run `helm upgrade/uninstall` on it, this repo is the
|
|
|
|
|
source of truth now.
|
|
|
|
|
- searxng's `TZ=America/Los Angeles` (missing underscore) is invalid tzdata —
|
|
|
|
|
container falls back to UTC.
|
|
|
|
|
- vaultwarden and home-assistant carry helm-chart-era label/name shapes but
|
|
|
|
|
their release records are gone; they are plain manifests now.
|
|
|
|
|
- pihole runs `privileged: true` with `PIHOLE_UID=0`.
|
|
|
|
|
|
|
|
|
|
## History
|
|
|
|
|
|
|
|
|
|
- 2026-07-25: cluster outage (rpi-master undervoltage → CNI cache corruption).
|
|
|
|
|
Recovery runbook in the Notes vault:
|
|
|
|
|
`rpi-master_undervoltage_CNI_corruption_incident-2026-07-25.md`.
|
|
|
|
|
- 2026-07-26: initial import of manifests from live cluster; HA ingress
|
|
|
|
|
defaultBackend fixed; unifiedstreaming-svc LoadBalancer→ClusterIP;
|
|
|
|
|
pihole/USP secrets moved to secretKeyRef (pending first apply).
|
