fe72f1e85a
A fresh node pull of pihole:latest silently upgraded v5->v6 (Core v6.2.2). v6's default listeningMode=LOCAL drops DNS from non-attached subnets, which killed all cross-node svclb DNS the moment the pod rescheduled off its old node. v6 also ignores the WEBPASSWORD env. - image pinned by digest (Core v6.2.2) - FTLCONF_dns_listeningMode=ALL - admin password via FTLCONF_webserver_api_password <- pihole-admin secret - tcpSocket:53 readiness probe so rollout status waits for FTL Applied to the cluster via kubectl replace; DNS verified answering on all four node IPs, admin API auth verified. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
114 lines
5.3 KiB
Markdown
114 lines
5.3 KiB
Markdown
# homelab-k3s
|
|
|
|
k3s deployments and change management for the homelab Raspberry Pi cluster
|
|
(rpi-master + rpi-worker-1/2/3, k3s v1.30.3, flannel/traefik/klipper-lb
|
|
defaults).
|
|
|
|
Manifests were reverse-engineered from the live cluster on 2026-07-26 and
|
|
cleaned of runtime fields. Except where a file header says **DIVERGENCE**,
|
|
they are faithful to what was running.
|
|
|
|
## Layout
|
|
|
|
Each app under `apps/` is a self-contained kustomization
|
|
(namespace + PV/PVC + deployment + service + ingress):
|
|
|
|
| App | Namespace | Hostname(s) |
|
|
|---|---|---|
|
|
| vaultwarden | bitwarden | pm.wittyoneoff.com |
|
|
| searxng | default | searxng.wittyoneoff.com, origin-searxng.wittyoneoff.com |
|
|
| home-assistant | home-assistant | ha.wittyoneoff.com, ha-origin.wittyoneoff.com |
|
|
| nginx | nginx | www.wittyoneoff.com, www.ramonaajj.com |
|
|
| pihole (+unbound) | pihole | admin on :8000 via klipper-lb, DNS on :53 |
|
|
| unified-streaming | unified-streaming | unified.wittyoneoff.com |
|
|
|
|
Not in this repo: socktop (own repo + pipeline), rancher/monitoring/logging
|
|
(rancher-managed), traefik (k3s packaged component).
|
|
|
|
## Applying
|
|
|
|
```sh
|
|
kubectl apply -k . # everything
|
|
kubectl apply -k apps/home-assistant # one app
|
|
```
|
|
|
|
### One-time secret bootstrap (per cluster)
|
|
|
|
Three apps read secrets that are NOT in git. Each app dir has a
|
|
`secret.example.yaml`; copy to `secret.yaml` (gitignored), fill in, apply.
|
|
|
|
| Secret | Namespace | Used for |
|
|
|---|---|---|
|
|
| `pihole-admin` | pihole | pi-hole admin `WEBPASSWORD` |
|
|
| `usp-license` | unified-streaming | `USP_LICENSE_KEY` |
|
|
| `searxng-1723974683-config` | default | searxng `settings.yml` |
|
|
|
|
~~**One-time migration (pihole + unified-streaming)**~~ — DONE 2026-07-26:
|
|
secrets `pihole-admin` and `usp-license` created in-cluster and both
|
|
deployments switched to `secretKeyRef` via `kubectl replace` (a plain
|
|
`kubectl apply` cannot patch an env var from `value` to `valueFrom`).
|
|
`kubectl apply -k .` now converges cleanly (verified `--dry-run=server`).
|
|
|
|
## CI/CD (Gitea Actions)
|
|
|
|
`.gitea/workflows/deploy.yaml`, modeled on socktop-webterm's pipeline:
|
|
|
|
- **PRs** → `kubectl apply -k . --dry-run=server` (validation only)
|
|
- **push to main** → `kubectl apply -k .` + `kubectl rollout status` on all
|
|
seven deployments
|
|
|
|
Repo Actions secret required: `KUBECONFIG` — base64-encoded kubeconfig for the
|
|
`gitea-deployer` ServiceAccount (same identity socktop-webterm deploys with;
|
|
regenerate anytime from `default/gitea-deployer-token`). Its RBAC is
|
|
`rbac/gitea-deployer.yaml` — admin-applied once, deliberately outside the root
|
|
kustomization so the pipeline token cannot escalate itself; it has no access
|
|
to secrets and no delete verbs.
|
|
|
|
## Cluster assumptions (state that lives outside these manifests)
|
|
|
|
- **Node labels** — scheduling relies on them: workers carry `cpu=arm`
|
|
(all nodes do), the control-plane node carries `controller=true`
|
|
(most apps use `controller NotIn (true)` to stay off the master), pis carry
|
|
`model=raspi` (pihole/unbound prefer it). Re-label with:
|
|
`kubectl label node <n> cpu=arm model=raspi` /
|
|
`kubectl label node rpi-master controller=true`.
|
|
- **GlusterFS** — every PV is `hostPath` under `/mnt/gvolume0/…`, which is the
|
|
gluster volume mounted identically on all nodes. The mount must exist before
|
|
pods schedule; PVs bind by explicit `volumeName` + `storageClassName: ""`.
|
|
- **Ingress/LB** — traefik (k3s default) terminates all HTTP on 80/443 via
|
|
svclb on every node. Because of that, no other LoadBalancer service can
|
|
claim host port 80 (that's why unifiedstreaming-svc is ClusterIP).
|
|
- **Home Assistant config** — lives on the PV, not in git.
|
|
`http.trusted_proxies` must include `10.42.0.0/16` (the cluster pod CIDR)
|
|
or HA 400s everything the moment traefik reschedules to another node.
|
|
|
|
## Known quirks (faithful to live state, fix at leisure)
|
|
|
|
- Most images are `:latest`/`:stable` — deploys are not reproducible until
|
|
pinned. vaultwarden is pinned (1.35.4); pihole is pinned by digest (Core
|
|
v6.2.2) after `:latest` silently jumped v5→v6 on a fresh node pull
|
|
(2026-07-26) and broke DNS for cross-node sources (v6 defaults to
|
|
`listeningMode=LOCAL`) — fixed via `FTLCONF_dns_listeningMode=ALL`.
|
|
- searxng keeps its helm-generated name suffix (`searxng-1723974683`) because
|
|
deployment selectors are immutable; its helm release record still exists in
|
|
the cluster — don't run `helm upgrade/uninstall` on it, this repo is the
|
|
source of truth now.
|
|
- searxng's `TZ=America/Los Angeles` (missing underscore) is invalid tzdata —
|
|
container falls back to UTC.
|
|
- vaultwarden and home-assistant carry helm-chart-era label/name shapes but
|
|
their release records are gone; they are plain manifests now.
|
|
- pihole runs `privileged: true` with `PIHOLE_UID=0`.
|
|
|
|
## History
|
|
|
|
- 2026-07-25: cluster outage (rpi-master undervoltage → CNI cache corruption).
|
|
Recovery runbook in the Notes vault:
|
|
`rpi-master_undervoltage_CNI_corruption_incident-2026-07-25.md`.
|
|
- 2026-07-26: initial import of manifests from live cluster; HA ingress
|
|
defaultBackend fixed; unifiedstreaming-svc LoadBalancer→ClusterIP;
|
|
pihole/USP secrets migrated to secretKeyRef in-cluster.
|
|
- 2026-07-26: pihole surprise v5→v6 upgrade via `:latest` on a fresh node
|
|
pull; image pinned by digest, `FTLCONF_dns_listeningMode=ALL` +
|
|
`FTLCONF_webserver_api_password` (v6 ignores WEBPASSWORD), readiness probe
|
|
on :53 added. Gitea Actions pipeline + deployer RBAC added.
|