jason/homelab-k3s

Fork 0

Files

T

jasonwitty dde8b17fc4

Validate and Deploy to K3s / validate (push) Successful in 19s

Details

Validate and Deploy to K3s / deploy (push) Successful in 25s

Details

Wait for dashwise and tududi in CI, document both

The deploy job only waited on the deployments that existed when it was
written, so a broken rollout of either new app would have been reported
as a successful deploy.

Also records the two new secrets in the bootstrap table, and the two
traps found while adding these: dashwise publishes :latest for amd64
only, and a server-side dry run of a brand new namespace reports every
resource inside it as missing until the namespace exists.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-12 13:06:02 -07:00

6.1 KiB

Raw Blame History

homelab-k3s

k3s deployments and change management for the homelab Raspberry Pi cluster (rpi-master + rpi-worker-1/2/3, k3s v1.30.3, flannel/traefik/klipper-lb defaults).

Manifests were reverse-engineered from the live cluster on 2026-07-26 and cleaned of runtime fields. Except where a file header says DIVERGENCE, they are faithful to what was running.

Layout

Each app under apps/ is a self-contained kustomization (namespace + PV/PVC + deployment + service + ingress):

App	Namespace	Hostname(s)
vaultwarden	bitwarden	pm.wittyoneoff.com
searxng	default	searxng.wittyoneoff.com, origin-searxng.wittyoneoff.com
home-assistant	home-assistant	ha.wittyoneoff.com, ha-origin.wittyoneoff.com
nginx	nginx	www.wittyoneoff.com, www.ramonaajj.com
pihole	pihole	admin on :8000 via klipper-lb, DNS on :53
unbound	pihole	pi-hole's upstream recursive resolver (ClusterIP only)
unified-streaming	unified-streaming	unified.wittyoneoff.com
dashwise	dashwise	home.wittyoneoff.com, origin-home.wittyoneoff.com
tududi	tududi	tududi.wittyoneoff.com, origin-tududi.wittyoneoff.com

Not in this repo: socktop (own repo + pipeline), rancher/monitoring/logging (rancher-managed), traefik (k3s packaged component).

Applying

kubectl apply -k .                      # everything
kubectl apply -k apps/home-assistant   # one app

One-time secret bootstrap (per cluster)

Five apps read secrets that are NOT in git. Each app dir has a secret.example.yaml; copy to secret.yaml (gitignored), fill in, apply.

Secret	Namespace	Used for
`pihole-admin`	pihole	pi-hole admin `WEBPASSWORD`
`usp-license`	unified-streaming	`USP_LICENSE_KEY`
`searxng-1723974683-config`	default	searxng `settings.yml`
`dashwise`	dashwise	PocketBase superuser (also the admin UI login)
`tududi`	tududi	login + `TUDUDI_SESSION_SECRET` cookie signing key

~~One-time migration (pihole + unified-streaming)~~ — DONE 2026-07-26: secrets pihole-admin and usp-license created in-cluster and both deployments switched to secretKeyRef via kubectl replace (a plain kubectl apply cannot patch an env var from value to valueFrom). kubectl apply -k . now converges cleanly (verified --dry-run=server).

CI/CD (Gitea Actions)

.gitea/workflows/deploy.yaml, modeled on socktop-webterm's pipeline:

PRs → kubectl apply -k . --dry-run=server (validation only)
push to main → kubectl apply -k . + kubectl rollout status on all nine deployments

Repo Actions secret required: KUBECONFIG — base64-encoded kubeconfig for the gitea-deployer ServiceAccount (same identity socktop-webterm deploys with; regenerate anytime from default/gitea-deployer-token). Its RBAC is rbac/gitea-deployer.yaml — admin-applied once, deliberately outside the root kustomization so the pipeline token cannot escalate itself; it has no access to secrets and no delete verbs.

Cluster assumptions (state that lives outside these manifests)

Node labels — scheduling relies on them: workers carry cpu=arm (all nodes do), the control-plane node carries controller=true (most apps use controller NotIn (true) to stay off the master), pis carry model=raspi (pihole/unbound prefer it). Re-label with: kubectl label node <n> cpu=arm model=raspi / kubectl label node rpi-master controller=true.
GlusterFS — every PV is hostPath under /mnt/gvolume0/…, which is the gluster volume mounted identically on all nodes. The mount must exist before pods schedule; PVs bind by explicit volumeName + storageClassName: "".
Ingress/LB — traefik (k3s default) terminates all HTTP on 80/443 via svclb on every node. Because of that, no other LoadBalancer service can claim host port 80 (that's why unifiedstreaming-svc is ClusterIP).
Home Assistant config — lives on the PV, not in git. http.trusted_proxies must include 10.42.0.0/16 (the cluster pod CIDR) or HA 400s everything the moment traefik reschedules to another node.

Known quirks (faithful to live state, fix at leisure)

Most images are :latest/:stable — deploys are not reproducible until pinned. vaultwarden is pinned (1.35.4); pihole is pinned by digest (Core v6.2.2) after :latest silently jumped v5→v6 on a fresh node pull (2026-07-26) and broke DNS for cross-node sources (v6 defaults to listeningMode=LOCAL) — fixed via FTLCONF_dns_listeningMode=ALL.
searxng keeps its helm-generated name suffix (searxng-1723974683) because deployment selectors are immutable; its helm release record still exists in the cluster — don't run helm upgrade/uninstall on it, this repo is the source of truth now.
~~searxng's TZ=America/Los Angeles (missing underscore) is invalid tzdata — container falls back to UTC.~~ Fixed 2026-08-05: corrected to America/Los_Angeles in the manifest and deployed via the pipeline.
vaultwarden and home-assistant carry helm-chart-era label/name shapes but their release records are gone; they are plain manifests now.
pihole runs privileged: true with PIHOLE_UID=0.
dashwise :latest and :stable are built for amd64 only and will not run on these nodes; it is pinned to a tag that publishes an arm64 manifest.
dashwise and tududi both create namespaces, so the very first --dry-run=server for a new app reports "namespace not found" for everything inside it. Create the namespace first, then validate.

History

2026-07-25: cluster outage (rpi-master undervoltage → CNI cache corruption). Recovery runbook in the Notes vault: rpi-master_undervoltage_CNI_corruption_incident-2026-07-25.md.
2026-07-26: initial import of manifests from live cluster; HA ingress defaultBackend fixed; unifiedstreaming-svc LoadBalancer→ClusterIP; pihole/USP secrets migrated to secretKeyRef in-cluster.
2026-07-26: pihole surprise v5→v6 upgrade via :latest on a fresh node pull; image pinned by digest, FTLCONF_dns_listeningMode=ALL + FTLCONF_webserver_api_password (v6 ignores WEBPASSWORD), readiness probe on :53 added. Gitea Actions pipeline + deployer RBAC added.

6.1 KiB Raw Blame History