jasonwitty
7b084dbddd
Two self-hosted apps behind traefik, both keeping their state on the GlusterFS volume that is mounted on every node: - dashwise (home.wittyoneoff.com) is an all-in-one image running its web server, a bundled PocketBase and valkey side by side. Only port 3000 is published: the frontend resolves its backend as window.location.origin and every PocketBase call is made server-side, so 8090 stays inside the pod. It is on the service so the PocketBase admin UI can be reached with kubectl port-forward. - tududi (tududi.wittyoneoff.com) stores a SQLite database and user uploads. Both are VOLUMEs in the image, so both are backed by the claim -- as one volume mounted twice with subPath, because naming the same claim as two volume entries wedges kubelet, which is what happened with limesurvey. Both images are pinned to tags that publish an arm64 manifest. dashwise :latest and :stable are amd64 only and would not start on these nodes. Namespaces and secrets are applied out of band, as with the other apps. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
homelab-k3s
k3s deployments and change management for the homelab Raspberry Pi cluster (rpi-master + rpi-worker-1/2/3, k3s v1.30.3, flannel/traefik/klipper-lb defaults).
Manifests were reverse-engineered from the live cluster on 2026-07-26 and cleaned of runtime fields. Except where a file header says DIVERGENCE, they are faithful to what was running.
Layout
Each app under apps/ is a self-contained kustomization
(namespace + PV/PVC + deployment + service + ingress):
| App | Namespace | Hostname(s) |
|---|---|---|
| vaultwarden | bitwarden | pm.wittyoneoff.com |
| searxng | default | searxng.wittyoneoff.com, origin-searxng.wittyoneoff.com |
| home-assistant | home-assistant | ha.wittyoneoff.com, ha-origin.wittyoneoff.com |
| nginx | nginx | www.wittyoneoff.com, www.ramonaajj.com |
| pihole | pihole | admin on :8000 via klipper-lb, DNS on :53 |
| unbound | pihole | pi-hole's upstream recursive resolver (ClusterIP only) |
| unified-streaming | unified-streaming | unified.wittyoneoff.com |
Not in this repo: socktop (own repo + pipeline), rancher/monitoring/logging (rancher-managed), traefik (k3s packaged component).
Applying
kubectl apply -k . # everything
kubectl apply -k apps/home-assistant # one app
One-time secret bootstrap (per cluster)
Three apps read secrets that are NOT in git. Each app dir has a
secret.example.yaml; copy to secret.yaml (gitignored), fill in, apply.
| Secret | Namespace | Used for |
|---|---|---|
pihole-admin |
pihole | pi-hole admin WEBPASSWORD |
usp-license |
unified-streaming | USP_LICENSE_KEY |
searxng-1723974683-config |
default | searxng settings.yml |
One-time migration (pihole + unified-streaming) — DONE 2026-07-26:
secrets pihole-admin and usp-license created in-cluster and both
deployments switched to secretKeyRef via kubectl replace (a plain
kubectl apply cannot patch an env var from value to valueFrom).
kubectl apply -k . now converges cleanly (verified --dry-run=server).
CI/CD (Gitea Actions)
.gitea/workflows/deploy.yaml, modeled on socktop-webterm's pipeline:
- PRs →
kubectl apply -k . --dry-run=server(validation only) - push to main →
kubectl apply -k .+kubectl rollout statuson all seven deployments
Repo Actions secret required: KUBECONFIG — base64-encoded kubeconfig for the
gitea-deployer ServiceAccount (same identity socktop-webterm deploys with;
regenerate anytime from default/gitea-deployer-token). Its RBAC is
rbac/gitea-deployer.yaml — admin-applied once, deliberately outside the root
kustomization so the pipeline token cannot escalate itself; it has no access
to secrets and no delete verbs.
Cluster assumptions (state that lives outside these manifests)
- Node labels — scheduling relies on them: workers carry
cpu=arm(all nodes do), the control-plane node carriescontroller=true(most apps usecontroller NotIn (true)to stay off the master), pis carrymodel=raspi(pihole/unbound prefer it). Re-label with:kubectl label node <n> cpu=arm model=raspi/kubectl label node rpi-master controller=true. - GlusterFS — every PV is
hostPathunder/mnt/gvolume0/…, which is the gluster volume mounted identically on all nodes. The mount must exist before pods schedule; PVs bind by explicitvolumeName+storageClassName: "". - Ingress/LB — traefik (k3s default) terminates all HTTP on 80/443 via svclb on every node. Because of that, no other LoadBalancer service can claim host port 80 (that's why unifiedstreaming-svc is ClusterIP).
- Home Assistant config — lives on the PV, not in git.
http.trusted_proxiesmust include10.42.0.0/16(the cluster pod CIDR) or HA 400s everything the moment traefik reschedules to another node.
Known quirks (faithful to live state, fix at leisure)
- Most images are
:latest/:stable— deploys are not reproducible until pinned. vaultwarden is pinned (1.35.4); pihole is pinned by digest (Core v6.2.2) after:latestsilently jumped v5→v6 on a fresh node pull (2026-07-26) and broke DNS for cross-node sources (v6 defaults tolisteningMode=LOCAL) — fixed viaFTLCONF_dns_listeningMode=ALL. - searxng keeps its helm-generated name suffix (
searxng-1723974683) because deployment selectors are immutable; its helm release record still exists in the cluster — don't runhelm upgrade/uninstallon it, this repo is the source of truth now. searxng'sFixed 2026-08-05: corrected toTZ=America/Los Angeles(missing underscore) is invalid tzdata — container falls back to UTC.America/Los_Angelesin the manifest and deployed via the pipeline.- vaultwarden and home-assistant carry helm-chart-era label/name shapes but their release records are gone; they are plain manifests now.
- pihole runs
privileged: truewithPIHOLE_UID=0.
History
- 2026-07-25: cluster outage (rpi-master undervoltage → CNI cache corruption).
Recovery runbook in the Notes vault:
rpi-master_undervoltage_CNI_corruption_incident-2026-07-25.md. - 2026-07-26: initial import of manifests from live cluster; HA ingress defaultBackend fixed; unifiedstreaming-svc LoadBalancer→ClusterIP; pihole/USP secrets migrated to secretKeyRef in-cluster.
- 2026-07-26: pihole surprise v5→v6 upgrade via
:lateston a fresh node pull; image pinned by digest,FTLCONF_dns_listeningMode=ALL+FTLCONF_webserver_api_password(v6 ignores WEBPASSWORD), readiness probe on :53 added. Gitea Actions pipeline + deployer RBAC added.