Builds the one genuinely new component from the plan - everything else
reuses CNPG/etcd as-is.
- infrastructure/ha-failover/manifests/heartbeat-writer.yaml: home writes
a fresh timestamp to etcd (/ha-failover/home-heartbeat) every 10s via
its local etcd member. No custom quorum/voting logic - etcd's own Raft
consensus (majority write/linearizable read) already gives the 'a
majority agrees' guarantee.
- scripts/ha-failover-watcher.sh + .service: runs on the VPS as a
systemd service (not k3s - must survive the VPS's own cluster being
unhealthy). Reads the heartbeat via its own local etcd member; if the
read itself fails, we can't tell if home is down or if this VPS is the
one partitioned, so it does NOT act (etcd's consensus requirement
provides the split-brain safety here, not custom code). If the read
succeeds and the heartbeat is stale (>45s, ~4-5 missed beats), home is
confirmed down by majority - promotes pg-authentik's Cluster CR
(spec.replica self/primary/source -> vps, no promotionToken - confirmed
live via dry-run that CNPG's webhook doesn't require one, it's only for
planned/graceful switchover LSN cross-checks that aren't available
during a genuine unplanned outage) and flips the
pg-authentik.ha.huskypup.net Cloudflare A record to the VPS's IP.
- scripts/ha-failback-authentik.sh: the deliberately manual reverse -
human confirms home is healthy and caught up before running this.
- Floating hostname plumbing: both sites' Authentik values now read
AUTHENTIK_POSTGRESQL__HOST/PORT as pg-authentik.ha.huskypup.net:61432
instead of the local pg-authentik-app secret's host/port, so app pods
on either site always reach whichever site is actually primary. Both
sites listen on the same external port (VPS gets a new
pg-authentik-forward.service socat forward, mirroring the existing
minio-forward.service pattern, so its NodePort 32433 is externally
reachable on 61432 same as home's UniFi-forwarded port).
- Found and worked around two real bugs surfaced while wiring this up
(fixed manually via scripts handed to the user - both blocked by the
Claude Code auto-mode classifier as credential-transmission /
cluster-DNS-edit actions):
- VPS's pg-authentik-app secret had a STALE password from before
streaming replication existed - the live Postgres role password now
replicates from home via WAL, but VPS's local K8s secret never got
updated to match. Needs a one-time sync (and again on any future
rotation).
- This UniFi does not support NAT hairpin/loopback for its own WAN
port-forwards - home's own pods resolving the floating hostname need
a local CoreDNS rewrite straight to pg-authentik-rw, confirmed live by
a DNS-resolves-but-TCP-connect-fails test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The DNS-flip watcher (scripts/vps-dns-failover/) was designed but never
actually installed on the VPS despite being tracked as done - real gap,
found when asked whether the standby services are actually reachable.
New design: instead of dynamically flipping *.kube.huskypup.net between
home and VPS IPs, give the VPS site its own permanent, always-resolving
subdomain - vault/gitea/auth/n8n/nextcloud.vps.huskypup.net, each with
real Ingress+TLS on the VPS's own Traefik+cert-manager (both already
installed by Phase 0 bootstrap, just never wired up). No token-scoping
decision needed since there's no dynamic flipping - reuses the same
cert-manager token pattern as home.
Also scales Authentik/n8n/Nextcloud from 0 to 1 replica on the VPS so
the replicated data is actually browsable at all times, not just
present-but-unreachable. Their CNPG clusters are still read-only
replicas (spec.replica.enabled: true) - writes will error until a
deliberate manual promotion, but reads/browsing work now. Vault and
Gitea were already running continuously.
- ZONE_NAME was "kube.huskypup.net" in both the home-ip-ddns CronJob and
the VPS failover watcher - that's a record, not a Cloudflare zone (the
actual zone is the parent "huskypup.net"). Caused home-ip-ddns to fail
every run (zone lookup returned zero results, curl -f exit 22) -
confirmed live and fixed.
- Added seccompProfile/non-root/dropped-capabilities securityContext to
the three CronJobs added this session that were missing it (flagged by
the cluster's "restricted" PodSecurity admission). Repointed the raft
snapshot job's mc binary install from /usr/local/bin to /tmp so it
still works running as non-root.
Foundation for a DR/backup path using an always-on VPS as a second
ArgoCD-managed cluster, plus DB/backup standardization work that fell
out of it:
- vps-standby ArgoCD cluster destination + AppProject, MinIO backup
receiver, VPS bootstrap script (k3s, Netbird, cert-manager)
- Dual-site DNS failover watcher + home-IP DDNS CronJob, Cloudflare
token moved out of git into Vault+ExternalSecret
- Nextcloud migrated from ad-hoc MariaDB to CNPG + redis-operator
(matches n8n/Authentik/GitLab's backup-native pattern)
- Authentik's CNPG manifests moved into the actual ArgoCD-synced
manifests/ path (were present but never wired into the sync path)
- Vault raft-snapshot CronJob, CNPG barmanObjectStore backups
(Authentik/n8n/Nextcloud), Nextcloud file-PVC restic sync - all
targeting the new VPS MinIO receiver
See VPS Warm-Standby plan doc for full design rationale.
