Confirmed live: CrashLoopBackOff (Bad Gateway from the ingress) caused
by the default 40s-runway liveness probe tripping on a transient DB
connection blip. The worker constantly retries a scheduled-task query
needing a write lock against the read-only CNPG replica (expected,
harmless - 'cannot execute SELECT FOR UPDATE in a read-only
transaction') which appears to add enough connection pressure that a
brief hiccup trips the server's stricter default probe.
The DNS-flip watcher (scripts/vps-dns-failover/) was designed but never
actually installed on the VPS despite being tracked as done - real gap,
found when asked whether the standby services are actually reachable.
New design: instead of dynamically flipping *.kube.huskypup.net between
home and VPS IPs, give the VPS site its own permanent, always-resolving
subdomain - vault/gitea/auth/n8n/nextcloud.vps.huskypup.net, each with
real Ingress+TLS on the VPS's own Traefik+cert-manager (both already
installed by Phase 0 bootstrap, just never wired up). No token-scoping
decision needed since there's no dynamic flipping - reuses the same
cert-manager token pattern as home.
Also scales Authentik/n8n/Nextcloud from 0 to 1 replica on the VPS so
the replicated data is actually browsable at all times, not just
present-but-unreachable. Their CNPG clusters are still read-only
replicas (spec.replica.enabled: true) - writes will error until a
deliberate manual promotion, but reads/browsing work now. Vault and
Gitea were already running continuously.
CNPG operator on the VPS (argocd-apps/vps-standby/cnpg-operator.yaml),
plus a CNPG 'replica cluster' for pg-authentik (infrastructure/
vps-standby/authentik/manifests/cnpg-cluster.yaml) that continuously
replays WAL shipped from home's pg-authentik via VPS MinIO's
cnpg-backups/pg-authentik bucket path - CNPG's native replica-cluster
mode, not a custom restore script like Vault needed (Vault has no
equivalent built-in continuous-replication-into-object-store feature).
The Authentik app itself runs at replicas: 0 in normal operation - the
replica DB is read-only until a deliberate manual promotion
(spec.replica.enabled: false), and a pod trying to write against a
read-only DB would just crashloop uselessly. values.yaml's authentik:
block is copied byte-identical from home's on purpose: the chart
deterministically derives its generated Secret's AUTHENTIK_SECRET_KEY
from these literal values.yaml strings, so both clusters land on the
same key without manually copying it - required since that key decrypts
things stored encrypted in the replicated DB.
Scooby Husky