Root cause of the Vault OIDC 'Missing auth_url' error (and the same
symptom on every other VPS-standby app): the Ingress tls: block makes
HTTPS available but Traefik still serves plain HTTP on the same host too
- nothing was forcing the redirect. A browser landing on
http://vault.vps.huskypup.net gets a Vault UI that computes its OIDC
callback using window.location.origin (http://...), which doesn't match
the https:// entry in allowed_redirect_uris - Vault silently returns an
empty auth_url rather than an obviously-config-looking error. Confirmed
via a HAR capture of the actual failing browser request.
Adds a per-namespace Traefik Middleware (redirectScheme -> https,
permanent) referenced via router.middlewares on each Ingress. Applied
directly via kubectl first to test - ArgoCD's selfHeal immediately
reverted it since it wasn't in git yet, confirming the fix needs to ship
through the normal pipeline rather than live kubectl edits on ArgoCD-
managed VPS resources.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The DNS-flip watcher (scripts/vps-dns-failover/) was designed but never
actually installed on the VPS despite being tracked as done - real gap,
found when asked whether the standby services are actually reachable.
New design: instead of dynamically flipping *.kube.huskypup.net between
home and VPS IPs, give the VPS site its own permanent, always-resolving
subdomain - vault/gitea/auth/n8n/nextcloud.vps.huskypup.net, each with
real Ingress+TLS on the VPS's own Traefik+cert-manager (both already
installed by Phase 0 bootstrap, just never wired up). No token-scoping
decision needed since there's no dynamic flipping - reuses the same
cert-manager token pattern as home.
Also scales Authentik/n8n/Nextcloud from 0 to 1 replica on the VPS so
the replicated data is actually browsable at all times, not just
present-but-unreachable. Their CNPG clusters are still read-only
replicas (spec.replica.enabled: true) - writes will error until a
deliberate manual promotion, but reads/browsing work now. Vault and
Gitea were already running continuously.
Postgres refuses to replay WAL past a checkpoint recorded with higher
max_connections/max_wal_senders/etc than the recovering instance's own
settings: 'recovery aborted because of insufficient parameter settings:
max_connections = 100 is a lower setting than on the primary server,
where its value was 200' - confirmed live on all 3 VPS clusters (they
had no postgresql.parameters block at all, defaulting to CNPG's 100).
Copied home's full postgresql.parameters block to remove any other
potential mismatch (max_wal_senders, max_worker_processes are subject
to the same check). Bumped memory requests/limits to match home too -
shared_buffers: 512MB needs headroom the previous 256Mi/1Gi didn't have.
CNPG's barmanObjectStore.serverName defaults to the externalClusters[]
entry's own .name field ('home-backup', an arbitrary label I chose) -
NOT the actual source Postgres cluster's name. Home's backup: block
(on the real pg-authentik/pg-n8n/pg-nextcloud Clusters) defaults
serverName to its own metadata.name instead. Without an explicit
override these two disagree, so recovery searches the object store
under the wrong server-name prefix and finds nothing - 'no target
backup found', confirmed live even after the ScheduledBackup fix
produced real, completed base backups.
Same CNPG replica-cluster pattern as Authentik (continuous WAL replay
from home via VPS MinIO, app at 0 replicas until manual promotion - see
infrastructure/vps-standby/authentik/manifests/cnpg-cluster.yaml for the
full rationale).
n8n: N8N_ENCRYPTION_KEY copied byte-identical from home (kubectl, not
git) - decrypts stored credentials in the replicated DB, same reasoning
as Vault's unseal key / root token copies.
Nextcloud: adds infrastructure/vps-standby/nextcloud/manifests/
pvc-restore-cronjob.yaml, the read side of home's existing
nextcloud-pvc-sync restic backup - restores the latest snapshot from VPS
MinIO into this standby's PVC daily at 04:00 (2h after home's 02:00
backup). No Redis on the VPS side (no redis-operator deployed there,
out of scope for a standby that isn't serving traffic - Nextcloud
degrades gracefully to DB-based locking without it).
