Root cause of the Vault OIDC 'Missing auth_url' error (and the same
symptom on every other VPS-standby app): the Ingress tls: block makes
HTTPS available but Traefik still serves plain HTTP on the same host too
- nothing was forcing the redirect. A browser landing on
http://vault.vps.huskypup.net gets a Vault UI that computes its OIDC
callback using window.location.origin (http://...), which doesn't match
the https:// entry in allowed_redirect_uris - Vault silently returns an
empty auth_url rather than an obviously-config-looking error. Confirmed
via a HAR capture of the actual failing browser request.
Adds a per-namespace Traefik Middleware (redirectScheme -> https,
permanent) referenced via router.middlewares on each Ingress. Applied
directly via kubectl first to test - ArgoCD's selfHeal immediately
reverted it since it wasn't in git yet, confirming the fix needs to ship
through the normal pipeline rather than live kubectl edits on ArgoCD-
managed VPS resources.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The DNS-flip watcher (scripts/vps-dns-failover/) was designed but never
actually installed on the VPS despite being tracked as done - real gap,
found when asked whether the standby services are actually reachable.
New design: instead of dynamically flipping *.kube.huskypup.net between
home and VPS IPs, give the VPS site its own permanent, always-resolving
subdomain - vault/gitea/auth/n8n/nextcloud.vps.huskypup.net, each with
real Ingress+TLS on the VPS's own Traefik+cert-manager (both already
installed by Phase 0 bootstrap, just never wired up). No token-scoping
decision needed since there's no dynamic flipping - reuses the same
cert-manager token pattern as home.
Also scales Authentik/n8n/Nextcloud from 0 to 1 replica on the VPS so
the replicated data is actually browsable at all times, not just
present-but-unreachable. Their CNPG clusters are still read-only
replicas (spec.replica.enabled: true) - writes will error until a
deliberate manual promotion, but reads/browsing work now. Vault and
Gitea were already running continuously.
Confirmed end-to-end tonight: condition=Ready correctly times out every
cycle since Vault can't be Ready while sealed (the unseal step comes
right after this wait) - harmless via the existing || true fallback, but
wastes up to 2 minutes per restore cycle waiting on a condition that can
never be met at this point. Poll for the container process merely being
started instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Without it, kubectl wait's internal watch retry loop never terminates
cleanly even with --timeout set - confirmed live, it kept retrying well
past the stated 120s timeout. The restore/unseal steps themselves still
succeed regardless (verified end-to-end with a manual unseal), this only
affects the job's own graceful detection of the pod coming back.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
raft snapshot restore is a privileged operation - unsealing alone isn't
enough, it needs an authenticated token. Missed this on first pass, caught
live: 'Code: 403. Errors: * permission denied'.
First run: uses the throwaway init's own fresh root token. Steady state
(already restored at least once): uses a copy of home's real root token,
which becomes valid on this Vault the moment the first restore completes
(its auth data becomes byte-for-byte home's as of that snapshot). Stored
the same way as the unseal key - kubectl directly on the VPS, not git.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
StatefulSet pod naming follows the Helm release name (vps-vault, matching
the ArgoCD Application name), not the chart's default.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Single-node Vault (raft storage, standalone mode) plus a restore CronJob
that pulls the latest raft snapshot from the VPS's own local MinIO
(already receiving snapshots every 6h from home) and restores it every
6h, offset 30 minutes after the home-side snapshot job.
Not an independent root of trust: raft snapshot restore replaces the
entire storage backend including the keyring, so after every restore
this Vault is sealed with HOME's actual keyring - unsealing it needs a
copy of home's real unseal key, stored directly on the VPS cluster
(kubectl, not git, same pattern as vps-minio-root-secret). First-run
bootstrap uses a throwaway single-Shamir-key init just to get through
the very first restore, then is irrelevant from then on.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
