- infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml:
creationPolicy Merge -> Owner (already applied live, committing to
match). Merge assumed CNPG creates a base pg-gitlab-app/pg-praefect-app
secret to merge a password into - wrong for these VPS Clusters since
Phase 1c's replica-cluster recreate; CNPG doesn't create a local
owner secret for a replicated role. Left pg-gitlab-app/pg-praefect-app
missing entirely, crash-looping vps-gitlab-webservice on
ActiveRecord::DatabaseConnectionError for ~39h.
- infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml:
exclude config.php from the restic restore. The job's own comment
assumed nextcloud runs at replicas: 0 on the VPS; live confirms
replicas: 1 (no such key was ever actually set) - restic couldn't
overwrite the live pod's config.php (permission denied), failing the
whole job every run for 2+ days even though everything else restored
fine. Excluding it is correct regardless of permissions - a live
pod's own config.php shouldn't be overwritten by a background
restore job.
Also fixed live (not git-tracked, config.php is PVC-persisted runtime
state, not sourced from git):
- home + VPS nextcloud config.php dbpassword: out of sync with CNPG's
actual current pg-nextcloud-app password (baked in once at install,
never re-synced). VPS's case was two-layered - its own
pg-nextcloud-app secret also didn't match the real Postgres role
password, since VPS's pg-nextcloud is a read-only streaming replica
(spec.replica.enabled) and the authoritative password lives on home.
- grafana: broke a RollingUpdate deadlock (single-replica Deployment +
RWO Ceph volume - new pod couldn't start while the old pod still
held the only-one-node-at-a-time attachment, and the rollout
wouldn't scale down the old pod until the new one was ready).
- suspended the stale gitlab-mirror-sync CronJob on the VPS (spamming
auth failures every ~15min since its stored token predates the
Postgres-replication cutover of VPS GitLab's DB) rather than
deleting it, per the plan's own note to defer that until Phase 2b is
verified.
Root cause of the Vault OIDC 'Missing auth_url' error (and the same
symptom on every other VPS-standby app): the Ingress tls: block makes
HTTPS available but Traefik still serves plain HTTP on the same host too
- nothing was forcing the redirect. A browser landing on
http://vault.vps.huskypup.net gets a Vault UI that computes its OIDC
callback using window.location.origin (http://...), which doesn't match
the https:// entry in allowed_redirect_uris - Vault silently returns an
empty auth_url rather than an obviously-config-looking error. Confirmed
via a HAR capture of the actual failing browser request.
Adds a per-namespace Traefik Middleware (redirectScheme -> https,
permanent) referenced via router.middlewares on each Ingress. Applied
directly via kubectl first to test - ArgoCD's selfHeal immediately
reverted it since it wasn't in git yet, confirming the fix needs to ship
through the normal pipeline rather than live kubectl edits on ArgoCD-
managed VPS resources.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Nextcloud: PHP's postgres driver tries to look up a client cert at
$HOME/.postgresql/postgresql.crt (HOME=/root in this image) for
verify-ca/verify-full sslmodes - 'Permission denied' there aborts the
connection entirely with a confusing 'password authentication failed'
secondary error. Confirmed live: psql with the identical credentials
connects fine (uses sslmode=prefer, no cert lookup). PGSSLMODE=disable
sidesteps it - this is an intra-cluster connection, not worth TLS here.
n8n: chart's default liveness/readiness probe timing is tighter than
n8n needs to actually finish starting - kubelet was killing the
container (exitCode 143/SIGTERM) before it ever bound :5678, in an
endless crashloop. Copied home's more generous probe timing.
The DNS-flip watcher (scripts/vps-dns-failover/) was designed but never
actually installed on the VPS despite being tracked as done - real gap,
found when asked whether the standby services are actually reachable.
New design: instead of dynamically flipping *.kube.huskypup.net between
home and VPS IPs, give the VPS site its own permanent, always-resolving
subdomain - vault/gitea/auth/n8n/nextcloud.vps.huskypup.net, each with
real Ingress+TLS on the VPS's own Traefik+cert-manager (both already
installed by Phase 0 bootstrap, just never wired up). No token-scoping
decision needed since there's no dynamic flipping - reuses the same
cert-manager token pattern as home.
Also scales Authentik/n8n/Nextcloud from 0 to 1 replica on the VPS so
the replicated data is actually browsable at all times, not just
present-but-unreachable. Their CNPG clusters are still read-only
replicas (spec.replica.enabled: true) - writes will error until a
deliberate manual promotion, but reads/browsing work now. Vault and
Gitea were already running continuously.
Postgres refuses to replay WAL past a checkpoint recorded with higher
max_connections/max_wal_senders/etc than the recovering instance's own
settings: 'recovery aborted because of insufficient parameter settings:
max_connections = 100 is a lower setting than on the primary server,
where its value was 200' - confirmed live on all 3 VPS clusters (they
had no postgresql.parameters block at all, defaulting to CNPG's 100).
Copied home's full postgresql.parameters block to remove any other
potential mismatch (max_wal_senders, max_worker_processes are subject
to the same check). Bumped memory requests/limits to match home too -
shared_buffers: 512MB needs headroom the previous 256Mi/1Gi didn't have.
CNPG's barmanObjectStore.serverName defaults to the externalClusters[]
entry's own .name field ('home-backup', an arbitrary label I chose) -
NOT the actual source Postgres cluster's name. Home's backup: block
(on the real pg-authentik/pg-n8n/pg-nextcloud Clusters) defaults
serverName to its own metadata.name instead. Without an explicit
override these two disagree, so recovery searches the object store
under the wrong server-name prefix and finds nothing - 'no target
backup found', confirmed live even after the ScheduledBackup fix
produced real, completed base backups.
Same CNPG replica-cluster pattern as Authentik (continuous WAL replay
from home via VPS MinIO, app at 0 replicas until manual promotion - see
infrastructure/vps-standby/authentik/manifests/cnpg-cluster.yaml for the
full rationale).
n8n: N8N_ENCRYPTION_KEY copied byte-identical from home (kubectl, not
git) - decrypts stored credentials in the replicated DB, same reasoning
as Vault's unseal key / root token copies.
Nextcloud: adds infrastructure/vps-standby/nextcloud/manifests/
pvc-restore-cronjob.yaml, the read side of home's existing
nextcloud-pvc-sync restic backup - restores the latest snapshot from VPS
MinIO into this standby's PVC daily at 04:00 (2h after home's 02:00
backup). No Redis on the VPS side (no redis-operator deployed there,
out of scope for a standby that isn't serving traffic - Nextcloud
degrades gracefully to DB-based locking without it).
Scooby Husky