From 779786262aefc936d725f216b8fbdcbab359328c Mon Sep 17 00:00:00 2001 From: Scooby Husky Date: Sat, 22 Aug 2026 15:54:05 -0500 Subject: [PATCH] Fix VPS GitLab + Nextcloud crash loops found during health sweep - infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml: creationPolicy Merge -> Owner (already applied live, committing to match). Merge assumed CNPG creates a base pg-gitlab-app/pg-praefect-app secret to merge a password into - wrong for these VPS Clusters since Phase 1c's replica-cluster recreate; CNPG doesn't create a local owner secret for a replicated role. Left pg-gitlab-app/pg-praefect-app missing entirely, crash-looping vps-gitlab-webservice on ActiveRecord::DatabaseConnectionError for ~39h. - infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml: exclude config.php from the restic restore. The job's own comment assumed nextcloud runs at replicas: 0 on the VPS; live confirms replicas: 1 (no such key was ever actually set) - restic couldn't overwrite the live pod's config.php (permission denied), failing the whole job every run for 2+ days even though everything else restored fine. Excluding it is correct regardless of permissions - a live pod's own config.php shouldn't be overwritten by a background restore job. Also fixed live (not git-tracked, config.php is PVC-persisted runtime state, not sourced from git): - home + VPS nextcloud config.php dbpassword: out of sync with CNPG's actual current pg-nextcloud-app password (baked in once at install, never re-synced). VPS's case was two-layered - its own pg-nextcloud-app secret also didn't match the real Postgres role password, since VPS's pg-nextcloud is a read-only streaming replica (spec.replica.enabled) and the authoritative password lives on home. - grafana: broke a RollingUpdate deadlock (single-replica Deployment + RWO Ceph volume - new pod couldn't start while the old pod still held the only-one-node-at-a-time attachment, and the rollout wouldn't scale down the old pod until the new one was ready). - suspended the stale gitlab-mirror-sync CronJob on the VPS (spamming auth failures every ~15min since its stored token predates the Postgres-replication cutover of VPS GitLab's DB) rather than deleting it, per the plan's own note to defer that until Phase 2b is verified. --- .../ha-postgres-app-externalsecret.yaml | 29 +++++++++++++++---- .../manifests/pvc-restore-cronjob.yaml | 18 ++++++++++-- 2 files changed, 38 insertions(+), 9 deletions(-) diff --git a/infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml b/infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml index c833806..8b827a0 100644 --- a/infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml +++ b/infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml @@ -2,10 +2,27 @@ # (apps/gitlab/manifests/ha-postgres-app-pushsecret.yaml has home's half). # Pulls the authoritative app-role passwords back down here. # -# Merge (not Owner/Replace): only overwrites the `password` key, leaving -# CNPG's own generated host/dbname/username fields intact - those are -# correctly LOCAL to each site (this secret's `host` key points at the -# VPS's own local -rw service, which CNPG itself still needs internally). +# creationPolicy REVERTED Merge -> Owner 2026-08-22: Merge assumed CNPG +# itself would generate a base pg-gitlab-app/pg-praefect-app secret +# (host/dbname/username keys) for this ExternalSecret to merge a +# password into - true for a normal owning/primary Cluster, but WRONG +# for these VPS Clusters since Phase 1c's recreate: `pg-gitlab`/ +# `pg-praefect` here are bootstrap.recovery REPLICA clusters +# (replica.self: vps) - the `app` role is part of the replicated +# global role/password state streamed from home's primary, so CNPG +# correctly never creates a local owner secret for it here. Merge had +# nothing to merge onto -> ExternalSecret condition +# "SecretMissing/CreationPolicy=Merge" -> pg-gitlab-app never existed +# -> vps-gitlab-webservice crash-looped on +# ActiveRecord::DatabaseConnectionError for ~39h before this was +# caught (confirmed live, unrelated to any same-day change). Owner: +# the ExternalSecret creates the whole secret itself. This chart only +# ever reads the `password` key from it - host/port/dbname/username +# are hardcoded directly in infrastructure/vps-standby/gitlab/ +# values.yaml's global.psql/global.praefect.psql blocks, not read from +# this secret's other keys, so a password-only Owner-created secret is +# sufficient (confirmed live: no other consumer of this secret's keys +# found). apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: @@ -18,7 +35,7 @@ spec: kind: ClusterSecretStore target: name: pg-gitlab-app - creationPolicy: Merge + creationPolicy: Owner data: - secretKey: password remoteRef: @@ -37,7 +54,7 @@ spec: kind: ClusterSecretStore target: name: pg-praefect-app - creationPolicy: Merge + creationPolicy: Owner data: - secretKey: password remoteRef: diff --git a/infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml b/infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml index 3c5caf0..e06e276 100644 --- a/infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml +++ b/infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml @@ -16,8 +16,20 @@ # kubectl -n nextcloud create secret generic nextcloud-restic-password \ # --from-literal=password= # -# Nextcloud itself isn't running (replicas: 0, see values.yaml) so there's -# no live-write conflict risk overwriting the PVC on every run. +# CORRECTED 2026-08-22: the "replicas: 0" assumption below was wrong - +# confirmed live vps-nextcloud actually runs continuously at replicas: 1 +# (no replicas key was ever set in values.yaml; the chart's own default +# applies). This job had been silently failing every run for 2+ days as +# a result: restic couldn't overwrite the live pod's config.php (mode +# 0640, owned by www-data/33; this job runs as UID 65534 with only +# group 33 via fsGroup, so it can read but not write that one file) - +# "ignoring error for /data/config/config.php: permission denied", +# Fatal: 1 error, even though the other 31000+ files/15GiB restored +# fine every time. Fixed by excluding config.php from the restore +# below rather than chasing permissions - overwriting a live, running +# app's own config.php from a background restore job is the wrong +# move regardless (it holds this site's actual current db credentials, +# etc.), not just a permissions bug to route around. apiVersion: batch/v1 kind: CronJob metadata: @@ -64,7 +76,7 @@ spec: fi echo "==> Restoring latest snapshot into /data..." - restic restore latest --tag nextcloud --host nextcloud-k8s --target / + restic restore latest --tag nextcloud --host nextcloud-k8s --target / --exclude /data/config/config.php echo "==> Done." env: