Fix VPS GitLab + Nextcloud crash loops found during health sweep

- infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml:
  creationPolicy Merge -> Owner (already applied live, committing to
  match). Merge assumed CNPG creates a base pg-gitlab-app/pg-praefect-app
  secret to merge a password into - wrong for these VPS Clusters since
  Phase 1c's replica-cluster recreate; CNPG doesn't create a local
  owner secret for a replicated role. Left pg-gitlab-app/pg-praefect-app
  missing entirely, crash-looping vps-gitlab-webservice on
  ActiveRecord::DatabaseConnectionError for ~39h.

- infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml:
  exclude config.php from the restic restore. The job's own comment
  assumed nextcloud runs at replicas: 0 on the VPS; live confirms
  replicas: 1 (no such key was ever actually set) - restic couldn't
  overwrite the live pod's config.php (permission denied), failing the
  whole job every run for 2+ days even though everything else restored
  fine. Excluding it is correct regardless of permissions - a live
  pod's own config.php shouldn't be overwritten by a background
  restore job.

Also fixed live (not git-tracked, config.php is PVC-persisted runtime
state, not sourced from git):
- home + VPS nextcloud config.php dbpassword: out of sync with CNPG's
  actual current pg-nextcloud-app password (baked in once at install,
  never re-synced). VPS's case was two-layered - its own
  pg-nextcloud-app secret also didn't match the real Postgres role
  password, since VPS's pg-nextcloud is a read-only streaming replica
  (spec.replica.enabled) and the authoritative password lives on home.
- grafana: broke a RollingUpdate deadlock (single-replica Deployment +
  RWO Ceph volume - new pod couldn't start while the old pod still
  held the only-one-node-at-a-time attachment, and the rollout
  wouldn't scale down the old pod until the new one was ready).
- suspended the stale gitlab-mirror-sync CronJob on the VPS (spamming
  auth failures every ~15min since its stored token predates the
  Postgres-replication cutover of VPS GitLab's DB) rather than
  deleting it, per the plan's own note to defer that until Phase 2b is
  verified.
This commit is contained in:
Scooby Husky
2026-08-22 15:54:05 -05:00
parent 9a9eb207b5
commit 779786262a
2 changed files with 38 additions and 9 deletions
@@ -2,10 +2,27 @@
# (apps/gitlab/manifests/ha-postgres-app-pushsecret.yaml has home's half).
# Pulls the authoritative app-role passwords back down here.
#
# Merge (not Owner/Replace): only overwrites the `password` key, leaving
# CNPG's own generated host/dbname/username fields intact - those are
# correctly LOCAL to each site (this secret's `host` key points at the
# VPS's own local -rw service, which CNPG itself still needs internally).
# creationPolicy REVERTED Merge -> Owner 2026-08-22: Merge assumed CNPG
# itself would generate a base pg-gitlab-app/pg-praefect-app secret
# (host/dbname/username keys) for this ExternalSecret to merge a
# password into - true for a normal owning/primary Cluster, but WRONG
# for these VPS Clusters since Phase 1c's recreate: `pg-gitlab`/
# `pg-praefect` here are bootstrap.recovery REPLICA clusters
# (replica.self: vps) - the `app` role is part of the replicated
# global role/password state streamed from home's primary, so CNPG
# correctly never creates a local owner secret for it here. Merge had
# nothing to merge onto -> ExternalSecret condition
# "SecretMissing/CreationPolicy=Merge" -> pg-gitlab-app never existed
# -> vps-gitlab-webservice crash-looped on
# ActiveRecord::DatabaseConnectionError for ~39h before this was
# caught (confirmed live, unrelated to any same-day change). Owner:
# the ExternalSecret creates the whole secret itself. This chart only
# ever reads the `password` key from it - host/port/dbname/username
# are hardcoded directly in infrastructure/vps-standby/gitlab/
# values.yaml's global.psql/global.praefect.psql blocks, not read from
# this secret's other keys, so a password-only Owner-created secret is
# sufficient (confirmed live: no other consumer of this secret's keys
# found).
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
@@ -18,7 +35,7 @@ spec:
kind: ClusterSecretStore
target:
name: pg-gitlab-app
creationPolicy: Merge
creationPolicy: Owner
data:
- secretKey: password
remoteRef:
@@ -37,7 +54,7 @@ spec:
kind: ClusterSecretStore
target:
name: pg-praefect-app
creationPolicy: Merge
creationPolicy: Owner
data:
- secretKey: password
remoteRef:
@@ -16,8 +16,20 @@
# kubectl -n nextcloud create secret generic nextcloud-restic-password \
# --from-literal=password=<home's nextcloud-restic-password>
#
# Nextcloud itself isn't running (replicas: 0, see values.yaml) so there's
# no live-write conflict risk overwriting the PVC on every run.
# CORRECTED 2026-08-22: the "replicas: 0" assumption below was wrong -
# confirmed live vps-nextcloud actually runs continuously at replicas: 1
# (no replicas key was ever set in values.yaml; the chart's own default
# applies). This job had been silently failing every run for 2+ days as
# a result: restic couldn't overwrite the live pod's config.php (mode
# 0640, owned by www-data/33; this job runs as UID 65534 with only
# group 33 via fsGroup, so it can read but not write that one file) -
# "ignoring error for /data/config/config.php: permission denied",
# Fatal: 1 error, even though the other 31000+ files/15GiB restored
# fine every time. Fixed by excluding config.php from the restore
# below rather than chasing permissions - overwriting a live, running
# app's own config.php from a background restore job is the wrong
# move regardless (it holds this site's actual current db credentials,
# etc.), not just a permissions bug to route around.
apiVersion: batch/v1
kind: CronJob
metadata:
@@ -64,7 +76,7 @@ spec:
fi
echo "==> Restoring latest snapshot into /data..."
restic restore latest --tag nextcloud --host nextcloud-k8s --target /
restic restore latest --tag nextcloud --host nextcloud-k8s --target / --exclude /data/config/config.php
echo "==> Done."
env: