mirror of
https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
synced 2026-08-23 12:56:46 +00:00
Fix VPS GitLab + Nextcloud crash loops found during health sweep
- infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml: creationPolicy Merge -> Owner (already applied live, committing to match). Merge assumed CNPG creates a base pg-gitlab-app/pg-praefect-app secret to merge a password into - wrong for these VPS Clusters since Phase 1c's replica-cluster recreate; CNPG doesn't create a local owner secret for a replicated role. Left pg-gitlab-app/pg-praefect-app missing entirely, crash-looping vps-gitlab-webservice on ActiveRecord::DatabaseConnectionError for ~39h. - infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml: exclude config.php from the restic restore. The job's own comment assumed nextcloud runs at replicas: 0 on the VPS; live confirms replicas: 1 (no such key was ever actually set) - restic couldn't overwrite the live pod's config.php (permission denied), failing the whole job every run for 2+ days even though everything else restored fine. Excluding it is correct regardless of permissions - a live pod's own config.php shouldn't be overwritten by a background restore job. Also fixed live (not git-tracked, config.php is PVC-persisted runtime state, not sourced from git): - home + VPS nextcloud config.php dbpassword: out of sync with CNPG's actual current pg-nextcloud-app password (baked in once at install, never re-synced). VPS's case was two-layered - its own pg-nextcloud-app secret also didn't match the real Postgres role password, since VPS's pg-nextcloud is a read-only streaming replica (spec.replica.enabled) and the authoritative password lives on home. - grafana: broke a RollingUpdate deadlock (single-replica Deployment + RWO Ceph volume - new pod couldn't start while the old pod still held the only-one-node-at-a-time attachment, and the rollout wouldn't scale down the old pod until the new one was ready). - suspended the stale gitlab-mirror-sync CronJob on the VPS (spamming auth failures every ~15min since its stored token predates the Postgres-replication cutover of VPS GitLab's DB) rather than deleting it, per the plan's own note to defer that until Phase 2b is verified.
This commit is contained in:
@@ -2,10 +2,27 @@
|
||||
# (apps/gitlab/manifests/ha-postgres-app-pushsecret.yaml has home's half).
|
||||
# Pulls the authoritative app-role passwords back down here.
|
||||
#
|
||||
# Merge (not Owner/Replace): only overwrites the `password` key, leaving
|
||||
# CNPG's own generated host/dbname/username fields intact - those are
|
||||
# correctly LOCAL to each site (this secret's `host` key points at the
|
||||
# VPS's own local -rw service, which CNPG itself still needs internally).
|
||||
# creationPolicy REVERTED Merge -> Owner 2026-08-22: Merge assumed CNPG
|
||||
# itself would generate a base pg-gitlab-app/pg-praefect-app secret
|
||||
# (host/dbname/username keys) for this ExternalSecret to merge a
|
||||
# password into - true for a normal owning/primary Cluster, but WRONG
|
||||
# for these VPS Clusters since Phase 1c's recreate: `pg-gitlab`/
|
||||
# `pg-praefect` here are bootstrap.recovery REPLICA clusters
|
||||
# (replica.self: vps) - the `app` role is part of the replicated
|
||||
# global role/password state streamed from home's primary, so CNPG
|
||||
# correctly never creates a local owner secret for it here. Merge had
|
||||
# nothing to merge onto -> ExternalSecret condition
|
||||
# "SecretMissing/CreationPolicy=Merge" -> pg-gitlab-app never existed
|
||||
# -> vps-gitlab-webservice crash-looped on
|
||||
# ActiveRecord::DatabaseConnectionError for ~39h before this was
|
||||
# caught (confirmed live, unrelated to any same-day change). Owner:
|
||||
# the ExternalSecret creates the whole secret itself. This chart only
|
||||
# ever reads the `password` key from it - host/port/dbname/username
|
||||
# are hardcoded directly in infrastructure/vps-standby/gitlab/
|
||||
# values.yaml's global.psql/global.praefect.psql blocks, not read from
|
||||
# this secret's other keys, so a password-only Owner-created secret is
|
||||
# sufficient (confirmed live: no other consumer of this secret's keys
|
||||
# found).
|
||||
apiVersion: external-secrets.io/v1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
@@ -18,7 +35,7 @@ spec:
|
||||
kind: ClusterSecretStore
|
||||
target:
|
||||
name: pg-gitlab-app
|
||||
creationPolicy: Merge
|
||||
creationPolicy: Owner
|
||||
data:
|
||||
- secretKey: password
|
||||
remoteRef:
|
||||
@@ -37,7 +54,7 @@ spec:
|
||||
kind: ClusterSecretStore
|
||||
target:
|
||||
name: pg-praefect-app
|
||||
creationPolicy: Merge
|
||||
creationPolicy: Owner
|
||||
data:
|
||||
- secretKey: password
|
||||
remoteRef:
|
||||
|
||||
Reference in New Issue
Block a user