Files
Homelabv4/infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml
T
Scooby Husky 779786262a Fix VPS GitLab + Nextcloud crash loops found during health sweep
- infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml:
  creationPolicy Merge -> Owner (already applied live, committing to
  match). Merge assumed CNPG creates a base pg-gitlab-app/pg-praefect-app
  secret to merge a password into - wrong for these VPS Clusters since
  Phase 1c's replica-cluster recreate; CNPG doesn't create a local
  owner secret for a replicated role. Left pg-gitlab-app/pg-praefect-app
  missing entirely, crash-looping vps-gitlab-webservice on
  ActiveRecord::DatabaseConnectionError for ~39h.

- infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml:
  exclude config.php from the restic restore. The job's own comment
  assumed nextcloud runs at replicas: 0 on the VPS; live confirms
  replicas: 1 (no such key was ever actually set) - restic couldn't
  overwrite the live pod's config.php (permission denied), failing the
  whole job every run for 2+ days even though everything else restored
  fine. Excluding it is correct regardless of permissions - a live
  pod's own config.php shouldn't be overwritten by a background
  restore job.

Also fixed live (not git-tracked, config.php is PVC-persisted runtime
state, not sourced from git):
- home + VPS nextcloud config.php dbpassword: out of sync with CNPG's
  actual current pg-nextcloud-app password (baked in once at install,
  never re-synced). VPS's case was two-layered - its own
  pg-nextcloud-app secret also didn't match the real Postgres role
  password, since VPS's pg-nextcloud is a read-only streaming replica
  (spec.replica.enabled) and the authoritative password lives on home.
- grafana: broke a RollingUpdate deadlock (single-replica Deployment +
  RWO Ceph volume - new pod couldn't start while the old pod still
  held the only-one-node-at-a-time attachment, and the rollout
  wouldn't scale down the old pod until the new one was ready).
- suspended the stale gitlab-mirror-sync CronJob on the VPS (spamming
  auth failures every ~15min since its stored token predates the
  Postgres-replication cutover of VPS GitLab's DB) rather than
  deleting it, per the plan's own note to defer that until Phase 2b is
  verified.
2026-08-22 15:54:05 -05:00

63 lines
2.3 KiB
YAML

# GitLab cross-site replication - VPS's half of the Vault pipeline
# (apps/gitlab/manifests/ha-postgres-app-pushsecret.yaml has home's half).
# Pulls the authoritative app-role passwords back down here.
#
# creationPolicy REVERTED Merge -> Owner 2026-08-22: Merge assumed CNPG
# itself would generate a base pg-gitlab-app/pg-praefect-app secret
# (host/dbname/username keys) for this ExternalSecret to merge a
# password into - true for a normal owning/primary Cluster, but WRONG
# for these VPS Clusters since Phase 1c's recreate: `pg-gitlab`/
# `pg-praefect` here are bootstrap.recovery REPLICA clusters
# (replica.self: vps) - the `app` role is part of the replicated
# global role/password state streamed from home's primary, so CNPG
# correctly never creates a local owner secret for it here. Merge had
# nothing to merge onto -> ExternalSecret condition
# "SecretMissing/CreationPolicy=Merge" -> pg-gitlab-app never existed
# -> vps-gitlab-webservice crash-looped on
# ActiveRecord::DatabaseConnectionError for ~39h before this was
# caught (confirmed live, unrelated to any same-day change). Owner:
# the ExternalSecret creates the whole secret itself. This chart only
# ever reads the `password` key from it - host/port/dbname/username
# are hardcoded directly in infrastructure/vps-standby/gitlab/
# values.yaml's global.psql/global.praefect.psql blocks, not read from
# this secret's other keys, so a password-only Owner-created secret is
# sufficient (confirmed live: no other consumer of this secret's keys
# found).
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: pg-gitlab-app-password
namespace: gitlab
spec:
refreshInterval: 5m
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: pg-gitlab-app
creationPolicy: Owner
data:
- secretKey: password
remoteRef:
key: vps/pg-gitlab-app
property: password
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: pg-praefect-app-password
namespace: gitlab
spec:
refreshInterval: 5m
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: pg-praefect-app
creationPolicy: Owner
data:
- secretKey: password
remoteRef:
key: vps/pg-praefect-app
property: password