Fix VPS GitLab + Nextcloud crash loops found during health sweep

- infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml:
  creationPolicy Merge -> Owner (already applied live, committing to
  match). Merge assumed CNPG creates a base pg-gitlab-app/pg-praefect-app
  secret to merge a password into - wrong for these VPS Clusters since
  Phase 1c's replica-cluster recreate; CNPG doesn't create a local
  owner secret for a replicated role. Left pg-gitlab-app/pg-praefect-app
  missing entirely, crash-looping vps-gitlab-webservice on
  ActiveRecord::DatabaseConnectionError for ~39h.

- infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml:
  exclude config.php from the restic restore. The job's own comment
  assumed nextcloud runs at replicas: 0 on the VPS; live confirms
  replicas: 1 (no such key was ever actually set) - restic couldn't
  overwrite the live pod's config.php (permission denied), failing the
  whole job every run for 2+ days even though everything else restored
  fine. Excluding it is correct regardless of permissions - a live
  pod's own config.php shouldn't be overwritten by a background
  restore job.

Also fixed live (not git-tracked, config.php is PVC-persisted runtime
state, not sourced from git):
- home + VPS nextcloud config.php dbpassword: out of sync with CNPG's
  actual current pg-nextcloud-app password (baked in once at install,
  never re-synced). VPS's case was two-layered - its own
  pg-nextcloud-app secret also didn't match the real Postgres role
  password, since VPS's pg-nextcloud is a read-only streaming replica
  (spec.replica.enabled) and the authoritative password lives on home.
- grafana: broke a RollingUpdate deadlock (single-replica Deployment +
  RWO Ceph volume - new pod couldn't start while the old pod still
  held the only-one-node-at-a-time attachment, and the rollout
  wouldn't scale down the old pod until the new one was ready).
- suspended the stale gitlab-mirror-sync CronJob on the VPS (spamming
  auth failures every ~15min since its stored token predates the
  Postgres-replication cutover of VPS GitLab's DB) rather than
  deleting it, per the plan's own note to defer that until Phase 2b is
  verified.

apps/gitlab/manifests/cnpg-cluster.yaml

@@ -53,17 +53,25 @@ spec:
       max_parallel_workers_per_gather: "2"
       max_parallel_workers: "4"
 
-  # Backup configuration to MinIO
+  # Backup configuration - repointed to VPS MinIO 2026-08-22 (GitLab
+  # cross-site replication plan, see
+  # /home/scooby/.claude/plans/jiggly-snacking-iverson.md's "GitLab True
+  # Cross-Site Replication" section). Was gitlab-minio-svc (in-cluster,
+  # same Ceph the backup is meant to protect against). Also doubles as
+  # the initial-seed source for the VPS's own pg-gitlab replica
+  # (bootstrap.recovery, Phase 1c) - same barmanObjectStore block serves
+  # both purposes, matching the Authentik precedent
+  # (infrastructure/authentik/manifests/cnpg-cluster.yaml).
   backup:
     barmanObjectStore:
-      destinationPath: s3://gitlab-backups/pg-gitlab
-      endpointURL: http://gitlab-minio-svc.gitlab.svc.cluster.local:9000
+      destinationPath: s3://cnpg-backups/pg-gitlab
+      endpointURL: http://100.108.113.41:9000
       s3Credentials:
         accessKeyId:
-          name: gitlab-minio-secret
+          name: vps-minio-secret
           key: accesskey
         secretAccessKey:
-          name: gitlab-minio-secret
+          name: vps-minio-secret
           key: secretkey
       wal:
         compression: gzip
@@ -72,3 +80,58 @@ spec:
 
   monitoring:
     enablePodMonitor: true
+
+  # GitLab cross-site replication Phase 1c stage 2 (see
+  # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - streaming
+  # replication to/from the VPS's pg-gitlab cluster, alongside (not
+  # replacing) the barmanObjectStore backup above. Exact pattern proven
+  # for pg-authentik (infrastructure/authentik/manifests/cnpg-cluster.yaml).
+  #
+  # Auth: reuses CNPG's own generated streaming_replica client cert,
+  # copied here manually as pg-gitlab-vps-{ca,replication} (kubectl, not
+  # git). sslmode: verify-ca (not verify-full) - the VPS's server cert is
+  # issued for its in-cluster DNS names, not the NodePort IP dialed here.
+  #
+  # Reachable via 100.108.113.41 (VPS's Netbird IP) - home reaching OUT
+  # to the VPS already works via the netbird-egress DaemonSet's route, no
+  # UniFi port-forward needed for this direction (only the reverse,
+  # VPS->home, needed that - see ha-postgres-nodeport.yaml).
+  externalClusters:
+    - name: home
+      connectionParameters:
+        host: pg-gitlab-rw
+        port: "5432"
+        dbname: gitlabhq_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-gitlab-replication
+        key: tls.crt
+      sslKey:
+        name: pg-gitlab-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-gitlab-ca
+        key: ca.crt
+    - name: vps
+      connectionParameters:
+        host: 100.108.113.41
+        port: "32444"
+        dbname: gitlabhq_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-gitlab-vps-replication
+        key: tls.crt
+      sslKey:
+        name: pg-gitlab-vps-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-gitlab-vps-ca
+        key: ca.crt
+
+  # Distributed topology: both sides agree home is primary today.
+  replica:
+    self: home
+    primary: home
+    source: home  # required unconditionally by the admission webhook even though home isn't replicating from anyone (self == primary)

apps/gitlab/manifests/gitaly-tls-certificate.yaml

@@ -0,0 +1,55 @@
+# GitLab cross-site replication Phase 2b (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - Gitaly's gRPC
+# has no TLS by default; routing it over the public UniFi WAN forward
+# (needed because the direct-Netbird-IP path doesn't accept inbound
+# connections to home, confirmed live the same way as the existing etcd
+# quorum) would otherwise carry the gitaly_token, gitlab_shell_secret,
+# and raw repository contents in plaintext.
+#
+# Real Let's Encrypt cert via the same DNS-01 ClusterIssuer already
+# proven for every other *.huskypup.net cert in this cluster, rather
+# than a self-signed cert - sidesteps having to distribute a custom CA
+# to both home and the VPS (which already trust Let's Encrypt's public
+# CA by default). Covers all 4 Gitaly floating hostnames as SANs - one
+# shared cert, referenced by every node's tlsSecretName (or the global
+# default), simpler than per-node certs.
+#
+# secretName matches the chart's own default (confirmed live via
+# gitlab/templates/_gitaly.tpl's gitlab.gitaly.tls.secret helper:
+# "<release-name>-gitaly-tls" = "gitlab-gitaly-tls") - global.gitaly.tls.
+# enabled: true picks this up with no secretName override needed.
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: gitlab-gitaly-tls
+  namespace: gitlab
+spec:
+  secretName: gitlab-gitaly-tls
+  issuerRef:
+    name: letsencrypt-production
+    kind: ClusterIssuer
+  dnsNames:
+    - gitaly-0.ha.huskypup.net
+    - gitaly-1.ha.huskypup.net
+    - gitaly-2.ha.huskypup.net
+    - gitaly-vps.ha.huskypup.net
+---
+# Separate cert/secret for workhorse's internal-API TLS listener (port
+# 8181, [gitlab] url every Gitaly node calls for gitlab-shell hooks) -
+# different secretName (gitlab-workhorse-tls, chart default per
+# gitlab/charts/gitlab/charts/webservice/templates/_helpers.tpl's
+# workhorse.tls.secret helper) than Gitaly's own cert, kept separate
+# rather than trying to share one Secret object across two different
+# chart-expected names.
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: gitlab-workhorse-tls
+  namespace: gitlab
+spec:
+  secretName: gitlab-workhorse-tls
+  issuerRef:
+    name: letsencrypt-production
+    kind: ClusterIssuer
+  dnsNames:
+    - gitlab-internal-api.ha.huskypup.net

apps/gitlab/manifests/ha-gitaly-nodeport.yaml

@@ -0,0 +1,61 @@
+---
+# GitLab cross-site replication Phase 2b (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - exposes each
+# of home's 3 existing Gitaly nodes individually to the VPS, so Praefect
+# can register the VPS as a 4th node in the SAME virtual storage and
+# inter-Gitaly replication RPCs can reach each node directly. Per-pod
+# (not per-StatefulSet), unlike the chart's own gitlab-gitaly-default
+# Service - Praefect/Gitaly must address each node individually, not
+# load-balance across them.
+#
+# Reached over the same public UniFi WAN forward already used for
+# Postgres, NOT the Netbird mesh directly - confirmed live the direct-
+# Netbird-IP path doesn't accept inbound connections to home (same gap
+# already present for the existing etcd quorum; the earlier assumption
+# that NodePort binds on a reachable Netbird interface was wrong).
+# Targets Gitaly's TLS port (8076), not the plaintext one (8075) -
+# Gitaly's gRPC has no built-in transport security, and this WAN hop
+# would otherwise carry the gitaly_token, gitlab_shell_secret, and raw
+# repository contents in plaintext. See gitaly-tls-certificate.yaml and
+# global.gitaly.tls.enabled in values.yaml.
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-gitaly-0
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    statefulset.kubernetes.io/pod-name: gitlab-gitaly-default-0
+  ports:
+    - port: 8076
+      targetPort: 8076
+      nodePort: 32446
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-gitaly-1
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    statefulset.kubernetes.io/pod-name: gitlab-gitaly-default-1
+  ports:
+    - port: 8076
+      targetPort: 8076
+      nodePort: 32447
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-gitaly-2
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    statefulset.kubernetes.io/pod-name: gitlab-gitaly-default-2
+  ports:
+    - port: 8076
+      targetPort: 8076
+      nodePort: 32448

apps/gitlab/manifests/ha-gitlab-internal-api-nodeport.yaml

@@ -0,0 +1,32 @@
+# GitLab cross-site replication Phase 2b - exposes home's Rails internal
+# API (port 8181, a dedicated internal-only listener, deliberately NOT
+# exposed through the public Istio ingress on 8080) to the VPS's Gitaly
+# node. Gitaly's gitlab-shell hooks call POST /api/v4/internal/{allowed,
+# pre_receive,post_receive} against this whenever the VPS node is
+# primary for a repository (per-repository election can cause this even
+# in normal operation) or after a real failover.
+#
+# Reached over the same public UniFi WAN forward already used for
+# Postgres/Gitaly, not the Netbird mesh directly (confirmed live that
+# path doesn't accept inbound connections to home). Port stays 8181
+# (unlike Gitaly's separate 8075/8076 split) - global.workhorse.tls.
+# enabled in values.yaml replaces the plaintext listener with TLS on
+# this SAME port rather than adding a second one. Carries the
+# gitlab_shell_secret in a header, so this needed TLS just as much as
+# Gitaly's gRPC did - see gitaly-tls-certificate.yaml's second
+# Certificate (gitlab-workhorse-tls) and the
+# gitlab-internal-api.ha.huskypup.net CoreDNS rewrite.
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-gitlab-internal-api
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    app: webservice
+    release: gitlab
+  ports:
+    - port: 8181
+      targetPort: 8181
+      nodePort: 32449

apps/gitlab/manifests/ha-gitlab-secrets-pushsecret.yaml

@@ -0,0 +1,99 @@
+# GitLab cross-site replication Phase 1b (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - pushes home's
+# chart-generated secrets into Vault so the VPS's ESO can pull them back
+# down BEFORE the VPS's Postgres becomes a real replica of home's.
+#
+# Why this matters: the GitLab chart generates these per-release, at
+# random, on first install. Home's and the VPS's values are currently
+# different. Once the VPS's Postgres is byte-for-byte replicating home's
+# rows, the VPS Rails app will be reading home's encrypted-at-rest data
+# (2FA secrets, CI/CD variables, integration tokens, PAT digests,
+# application_settings encrypted columns) with the WRONG key unless these
+# match - Rails raises OpenSSL::Cipher::CipherError on boot otherwise.
+#
+# gitlab-gitaly-secret/gitlab-praefect-secret are also needed for
+# Phase 2's cross-site gRPC auth (the VPS's Gitaly/Praefect need the same
+# tokens home's do, to be trusted as nodes of the same virtual storage).
+#
+# Same pattern as infrastructure/authentik/manifests/ha-postgres-app-pushsecret.yaml.
+# Each of these secrets happens to have exactly one key (confirmed live),
+# so one PushSecret + one ExternalSecret per secret, no enumeration needed.
+apiVersion: external-secrets.io/v1alpha1
+kind: PushSecret
+metadata:
+  name: gitlab-rails-secret-to-vps
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRefs:
+    - name: vault-backend
+      kind: ClusterSecretStore
+  selector:
+    secret:
+      name: gitlab-rails-secret
+  data:
+    - match:
+        secretKey: secrets.yml
+        remoteRef:
+          remoteKey: vps/gitlab-rails-secret
+          property: secrets.yml
+---
+apiVersion: external-secrets.io/v1alpha1
+kind: PushSecret
+metadata:
+  name: gitlab-shell-secret-to-vps
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRefs:
+    - name: vault-backend
+      kind: ClusterSecretStore
+  selector:
+    secret:
+      name: gitlab-gitlab-shell-secret
+  data:
+    - match:
+        secretKey: secret
+        remoteRef:
+          remoteKey: vps/gitlab-gitlab-shell-secret
+          property: secret
+---
+apiVersion: external-secrets.io/v1alpha1
+kind: PushSecret
+metadata:
+  name: gitlab-gitaly-secret-to-vps
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRefs:
+    - name: vault-backend
+      kind: ClusterSecretStore
+  selector:
+    secret:
+      name: gitlab-gitaly-secret
+  data:
+    - match:
+        secretKey: token
+        remoteRef:
+          remoteKey: vps/gitlab-gitaly-secret
+          property: token
+---
+apiVersion: external-secrets.io/v1alpha1
+kind: PushSecret
+metadata:
+  name: gitlab-praefect-secret-to-vps
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRefs:
+    - name: vault-backend
+      kind: ClusterSecretStore
+  selector:
+    secret:
+      name: gitlab-praefect-secret
+  data:
+    - match:
+        secretKey: token
+        remoteRef:
+          remoteKey: vps/gitlab-praefect-secret
+          property: token

apps/gitlab/manifests/ha-peerauth.yaml

@@ -0,0 +1,90 @@
+# GitLab cross-site replication (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - direct copy of
+# infrastructure/authentik/manifests/ha-postgres-peerauth.yaml's proven
+# fix. The gitlab namespace carries the same ambient-mesh STRICT default
+# as authentik; without this, ztunnel resets the VPS/witness's connection
+# (no mesh identity - they're not in this cluster) before the Postgres
+# TLS handshake can even start, surfacing as "server closed the
+# connection unexpectedly" on the replica side despite the NodePort/
+# firewall path being entirely correct.
+#
+# Scoped to just each CNPG primary pod's ports via portLevelMtls, not the
+# whole namespace - GitLab's own in-mesh east-west traffic (webservice
+# <-> gitaly <-> redis <-> sidekiq <-> praefect) stays STRICT.
+apiVersion: security.istio.io/v1
+kind: PeerAuthentication
+metadata:
+  name: allow-ha-gitlab-postgres-replication
+  namespace: gitlab
+spec:
+  selector:
+    matchLabels:
+      cnpg.io/cluster: pg-gitlab
+  mtls:
+    mode: STRICT
+  portLevelMtls:
+    "5432":
+      mode: PERMISSIVE
+    # Floating-hostname port (see ha-postgres-nodeport.yaml) - needs its
+    # own entry, ambient's port-level mTLS enforcement is keyed on the
+    # port actually dialed (61442), not just the pod's real containerPort
+    # (5432) it eventually reaches (confirmed live for the authentik
+    # precedent, same mechanism here).
+    "61442":
+      mode: PERMISSIVE
+---
+apiVersion: security.istio.io/v1
+kind: PeerAuthentication
+metadata:
+  name: allow-ha-praefect-postgres-replication
+  namespace: gitlab
+spec:
+  selector:
+    matchLabels:
+      cnpg.io/cluster: pg-praefect
+  mtls:
+    mode: STRICT
+  portLevelMtls:
+    "5432":
+      mode: PERMISSIVE
+    "61443":
+      mode: PERMISSIVE
+---
+# Phase 2b (Gitaly cross-site replication) - the 3 Gitaly pods, exposed
+# individually via ha-gitaly-nodeport.yaml for the VPS's 4th node to
+# join the same virtual storage. Gitaly pods already carry
+# istio.io/use-waypoint: none (chart default, gRPC doesn't tolerate L7
+# waypoint processing well) but that only skips L7 processing - L4
+# ztunnel mTLS enforcement is a separate concern, same fix needed as the
+# Postgres precedent.
+apiVersion: security.istio.io/v1
+kind: PeerAuthentication
+metadata:
+  name: allow-ha-gitaly-replication
+  namespace: gitlab
+spec:
+  selector:
+    matchLabels:
+      app: gitaly
+  mtls:
+    mode: STRICT
+  portLevelMtls:
+    "8075":
+      mode: PERMISSIVE
+---
+# Phase 2b - the VPS's Gitaly node calling home's Rails internal API
+# (ha-gitlab-internal-api-nodeport.yaml) for gitlab-shell hooks.
+apiVersion: security.istio.io/v1
+kind: PeerAuthentication
+metadata:
+  name: allow-ha-gitlab-internal-api
+  namespace: gitlab
+spec:
+  selector:
+    matchLabels:
+      app: webservice
+  mtls:
+    mode: STRICT
+  portLevelMtls:
+    "8181":
+      mode: PERMISSIVE

apps/gitlab/manifests/ha-postgres-app-pushsecret.yaml

@@ -0,0 +1,50 @@
+# GitLab cross-site replication (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - same pattern
+# and same reasoning as
+# infrastructure/authentik/manifests/ha-postgres-app-pushsecret.yaml:
+# pg-gitlab-app/pg-praefect-app are CNPG-generated, not Vault-native -
+# each cluster independently generated its own random `app` role
+# password at bootstrap time. Streaming replication keeps the two
+# clusters' actual live passwords in sync (WAL includes role/password
+# changes), but the two clusters' K8s Secret OBJECTS never resync on
+# their own - this closes that gap, self-healing on refreshInterval
+# instead of a one-time fix that goes stale on the next rotation.
+apiVersion: external-secrets.io/v1alpha1
+kind: PushSecret
+metadata:
+  name: pg-gitlab-app-to-vps
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRefs:
+    - name: vault-backend
+      kind: ClusterSecretStore
+  selector:
+    secret:
+      name: pg-gitlab-app
+  data:
+    - match:
+        secretKey: password
+        remoteRef:
+          remoteKey: vps/pg-gitlab-app
+          property: password
+---
+apiVersion: external-secrets.io/v1alpha1
+kind: PushSecret
+metadata:
+  name: pg-praefect-app-to-vps
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRefs:
+    - name: vault-backend
+      kind: ClusterSecretStore
+  selector:
+    secret:
+      name: pg-praefect-app
+  data:
+    - match:
+        secretKey: password
+        remoteRef:
+          remoteKey: vps/pg-praefect-app
+          property: password

apps/gitlab/manifests/ha-postgres-nodeport.yaml

@@ -0,0 +1,56 @@
+---
+# GitLab cross-site replication (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md's "GitLab True
+# Cross-Site Replication" section) - direct copy of the proven pattern
+# from infrastructure/authentik/manifests/ha-postgres-nodeport.yaml.
+# Exposes pg-gitlab's and pg-praefect's current primaries to the
+# VPS/witness over the Netbird mesh for streaming replication - same
+# selectors CNPG's own -rw ClusterIP Services use, just NodePort instead,
+# since CNPG doesn't manage these objects and won't fight with them.
+# Needs infrastructure/kyverno/policies/ha-failover-nodeport-exception.yaml
+# (name pattern ha-* already matches, no change needed there).
+#
+# Second port on each (61442/61443, ClusterIP only) for the CoreDNS
+# rewrite in infrastructure/coredns/manifests/: home's own pods resolving
+# pg-gitlab.ha.huskypup.net / pg-praefect.ha.huskypup.net rewrite to
+# THESE services specifically so they can reach them on the SAME port
+# number the app tier will eventually be configured with (Phase 3) -
+# rewrite only changes the resolved NAME, not the port a client then
+# connects to (confirmed live for the authentik precedent).
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-gitlab-postgres
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    cnpg.io/cluster: pg-gitlab
+    cnpg.io/instanceRole: primary
+  ports:
+    - name: nodeport-5432
+      port: 5432
+      targetPort: 5432
+      nodePort: 32442
+    - name: floating-port-61442
+      port: 61442
+      targetPort: 5432
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-praefect-postgres
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    cnpg.io/cluster: pg-praefect
+    cnpg.io/instanceRole: primary
+  ports:
+    - name: nodeport-5432
+      port: 5432
+      targetPort: 5432
+      nodePort: 32443
+    - name: floating-port-61443
+      port: 61443
+      targetPort: 5432

apps/gitlab/manifests/praefect-cnpg-cluster.yaml

@@ -26,10 +26,94 @@ spec:
 
   primaryUpdateStrategy: unsupervised
 
+  # Explicit params added 2026-08-22 (GitLab cross-site replication plan)
+  # - this cluster ran on bare Postgres defaults before. A VPS replica
+  # can't replay WAL past a checkpoint recorded with higher settings than
+  # its own ("recovery aborted because of insufficient parameter
+  # settings", confirmed live for pg-authentik) - giving both sides an
+  # identical explicit block up front means nobody can tune one side
+  # later and silently break replication.
+  postgresql:
+    parameters:
+      max_connections: "200"
+      shared_buffers: "512MB"
+      effective_cache_size: "1536MB"
+      maintenance_work_mem: "128MB"
+      checkpoint_completion_target: "0.9"
+      wal_buffers: "16MB"
+      default_statistics_target: "100"
+      random_page_cost: "1.1"
+      effective_io_concurrency: "200"
+      work_mem: "4MB"
+      min_wal_size: "1GB"
+      max_wal_size: "4GB"
+      max_worker_processes: "4"
+
   bootstrap:
     initdb:
       database: praefect_production
       owner: app
 
+  # Backup to VPS MinIO, also the initial-seed source for the VPS's own
+  # pg-praefect replica (bootstrap.recovery, Phase 1c) - this cluster had
+  # no backup block at all before. See the matching comment on
+  # cnpg-cluster.yaml (pg-gitlab) in this same directory.
+  backup:
+    barmanObjectStore:
+      destinationPath: s3://cnpg-backups/pg-praefect
+      endpointURL: http://100.108.113.41:9000
+      s3Credentials:
+        accessKeyId:
+          name: vps-minio-secret
+          key: accesskey
+        secretAccessKey:
+          name: vps-minio-secret
+          key: secretkey
+      wal:
+        compression: gzip
+        maxParallel: 2
+    retentionPolicy: "30d"
+
   monitoring:
     enablePodMonitor: true
+
+  # GitLab cross-site replication Phase 1c stage 2 - same pattern as
+  # cnpg-cluster.yaml (pg-gitlab) in this same directory.
+  externalClusters:
+    - name: home
+      connectionParameters:
+        host: pg-praefect-rw
+        port: "5432"
+        dbname: praefect_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-praefect-replication
+        key: tls.crt
+      sslKey:
+        name: pg-praefect-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-praefect-ca
+        key: ca.crt
+    - name: vps
+      connectionParameters:
+        host: 100.108.113.41
+        port: "32445"
+        dbname: praefect_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-praefect-vps-replication
+        key: tls.crt
+      sslKey:
+        name: pg-praefect-vps-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-praefect-vps-ca
+        key: ca.crt
+
+  replica:
+    self: home
+    primary: home
+    source: home

apps/gitlab/manifests/praefect-ha-configmap.yaml

@@ -0,0 +1,88 @@
+# GitLab cross-site replication (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - the chart has
+# no support for registering an externally-hosted Gitaly node into an
+# existing Praefect virtual storage (confirmed live: no
+# virtualStorages[].nodes/address-override key exists in the template,
+# gitalyReplicas just counts StatefulSet ordinals). Overriding Praefect's
+# rendered config.toml is the only way to add one.
+#
+# A values-only CONFIG_TEMPLATE_DIRECTORY redirect (extraEnv +
+# extraVolumes/extraVolumeMounts) was tried first and confirmed NOT to
+# work: charts/gitlab/charts/praefect/templates/statefulset.yaml's own
+# volumes: list never calls the gitlab.extraVolumes helper (only
+# volumeMounts does) - a chart limitation specific to this subchart, no
+# values-only fix exists.
+#
+# THIS ConfigMap (praefect-ha-config) is deployed as a harmless,
+# otherwise-unused object - it exists purely as a git-tracked reference
+# copy of the content that actually matters. The REAL live config lives
+# on the chart's own gitlab-praefect ConfigMap, which
+# argocd-apps/apps/gitlab.yaml's ignoreDifferences now exempts from
+# ArgoCD's normal drift-reconciliation (its `data` field specifically) -
+# apply this file's content to it directly:
+#   kubectl -n gitlab patch configmap gitlab-praefect --type merge \
+#     -p "{\"data\":{\"config.toml.tpl\":\"$(kubectl -n gitlab get cm \
+#     praefect-ha-config -o jsonpath='{.data.config\.toml\.tpl}' | \
+#     python3 -c 'import sys,json; print(json.dumps(sys.stdin.read())[1:-1])')\"}}"
+# (or simpler: kubectl -n gitlab get cm praefect-ha-config -o
+# jsonpath='{.data}' | kubectl -n gitlab patch cm gitlab-praefect --type
+# merge -p "{\"data\":$(cat -)}")
+#
+# Content below has since moved past byte-identical: nodes 0-2 now use
+# tls:// on port 8076 (matching global.gitaly.tls.enabled in
+# values.yaml) and the floating hostnames gitaly-{0,1,2}.ha.huskypup.net
+# (CoreDNS-rewritten locally for home's own traffic, same reasoning as
+# pg-gitlab.ha.huskypup.net) rather than in-cluster DNS names or the
+# plaintext scheme - this is what makes the SAME addresses usable once
+# the VPS joins as a 4th node, without home's own Praefect needing a
+# different config than what it hands the VPS as peer addresses. Praefect
+# trusts the cert via the pod's normal system CA bundle - it's a real
+# Let's Encrypt cert (gitaly-tls-certificate.yaml), not self-signed, so
+# no extra CA trust config is needed here.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: praefect-ha-config
+  namespace: gitlab
+data:
+  config.toml.tpl: |
+    # TCP address to listen on
+    listen_addr = '0.0.0.0:8075'
+    prometheus_listen_addr = '0.0.0.0:9236'
+
+    prometheus_exclude_database_from_default_metrics = true
+
+    [failover]
+    enabled = true
+    read_only_after_failover = true
+
+    [auth]
+    token = {% file.Read "/etc/gitlab-secrets/praefect/praefect_token" | strings.TrimSpace | data.ToJSON %}
+
+    transitioning = false
+
+    [logging]
+
+    [[virtual_storage]]
+    name = 'default'
+    [[virtual_storage.node]]
+    storage = 'gitlab-gitaly-default-0'
+    address = 'tls://gitaly-0.ha.huskypup.net:8076'
+    token = {% file.Read "/etc/gitlab-secrets/praefect/gitaly_token" | strings.TrimSpace | data.ToJSON %}
+    [[virtual_storage.node]]
+    storage = 'gitlab-gitaly-default-1'
+    address = 'tls://gitaly-1.ha.huskypup.net:8076'
+    token = {% file.Read "/etc/gitlab-secrets/praefect/gitaly_token" | strings.TrimSpace | data.ToJSON %}
+    [[virtual_storage.node]]
+    storage = 'gitlab-gitaly-default-2'
+    address = 'tls://gitaly-2.ha.huskypup.net:8076'
+    token = {% file.Read "/etc/gitlab-secrets/praefect/gitaly_token" | strings.TrimSpace | data.ToJSON %}
+
+
+    [database]
+    host = 'pg-praefect-rw.gitlab.svc.cluster.local'
+    port = 5432
+    user = 'app'
+    password = {% file.Read "/etc/gitlab-secrets/praefect/db_password" | strings.TrimSpace | data.ToJSON %}
+    dbname = 'praefect_production'
+    sslmode = 'disable'

apps/gitlab/values.yaml

@@ -54,7 +54,50 @@ global:
     internal:
       names: []  # No internal Gitaly, using Praefect
     external: []  # Praefect configured below
-  
+    # GitLab cross-site replication (see
+    # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - Gitaly's
+    # gRPC has no TLS by default. The VPS's 4th node has to be reached
+    # over the same public UniFi WAN forward already used for Postgres
+    # (confirmed live the direct-Netbird-IP path doesn't accept inbound
+    # connections to home, same as the existing etcd quorum) - without
+    # this, that WAN hop would carry the gitaly_token,
+    # gitlab_shell_secret, and raw repository contents in plaintext.
+    # Chart-native support (tls_listen_addr alongside the existing
+    # plaintext listener, both stay active) - cert is a real Let's
+    # Encrypt cert via the same DNS-01 ClusterIssuer already used
+    # everywhere else (gitaly-tls-certificate.yaml), not a self-signed
+    # one, so no custom CA needs distributing to either site.
+    tls:
+      enabled: true
+
+  # Gitaly's [gitlab] url (the internal-API callback every Gitaly node
+  # uses for gitlab-shell hooks) - a real, TLS-supported override,
+  # confirmed live via gitlab/charts/gitlab/templates/_workhorse.tpl.
+  # Uses the SAME floating-hostname + CoreDNS-local-rewrite pattern
+  # already proven for Postgres and Gitaly's own node addresses: home's
+  # 3 Gitaly nodes resolve this locally (no WAN round-trip), the VPS's
+  # 4th node resolves it via the UniFi WAN forward - one consistent
+  # URL for every node regardless of site.
+  #
+  # tls.enabled REVERTED 2026-08-22 - broke the live site (HTTP 400
+  # "Client sent an HTTP request to an HTTPS server" on
+  # gitlab.kube.huskypup.net). Root cause: port 8181 is NOT internal-API
+  # -only - confirmed live via `kubectl get httproute gitlab-web` that
+  # ALL public traffic (gitlab.kube.huskypup.net, routed through Envoy/
+  # Istio edge gateway) also targets gitlab-webservice-default:8181.
+  # global.workhorse.tls.enabled replaces the plaintext listener with
+  # TLS on that SAME port chart-wide, for both the public site AND the
+  # internal API - there's no chart-native way to TLS one without the
+  # other. Envoy terminates public HTTPS and proxies to the pod in
+  # plaintext, so a TLS-only pod listener 400s every public request.
+  # host: left in place (harmless without tls.enabled - only used for
+  # the [gitlab] url string, not port selection). Internal-API-over-WAN
+  # transport security for the VPS's future Gitaly node needs a
+  # different mechanism (e.g. network-layer TLS/mTLS via Istio, or a
+  # second dedicated port if the chart ever adds one) - not solved here.
+  workhorse:
+    host: gitlab-internal-api.ha.huskypup.net
+
   # Praefect configuration
   praefect:
     enabled: true
@@ -68,6 +111,12 @@ global:
     - name: default
       gitalyReplicas: 3  # Production HA
       maxUnavailable: 1
+      # Required once global.gitaly.tls.enabled: true - the chart's own
+      # NOTES.txt validation rejects the render without it ("not
+      # specified ('default')... not supported", confirmed live).
+      # Matches gitlab.gitaly.tls.secret's default name/keys
+      # (gitaly-tls-certificate.yaml issues into this exact secret).
+      tlsSecretName: gitlab-gitaly-tls
     
     # Praefect PostgreSQL configuration
     psql:
@@ -285,14 +334,14 @@ gitlab:
     enabled: true
     minReplicas: 1
     maxReplicas: 1
-    
+
     # Note: Praefect PostgreSQL config is in global.praefect.psql
-    
+
     # Use CNPG-generated database secret
     dbSecret:
       secret: pg-praefect-app
       key: password
-    
+
     # Resources
     resources:
       requests:
@@ -301,13 +350,37 @@ gitlab:
       limits:
         cpu: 250m
         memory: 512Mi
-    
-    # Virtual storage configuration
-    virtualStorages:
-    - name: default
-      gitalyReplicas: 1  # Homelab sizing
-      maxUnavailable: 1
-  
+
+    # (dead config removed 2026-08-22: this virtualStorages block never
+    # actually did anything - confirmed live the chart's Praefect
+    # ConfigMap template iterates global.praefect.virtualStorages, not
+    # this one, so gitalyReplicas: 3 up there always won regardless of
+    # what this said. See global.praefect.virtualStorages above for the
+    # real config.)
+    #
+    # GitLab cross-site replication (see
+    # /home/scooby/.claude/plans/jiggly-snacking-iverson.md): tried a
+    # values-only CONFIG_TEMPLATE_DIRECTORY redirect (extraEnv +
+    # extraVolumes/extraVolumeMounts) to add a VPS-hosted Gitaly node the
+    # chart has no mechanism to register - confirmed live this doesn't
+    # work: extraVolumeMounts and extraEnv both render correctly, but
+    # extraVolumes is a dead values key for THIS subchart specifically -
+    # charts/gitlab/charts/praefect/templates/statefulset.yaml's own
+    # volumes: list never calls the gitlab.extraVolumes helper at all
+    # (only volumeMounts does), so the mount has nothing to mount and the
+    # pod would fail to schedule. No values-only fix exists.
+    #
+    # Real fix: argocd-apps/apps/gitlab.yaml's ignoreDifferences now
+    # covers ConfigMap gitlab-praefect's `data` field, so Helm creates
+    # the object (with its own 3-node content) but ArgoCD stops
+    # reconciling its content afterward. The actual multi-node content
+    # lives in apps/gitlab/manifests/praefect-ha-configmap.yaml as a
+    # git-tracked reference/documentation copy (deployed under its own
+    # name, praefect-ha-config, harmless and otherwise unused) - the
+    # live gitlab-praefect ConfigMap gets kubectl-patched with that same
+    # content directly, same "documented but manually-applied"
+    # convention as every other cross-cluster secret in this plan.
+
   # GitLab Exporter for Prometheus metrics
   gitlab-exporter:
     enabled: true

argocd-apps/apps/gitlab.yaml

@@ -33,6 +33,19 @@ spec:
       jsonPointers:
         - /spec/ports
         - /spec/selector
+    # GitLab cross-site replication (see
+    # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - the chart
+    # has no mechanism to register an externally-hosted (VPS) Gitaly node
+    # into Praefect's virtual storage. Lets the live ConfigMap's `data`
+    # be kubectl-patched directly (see apps/gitlab/manifests/
+    # praefect-ha-configmap.yaml for the content + patch command) without
+    # ArgoCD reverting it on the next sync - Helm still creates the
+    # object and everything else about it stays normally managed.
+    - group: ""
+      kind: ConfigMap
+      name: gitlab-praefect
+      jsonPointers:
+        - /data
   syncPolicy:
     automated:
       prune: true

infrastructure/cert-manager/manifests/letsencrypt-issuer.yaml

@@ -1,15 +1,15 @@
----
-apiVersion: cert-manager.io/v1
-kind: ClusterIssuer
-metadata:
-  name: letsencrypt-production
-spec:
-  acme:
-    server: https://acme-v02.api.letsencrypt.org/directory
-    email: garrettstone499@gmail.com
-    privateKeySecretRef:
-      name: letsencrypt-production
-    solvers:
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-production
+spec:
+  acme:
+    server: https://acme-v02.api.letsencrypt.org/directory
+    email: garrettstone499@gmail.com
+    privateKeySecretRef:
+      name: letsencrypt-production
+    solvers:
       - dns01:
           cloudflare:
             email: garrettstone499@gmail.com
@@ -19,6 +19,17 @@ spec:
         selector:
           dnsZones:
             - "kube.huskypup.net"
+            # GitLab cross-site replication (see
+            # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) -
+            # first real Let's Encrypt cert needed for anything under
+            # ha.huskypup.net (the floating-hostname zone used for
+            # cross-site Postgres/Gitaly addressing) - everything
+            # before this used CNPG's own self-signed certs, never hit
+            # this gap. Confirmed live: DNS-01 order failed with "no
+            # configured challenge solvers can be used for this
+            # challenge" until this zone was added. Same Cloudflare
+            # account/token as kube.huskypup.net.
+            - "ha.huskypup.net"
 
 ---
 apiVersion: cert-manager.io/v1

infrastructure/coredns/manifests/coredns-configmap.yaml

@@ -58,6 +58,30 @@ data:
         # specifically so this rewrite target actually has that port.
         rewrite name pg-authentik.ha.huskypup.net ha-authentik-postgres.authentik.svc.cluster.local
 
+        # GitLab cross-site replication (see
+        # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - same
+        # reasoning as pg-authentik.ha.huskypup.net above: home's own pods
+        # go straight to the local NodePort Service instead of round-
+        # tripping out through the UniFi WAN forward and back in (no NAT
+        # hairpin support, confirmed live for the authentik precedent).
+        rewrite name pg-gitlab.ha.huskypup.net ha-gitlab-postgres.gitlab.svc.cluster.local
+        rewrite name pg-praefect.ha.huskypup.net ha-praefect-postgres.gitlab.svc.cluster.local
+
+        # GitLab cross-site replication Phase 2b (Gitaly) - home's own
+        # Praefect dials these SAME floating hostnames it hands to the
+        # VPS's Gitaly node as peer addresses (Praefect uses one address
+        # per node for everyone, not different addresses per caller), so
+        # home's own in-cluster traffic to its own local Gitaly nodes
+        # needs to resolve locally too, same reasoning as pg-*.ha above.
+        rewrite name gitaly-0.ha.huskypup.net ha-gitaly-0.gitlab.svc.cluster.local
+        rewrite name gitaly-1.ha.huskypup.net ha-gitaly-1.gitlab.svc.cluster.local
+        rewrite name gitaly-2.ha.huskypup.net ha-gitaly-2.gitlab.svc.cluster.local
+
+        # global.workhorse.host override (apps/gitlab/values.yaml) - every
+        # Gitaly node's [gitlab] url, home's own included, same local-
+        # rewrite reasoning as the entries above.
+        rewrite name gitlab-internal-api.ha.huskypup.net ha-gitlab-internal-api.gitlab.svc.cluster.local
+
         kubernetes cluster.local in-addr.arpa ip6.arpa {
             pods insecure
             fallthrough in-addr.arpa ip6.arpa

infrastructure/istio/manifests/apps/authorization-policies-apps.yaml

@@ -83,6 +83,25 @@ spec:
         - source:
             ipBlocks:
               - "10.244.0.0/16"
+    # GitLab cross-site replication (see
+    # /home/scooby/.claude/plans/jiggly-snacking-iverson.md): CNPG
+    # streaming replication from the VPS/witness, arriving via the
+    # ha-gitlab-postgres/ha-praefect-postgres NodePorts - no mesh
+    # identity at all, so no source.namespaces rule above can match them.
+    # Scoped by destination port instead, matching the port-scoped
+    # PERMISSIVE PeerAuthentication in ha-peerauth.yaml (same root cause,
+    # same fix, one layer up - mTLS being allowed through isn't enough by
+    # itself, this ALLOW policy independently denies anything not
+    # matching one of the rules above). Same pattern already proven for
+    # Authentik in infrastructure/istio/manifests/mesh/authorization-policies.yaml.
+    - to:
+        - operation:
+            ports:
+              - "5432"
+              - "61442" # pg-gitlab floating-hostname port, see ha-postgres-nodeport.yaml
+              - "61443" # pg-praefect floating-hostname port, see ha-postgres-nodeport.yaml
+              - "8075" # Gitaly gRPC, see ha-gitaly-nodeport.yaml
+              - "8181" # Rails internal API, see ha-gitlab-internal-api-nodeport.yaml
 
 ---
 # --- n8n: Allow ingress + Prometheus + intra-namespace ---

infrastructure/vps-standby/gitlab/manifests/cnpg-cluster.yaml

@@ -1,8 +1,24 @@
 ---
-# Real GitLab on the VPS - simplified single-node equivalent of
-# apps/gitlab/manifests/cnpg-cluster.yaml (home's is 3-instance HA with
-# Rook-Ceph; this is 1 instance on local-path, matching the "keep it
-# lighter" pattern already used for the rest of vps-standby).
+# GitLab cross-site replication Phase 1c (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - was a fully
+# independent standalone cluster (own initdb, own root/Scooby test data,
+# never meant to survive this). Now a real replica of home's pg-gitlab
+# (apps/gitlab/manifests/cnpg-cluster.yaml), matching the pattern already
+# proven for pg-authentik.
+#
+# Deployed in two stages, same as the original authentik/n8n/nextcloud
+# VPS replicas were: first as a plain bootstrap.recovery replica (this
+# commit - no externalClusters/replica streaming config yet, since that
+# needs THIS cluster's own CNPG-generated certs to exist first, which
+# only happens once the cluster itself exists), then a second commit adds
+# the streaming replication config once those certs have been copied
+# cross-cluster.
+#
+# postgresql.parameters MUST match-or-exceed home's - copied verbatim
+# from apps/gitlab/manifests/cnpg-cluster.yaml. A replica can't replay
+# WAL past a checkpoint recorded with higher settings than its own
+# ("recovery aborted because of insufficient parameter settings",
+# confirmed live for pg-authentik).
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
@@ -12,43 +28,117 @@ spec:
   imageName: ghcr.io/cloudnative-pg/postgresql:16
   instances: 1
 
+  # shared_buffers 1GB below needs headroom - matches home's requests/limits.
   resources:
     requests:
-      memory: "512Mi"
+      memory: "1Gi"
       cpu: "50m"
     limits:
-      memory: "2Gi"
+      memory: "4Gi"
       cpu: "500m"
 
   storage:
-    size: 20Gi
+    size: 50Gi  # matches home's size
     storageClass: local-path
 
   primaryUpdateStrategy: unsupervised
 
-  bootstrap:
-    initdb:
-      database: gitlabhq_production
-      owner: app
-      postInitSQL:
-        - CREATE EXTENSION IF NOT EXISTS pg_trgm;
-        - CREATE EXTENSION IF NOT EXISTS btree_gist;
-
-  # Same tuning as home's, GitLab needs these regardless of scale
   postgresql:
     parameters:
-      max_connections: "200"
-      shared_buffers: "512MB"
-      effective_cache_size: "1536MB"
-      maintenance_work_mem: "128MB"
+      max_connections: "400"
+      shared_buffers: "1GB"
+      effective_cache_size: "3GB"
+      maintenance_work_mem: "256MB"
       checkpoint_completion_target: "0.9"
       wal_buffers: "16MB"
       default_statistics_target: "100"
       random_page_cost: "1.1"
       effective_io_concurrency: "200"
-      work_mem: "8MB"
+      work_mem: "16MB"
       min_wal_size: "1GB"
       max_wal_size: "4GB"
+      max_worker_processes: "4"
+      max_parallel_workers_per_gather: "2"
+      max_parallel_workers: "4"
+
+  bootstrap:
+    recovery:
+      source: home
+
+  # Stage 2 (see /home/scooby/.claude/plans/jiggly-snacking-iverson.md):
+  # connectionParameters added alongside the existing barmanObjectStore
+  # on the SAME "home" entry - the recovery-mode initial seed stays
+  # exactly as-is, ongoing replication now streams instead of relying on
+  # WAL-archive polling (the source of the crash-looping/probe-tuning
+  # fights documented for the pg-authentik precedent). Also needs a
+  # self-referencing "vps" entry - replica.self must match an
+  # externalClusters name too, even though it's this very cluster
+  # (confirmed live: "External cluster vps not found" until one was
+  # added).
+  #
+  # Reachable via home.kube.huskypup.net:61442 - a UniFi WAN port-forward
+  # to talos-cp-01's real LAN IP, source-IP-restricted to the VPS/witness
+  # public IPs (see apps/gitlab/manifests/ha-postgres-nodeport.yaml). NOT
+  # the Netbird mesh directly - UniFi has no route to that CIDR
+  # (confirmed live for the pg-authentik precedent).
+  externalClusters:
+    - name: home
+      barmanObjectStore:
+        # serverName defaults to the externalClusters[].name, NOT the
+        # source Postgres cluster's actual metadata.name - home's own
+        # backup: block defaults serverName to "pg-gitlab" (its own
+        # name). Without this override, recovery looks under the wrong
+        # server-name prefix and finds nothing (confirmed live for the
+        # pg-authentik precedent).
+        serverName: pg-gitlab
+        destinationPath: s3://cnpg-backups/pg-gitlab
+        endpointURL: http://vps-minio.minio.svc.cluster.local:9000
+        s3Credentials:
+          accessKeyId:
+            name: vps-minio-secret
+            key: accesskey
+          secretAccessKey:
+            name: vps-minio-secret
+            key: secretkey
+      connectionParameters:
+        host: home.kube.huskypup.net
+        port: "61442"
+        dbname: gitlabhq_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-gitlab-home-replication
+        key: tls.crt
+      sslKey:
+        name: pg-gitlab-home-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-gitlab-home-ca
+        key: ca.crt
+    - name: vps
+      connectionParameters:
+        host: pg-gitlab-rw
+        port: "5432"
+        dbname: gitlabhq_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-gitlab-replication
+        key: tls.crt
+      sslKey:
+        name: pg-gitlab-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-gitlab-ca
+        key: ca.crt
+
+  # Distributed topology: both sides agree home is primary today. No
+  # replica.enabled - mutually exclusive with primary/self (confirmed
+  # live for the pg-authentik precedent).
+  replica:
+    self: vps
+    primary: home
+    source: home
 
   monitoring:
     enablePodMonitor: false  # no Prometheus on the VPS cluster

infrastructure/vps-standby/gitlab/manifests/ha-gitlab-secrets-externalsecret.yaml

@@ -0,0 +1,87 @@
+# GitLab cross-site replication Phase 1b - VPS's half of the Vault
+# pipeline (apps/gitlab/manifests/ha-gitlab-secrets-pushsecret.yaml has
+# home's half, which pushes the authoritative values into Vault at
+# secret/vps/gitlab-{rails,gitlab-shell,gitaly,praefect}-secret). Pulls
+# them back down here, BEFORE Phase 1c's Postgres replication goes live.
+#
+# creationPolicy: Owner (not Merge) - each of these secrets has exactly
+# one key with nothing else to preserve, unlike pg-authentik-app which
+# has CNPG-local host/dbname/username fields alongside the shared
+# password. The chart's shared-secrets job skips creating a key that
+# already exists, so having ESO own these outright is sufficient to stop
+# the VPS from ever generating its own divergent values.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: gitlab-rails-secret
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: gitlab-rails-secret
+    creationPolicy: Owner
+  data:
+    - secretKey: secrets.yml
+      remoteRef:
+        key: vps/gitlab-rails-secret
+        property: secrets.yml
+---
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: gitlab-gitlab-shell-secret
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: gitlab-gitlab-shell-secret
+    creationPolicy: Owner
+  data:
+    - secretKey: secret
+      remoteRef:
+        key: vps/gitlab-gitlab-shell-secret
+        property: secret
+---
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: gitlab-gitaly-secret
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: gitlab-gitaly-secret
+    creationPolicy: Owner
+  data:
+    - secretKey: token
+      remoteRef:
+        key: vps/gitlab-gitaly-secret
+        property: token
+---
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: gitlab-praefect-secret
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: gitlab-praefect-secret
+    creationPolicy: Owner
+  data:
+    - secretKey: token
+      remoteRef:
+        key: vps/gitlab-praefect-secret
+        property: token

infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml

@@ -0,0 +1,62 @@
+# GitLab cross-site replication - VPS's half of the Vault pipeline
+# (apps/gitlab/manifests/ha-postgres-app-pushsecret.yaml has home's half).
+# Pulls the authoritative app-role passwords back down here.
+#
+# creationPolicy REVERTED Merge -> Owner 2026-08-22: Merge assumed CNPG
+# itself would generate a base pg-gitlab-app/pg-praefect-app secret
+# (host/dbname/username keys) for this ExternalSecret to merge a
+# password into - true for a normal owning/primary Cluster, but WRONG
+# for these VPS Clusters since Phase 1c's recreate: `pg-gitlab`/
+# `pg-praefect` here are bootstrap.recovery REPLICA clusters
+# (replica.self: vps) - the `app` role is part of the replicated
+# global role/password state streamed from home's primary, so CNPG
+# correctly never creates a local owner secret for it here. Merge had
+# nothing to merge onto -> ExternalSecret condition
+# "SecretMissing/CreationPolicy=Merge" -> pg-gitlab-app never existed
+# -> vps-gitlab-webservice crash-looped on
+# ActiveRecord::DatabaseConnectionError for ~39h before this was
+# caught (confirmed live, unrelated to any same-day change). Owner:
+# the ExternalSecret creates the whole secret itself. This chart only
+# ever reads the `password` key from it - host/port/dbname/username
+# are hardcoded directly in infrastructure/vps-standby/gitlab/
+# values.yaml's global.psql/global.praefect.psql blocks, not read from
+# this secret's other keys, so a password-only Owner-created secret is
+# sufficient (confirmed live: no other consumer of this secret's keys
+# found).
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: pg-gitlab-app-password
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: pg-gitlab-app
+    creationPolicy: Owner
+  data:
+    - secretKey: password
+      remoteRef:
+        key: vps/pg-gitlab-app
+        property: password
+---
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: pg-praefect-app-password
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: pg-praefect-app
+    creationPolicy: Owner
+  data:
+    - secretKey: password
+      remoteRef:
+        key: vps/pg-praefect-app
+        property: password

infrastructure/vps-standby/gitlab/manifests/ha-postgres-nodeport.yaml

@@ -0,0 +1,47 @@
+---
+# Exposes the VPS's pg-gitlab/pg-praefect primaries (replicas today) back
+# to home over the Netbird mesh, for GitLab cross-site replication (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - needed for
+# failback (home rejoining as a replica once it's back). Same pattern as
+# infrastructure/vps-standby/authentik/manifests/ha-postgres-nodeport.yaml.
+#
+# No UniFi port-forward or Kyverno PolicyException needed here - the
+# VPS's k3s has no NodePort restriction, and home reaching OUT to the VPS
+# already works today via the netbird-egress DaemonSet's route (confirmed
+# live all session for MinIO/CrowdSec/pg-authentik) - it's only the
+# reverse direction (external peers reaching INTO home) that needs the
+# UniFi workaround (see apps/gitlab/manifests/ha-postgres-nodeport.yaml).
+#
+# Different nodePort numbers than home's side (32442/32443) - not
+# required since these are different machines, but matches the
+# authentik precedent's convention of keeping each site's numbers
+# distinct to avoid confusion.
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-gitlab-postgres
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    cnpg.io/cluster: pg-gitlab
+    cnpg.io/instanceRole: primary
+  ports:
+    - port: 5432
+      targetPort: 5432
+      nodePort: 32444
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-praefect-postgres
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    cnpg.io/cluster: pg-praefect
+    cnpg.io/instanceRole: primary
+  ports:
+    - port: 5432
+      targetPort: 5432
+      nodePort: 32445

infrastructure/vps-standby/gitlab/manifests/praefect-cnpg-cluster.yaml

@@ -0,0 +1,108 @@
+---
+# GitLab cross-site replication Phase 1c (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - the VPS never
+# had a pg-praefect cluster before (VPS ran Gitaly without Praefect,
+# global.praefect.enabled: false). This is new, matching home's
+# apps/gitlab/manifests/praefect-cnpg-cluster.yaml as a real replica.
+#
+# Same two-stage deployment as cnpg-cluster.yaml in this directory: this
+# commit is bootstrap.recovery only, streaming config added once this
+# cluster's own certs exist to copy cross-cluster.
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: pg-praefect
+  namespace: gitlab
+spec:
+  imageName: ghcr.io/cloudnative-pg/postgresql:16
+  instances: 1
+
+  resources:
+    requests:
+      memory: "512Mi"
+      cpu: "50m"
+    limits:
+      memory: "2Gi"
+
+  storage:
+    size: 10Gi  # matches home's size
+    storageClass: local-path
+
+  primaryUpdateStrategy: unsupervised
+
+  # Must match-or-exceed home's - copied verbatim from
+  # apps/gitlab/manifests/praefect-cnpg-cluster.yaml.
+  postgresql:
+    parameters:
+      max_connections: "200"
+      shared_buffers: "512MB"
+      effective_cache_size: "1536MB"
+      maintenance_work_mem: "128MB"
+      checkpoint_completion_target: "0.9"
+      wal_buffers: "16MB"
+      default_statistics_target: "100"
+      random_page_cost: "1.1"
+      effective_io_concurrency: "200"
+      work_mem: "4MB"
+      min_wal_size: "1GB"
+      max_wal_size: "4GB"
+      max_worker_processes: "4"
+
+  bootstrap:
+    recovery:
+      source: home
+
+  # Stage 2 - same pattern as cnpg-cluster.yaml (pg-gitlab) in this same
+  # directory.
+  externalClusters:
+    - name: home
+      barmanObjectStore:
+        serverName: pg-praefect
+        destinationPath: s3://cnpg-backups/pg-praefect
+        endpointURL: http://vps-minio.minio.svc.cluster.local:9000
+        s3Credentials:
+          accessKeyId:
+            name: vps-minio-secret
+            key: accesskey
+          secretAccessKey:
+            name: vps-minio-secret
+            key: secretkey
+      connectionParameters:
+        host: home.kube.huskypup.net
+        port: "61443"
+        dbname: praefect_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-praefect-home-replication
+        key: tls.crt
+      sslKey:
+        name: pg-praefect-home-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-praefect-home-ca
+        key: ca.crt
+    - name: vps
+      connectionParameters:
+        host: pg-praefect-rw
+        port: "5432"
+        dbname: praefect_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-praefect-replication
+        key: tls.crt
+      sslKey:
+        name: pg-praefect-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-praefect-ca
+        key: ca.crt
+
+  replica:
+    self: vps
+    primary: home
+    source: home
+
+  monitoring:
+    enablePodMonitor: false  # no Prometheus on the VPS cluster

infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml

@@ -16,8 +16,20 @@
 #   kubectl -n nextcloud create secret generic nextcloud-restic-password \
 #     --from-literal=password=<home's nextcloud-restic-password>
 #
-# Nextcloud itself isn't running (replicas: 0, see values.yaml) so there's
-# no live-write conflict risk overwriting the PVC on every run.
+# CORRECTED 2026-08-22: the "replicas: 0" assumption below was wrong -
+# confirmed live vps-nextcloud actually runs continuously at replicas: 1
+# (no replicas key was ever set in values.yaml; the chart's own default
+# applies). This job had been silently failing every run for 2+ days as
+# a result: restic couldn't overwrite the live pod's config.php (mode
+# 0640, owned by www-data/33; this job runs as UID 65534 with only
+# group 33 via fsGroup, so it can read but not write that one file) -
+# "ignoring error for /data/config/config.php: permission denied",
+# Fatal: 1 error, even though the other 31000+ files/15GiB restored
+# fine every time. Fixed by excluding config.php from the restore
+# below rather than chasing permissions - overwriting a live, running
+# app's own config.php from a background restore job is the wrong
+# move regardless (it holds this site's actual current db credentials,
+# etc.), not just a permissions bug to route around.
 apiVersion: batch/v1
 kind: CronJob
 metadata:
@@ -64,7 +76,7 @@ spec:
                   fi
 
                   echo "==> Restoring latest snapshot into /data..."
-                  restic restore latest --tag nextcloud --host nextcloud-k8s --target /
+                  restic restore latest --tag nextcloud --host nextcloud-k8s --target / --exclude /data/config/config.php
 
                   echo "==> Done."
               env: