Fix VPS GitLab + Nextcloud crash loops found during health sweep

- infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml:
  creationPolicy Merge -> Owner (already applied live, committing to
  match). Merge assumed CNPG creates a base pg-gitlab-app/pg-praefect-app
  secret to merge a password into - wrong for these VPS Clusters since
  Phase 1c's replica-cluster recreate; CNPG doesn't create a local
  owner secret for a replicated role. Left pg-gitlab-app/pg-praefect-app
  missing entirely, crash-looping vps-gitlab-webservice on
  ActiveRecord::DatabaseConnectionError for ~39h.

- infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml:
  exclude config.php from the restic restore. The job's own comment
  assumed nextcloud runs at replicas: 0 on the VPS; live confirms
  replicas: 1 (no such key was ever actually set) - restic couldn't
  overwrite the live pod's config.php (permission denied), failing the
  whole job every run for 2+ days even though everything else restored
  fine. Excluding it is correct regardless of permissions - a live
  pod's own config.php shouldn't be overwritten by a background
  restore job.

Also fixed live (not git-tracked, config.php is PVC-persisted runtime
state, not sourced from git):
- home + VPS nextcloud config.php dbpassword: out of sync with CNPG's
  actual current pg-nextcloud-app password (baked in once at install,
  never re-synced). VPS's case was two-layered - its own
  pg-nextcloud-app secret also didn't match the real Postgres role
  password, since VPS's pg-nextcloud is a read-only streaming replica
  (spec.replica.enabled) and the authoritative password lives on home.
- grafana: broke a RollingUpdate deadlock (single-replica Deployment +
  RWO Ceph volume - new pod couldn't start while the old pod still
  held the only-one-node-at-a-time attachment, and the rollout
  wouldn't scale down the old pod until the new one was ready).
- suspended the stale gitlab-mirror-sync CronJob on the VPS (spamming
  auth failures every ~15min since its stored token predates the
  Postgres-replication cutover of VPS GitLab's DB) rather than
  deleting it, per the plan's own note to defer that until Phase 2b is
  verified.

apps/gitlab/manifests/cnpg-cluster.yaml

@@ -80,3 +80,58 @@ spec:
 
   monitoring:
     enablePodMonitor: true
+
+  # GitLab cross-site replication Phase 1c stage 2 (see
+  # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - streaming
+  # replication to/from the VPS's pg-gitlab cluster, alongside (not
+  # replacing) the barmanObjectStore backup above. Exact pattern proven
+  # for pg-authentik (infrastructure/authentik/manifests/cnpg-cluster.yaml).
+  #
+  # Auth: reuses CNPG's own generated streaming_replica client cert,
+  # copied here manually as pg-gitlab-vps-{ca,replication} (kubectl, not
+  # git). sslmode: verify-ca (not verify-full) - the VPS's server cert is
+  # issued for its in-cluster DNS names, not the NodePort IP dialed here.
+  #
+  # Reachable via 100.108.113.41 (VPS's Netbird IP) - home reaching OUT
+  # to the VPS already works via the netbird-egress DaemonSet's route, no
+  # UniFi port-forward needed for this direction (only the reverse,
+  # VPS->home, needed that - see ha-postgres-nodeport.yaml).
+  externalClusters:
+    - name: home
+      connectionParameters:
+        host: pg-gitlab-rw
+        port: "5432"
+        dbname: gitlabhq_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-gitlab-replication
+        key: tls.crt
+      sslKey:
+        name: pg-gitlab-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-gitlab-ca
+        key: ca.crt
+    - name: vps
+      connectionParameters:
+        host: 100.108.113.41
+        port: "32444"
+        dbname: gitlabhq_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-gitlab-vps-replication
+        key: tls.crt
+      sslKey:
+        name: pg-gitlab-vps-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-gitlab-vps-ca
+        key: ca.crt
+
+  # Distributed topology: both sides agree home is primary today.
+  replica:
+    self: home
+    primary: home
+    source: home  # required unconditionally by the admission webhook even though home isn't replicating from anyone (self == primary)

apps/gitlab/manifests/gitaly-tls-certificate.yaml

@@ -0,0 +1,55 @@
+# GitLab cross-site replication Phase 2b (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - Gitaly's gRPC
+# has no TLS by default; routing it over the public UniFi WAN forward
+# (needed because the direct-Netbird-IP path doesn't accept inbound
+# connections to home, confirmed live the same way as the existing etcd
+# quorum) would otherwise carry the gitaly_token, gitlab_shell_secret,
+# and raw repository contents in plaintext.
+#
+# Real Let's Encrypt cert via the same DNS-01 ClusterIssuer already
+# proven for every other *.huskypup.net cert in this cluster, rather
+# than a self-signed cert - sidesteps having to distribute a custom CA
+# to both home and the VPS (which already trust Let's Encrypt's public
+# CA by default). Covers all 4 Gitaly floating hostnames as SANs - one
+# shared cert, referenced by every node's tlsSecretName (or the global
+# default), simpler than per-node certs.
+#
+# secretName matches the chart's own default (confirmed live via
+# gitlab/templates/_gitaly.tpl's gitlab.gitaly.tls.secret helper:
+# "<release-name>-gitaly-tls" = "gitlab-gitaly-tls") - global.gitaly.tls.
+# enabled: true picks this up with no secretName override needed.
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: gitlab-gitaly-tls
+  namespace: gitlab
+spec:
+  secretName: gitlab-gitaly-tls
+  issuerRef:
+    name: letsencrypt-production
+    kind: ClusterIssuer
+  dnsNames:
+    - gitaly-0.ha.huskypup.net
+    - gitaly-1.ha.huskypup.net
+    - gitaly-2.ha.huskypup.net
+    - gitaly-vps.ha.huskypup.net
+---
+# Separate cert/secret for workhorse's internal-API TLS listener (port
+# 8181, [gitlab] url every Gitaly node calls for gitlab-shell hooks) -
+# different secretName (gitlab-workhorse-tls, chart default per
+# gitlab/charts/gitlab/charts/webservice/templates/_helpers.tpl's
+# workhorse.tls.secret helper) than Gitaly's own cert, kept separate
+# rather than trying to share one Secret object across two different
+# chart-expected names.
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: gitlab-workhorse-tls
+  namespace: gitlab
+spec:
+  secretName: gitlab-workhorse-tls
+  issuerRef:
+    name: letsencrypt-production
+    kind: ClusterIssuer
+  dnsNames:
+    - gitlab-internal-api.ha.huskypup.net

apps/gitlab/manifests/ha-gitaly-nodeport.yaml

@@ -0,0 +1,61 @@
+---
+# GitLab cross-site replication Phase 2b (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - exposes each
+# of home's 3 existing Gitaly nodes individually to the VPS, so Praefect
+# can register the VPS as a 4th node in the SAME virtual storage and
+# inter-Gitaly replication RPCs can reach each node directly. Per-pod
+# (not per-StatefulSet), unlike the chart's own gitlab-gitaly-default
+# Service - Praefect/Gitaly must address each node individually, not
+# load-balance across them.
+#
+# Reached over the same public UniFi WAN forward already used for
+# Postgres, NOT the Netbird mesh directly - confirmed live the direct-
+# Netbird-IP path doesn't accept inbound connections to home (same gap
+# already present for the existing etcd quorum; the earlier assumption
+# that NodePort binds on a reachable Netbird interface was wrong).
+# Targets Gitaly's TLS port (8076), not the plaintext one (8075) -
+# Gitaly's gRPC has no built-in transport security, and this WAN hop
+# would otherwise carry the gitaly_token, gitlab_shell_secret, and raw
+# repository contents in plaintext. See gitaly-tls-certificate.yaml and
+# global.gitaly.tls.enabled in values.yaml.
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-gitaly-0
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    statefulset.kubernetes.io/pod-name: gitlab-gitaly-default-0
+  ports:
+    - port: 8076
+      targetPort: 8076
+      nodePort: 32446
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-gitaly-1
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    statefulset.kubernetes.io/pod-name: gitlab-gitaly-default-1
+  ports:
+    - port: 8076
+      targetPort: 8076
+      nodePort: 32447
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-gitaly-2
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    statefulset.kubernetes.io/pod-name: gitlab-gitaly-default-2
+  ports:
+    - port: 8076
+      targetPort: 8076
+      nodePort: 32448

apps/gitlab/manifests/ha-gitlab-internal-api-nodeport.yaml

@@ -0,0 +1,32 @@
+# GitLab cross-site replication Phase 2b - exposes home's Rails internal
+# API (port 8181, a dedicated internal-only listener, deliberately NOT
+# exposed through the public Istio ingress on 8080) to the VPS's Gitaly
+# node. Gitaly's gitlab-shell hooks call POST /api/v4/internal/{allowed,
+# pre_receive,post_receive} against this whenever the VPS node is
+# primary for a repository (per-repository election can cause this even
+# in normal operation) or after a real failover.
+#
+# Reached over the same public UniFi WAN forward already used for
+# Postgres/Gitaly, not the Netbird mesh directly (confirmed live that
+# path doesn't accept inbound connections to home). Port stays 8181
+# (unlike Gitaly's separate 8075/8076 split) - global.workhorse.tls.
+# enabled in values.yaml replaces the plaintext listener with TLS on
+# this SAME port rather than adding a second one. Carries the
+# gitlab_shell_secret in a header, so this needed TLS just as much as
+# Gitaly's gRPC did - see gitaly-tls-certificate.yaml's second
+# Certificate (gitlab-workhorse-tls) and the
+# gitlab-internal-api.ha.huskypup.net CoreDNS rewrite.
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-gitlab-internal-api
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    app: webservice
+    release: gitlab
+  ports:
+    - port: 8181
+      targetPort: 8181
+      nodePort: 32449

apps/gitlab/manifests/ha-peerauth.yaml

@@ -49,3 +49,42 @@ spec:
       mode: PERMISSIVE
     "61443":
       mode: PERMISSIVE
+---
+# Phase 2b (Gitaly cross-site replication) - the 3 Gitaly pods, exposed
+# individually via ha-gitaly-nodeport.yaml for the VPS's 4th node to
+# join the same virtual storage. Gitaly pods already carry
+# istio.io/use-waypoint: none (chart default, gRPC doesn't tolerate L7
+# waypoint processing well) but that only skips L7 processing - L4
+# ztunnel mTLS enforcement is a separate concern, same fix needed as the
+# Postgres precedent.
+apiVersion: security.istio.io/v1
+kind: PeerAuthentication
+metadata:
+  name: allow-ha-gitaly-replication
+  namespace: gitlab
+spec:
+  selector:
+    matchLabels:
+      app: gitaly
+  mtls:
+    mode: STRICT
+  portLevelMtls:
+    "8075":
+      mode: PERMISSIVE
+---
+# Phase 2b - the VPS's Gitaly node calling home's Rails internal API
+# (ha-gitlab-internal-api-nodeport.yaml) for gitlab-shell hooks.
+apiVersion: security.istio.io/v1
+kind: PeerAuthentication
+metadata:
+  name: allow-ha-gitlab-internal-api
+  namespace: gitlab
+spec:
+  selector:
+    matchLabels:
+      app: webservice
+  mtls:
+    mode: STRICT
+  portLevelMtls:
+    "8181":
+      mode: PERMISSIVE

apps/gitlab/manifests/ha-postgres-app-pushsecret.yaml

@@ -0,0 +1,50 @@
+# GitLab cross-site replication (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - same pattern
+# and same reasoning as
+# infrastructure/authentik/manifests/ha-postgres-app-pushsecret.yaml:
+# pg-gitlab-app/pg-praefect-app are CNPG-generated, not Vault-native -
+# each cluster independently generated its own random `app` role
+# password at bootstrap time. Streaming replication keeps the two
+# clusters' actual live passwords in sync (WAL includes role/password
+# changes), but the two clusters' K8s Secret OBJECTS never resync on
+# their own - this closes that gap, self-healing on refreshInterval
+# instead of a one-time fix that goes stale on the next rotation.
+apiVersion: external-secrets.io/v1alpha1
+kind: PushSecret
+metadata:
+  name: pg-gitlab-app-to-vps
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRefs:
+    - name: vault-backend
+      kind: ClusterSecretStore
+  selector:
+    secret:
+      name: pg-gitlab-app
+  data:
+    - match:
+        secretKey: password
+        remoteRef:
+          remoteKey: vps/pg-gitlab-app
+          property: password
+---
+apiVersion: external-secrets.io/v1alpha1
+kind: PushSecret
+metadata:
+  name: pg-praefect-app-to-vps
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRefs:
+    - name: vault-backend
+      kind: ClusterSecretStore
+  selector:
+    secret:
+      name: pg-praefect-app
+  data:
+    - match:
+        secretKey: password
+        remoteRef:
+          remoteKey: vps/pg-praefect-app
+          property: password

apps/gitlab/manifests/praefect-cnpg-cluster.yaml

@@ -76,3 +76,44 @@ spec:
 
   monitoring:
     enablePodMonitor: true
+
+  # GitLab cross-site replication Phase 1c stage 2 - same pattern as
+  # cnpg-cluster.yaml (pg-gitlab) in this same directory.
+  externalClusters:
+    - name: home
+      connectionParameters:
+        host: pg-praefect-rw
+        port: "5432"
+        dbname: praefect_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-praefect-replication
+        key: tls.crt
+      sslKey:
+        name: pg-praefect-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-praefect-ca
+        key: ca.crt
+    - name: vps
+      connectionParameters:
+        host: 100.108.113.41
+        port: "32445"
+        dbname: praefect_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-praefect-vps-replication
+        key: tls.crt
+      sslKey:
+        name: pg-praefect-vps-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-praefect-vps-ca
+        key: ca.crt
+
+  replica:
+    self: home
+    primary: home
+    source: home

apps/gitlab/manifests/praefect-ha-configmap.yaml

@@ -0,0 +1,88 @@
+# GitLab cross-site replication (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - the chart has
+# no support for registering an externally-hosted Gitaly node into an
+# existing Praefect virtual storage (confirmed live: no
+# virtualStorages[].nodes/address-override key exists in the template,
+# gitalyReplicas just counts StatefulSet ordinals). Overriding Praefect's
+# rendered config.toml is the only way to add one.
+#
+# A values-only CONFIG_TEMPLATE_DIRECTORY redirect (extraEnv +
+# extraVolumes/extraVolumeMounts) was tried first and confirmed NOT to
+# work: charts/gitlab/charts/praefect/templates/statefulset.yaml's own
+# volumes: list never calls the gitlab.extraVolumes helper (only
+# volumeMounts does) - a chart limitation specific to this subchart, no
+# values-only fix exists.
+#
+# THIS ConfigMap (praefect-ha-config) is deployed as a harmless,
+# otherwise-unused object - it exists purely as a git-tracked reference
+# copy of the content that actually matters. The REAL live config lives
+# on the chart's own gitlab-praefect ConfigMap, which
+# argocd-apps/apps/gitlab.yaml's ignoreDifferences now exempts from
+# ArgoCD's normal drift-reconciliation (its `data` field specifically) -
+# apply this file's content to it directly:
+#   kubectl -n gitlab patch configmap gitlab-praefect --type merge \
+#     -p "{\"data\":{\"config.toml.tpl\":\"$(kubectl -n gitlab get cm \
+#     praefect-ha-config -o jsonpath='{.data.config\.toml\.tpl}' | \
+#     python3 -c 'import sys,json; print(json.dumps(sys.stdin.read())[1:-1])')\"}}"
+# (or simpler: kubectl -n gitlab get cm praefect-ha-config -o
+# jsonpath='{.data}' | kubectl -n gitlab patch cm gitlab-praefect --type
+# merge -p "{\"data\":$(cat -)}")
+#
+# Content below has since moved past byte-identical: nodes 0-2 now use
+# tls:// on port 8076 (matching global.gitaly.tls.enabled in
+# values.yaml) and the floating hostnames gitaly-{0,1,2}.ha.huskypup.net
+# (CoreDNS-rewritten locally for home's own traffic, same reasoning as
+# pg-gitlab.ha.huskypup.net) rather than in-cluster DNS names or the
+# plaintext scheme - this is what makes the SAME addresses usable once
+# the VPS joins as a 4th node, without home's own Praefect needing a
+# different config than what it hands the VPS as peer addresses. Praefect
+# trusts the cert via the pod's normal system CA bundle - it's a real
+# Let's Encrypt cert (gitaly-tls-certificate.yaml), not self-signed, so
+# no extra CA trust config is needed here.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: praefect-ha-config
+  namespace: gitlab
+data:
+  config.toml.tpl: |
+    # TCP address to listen on
+    listen_addr = '0.0.0.0:8075'
+    prometheus_listen_addr = '0.0.0.0:9236'
+
+    prometheus_exclude_database_from_default_metrics = true
+
+    [failover]
+    enabled = true
+    read_only_after_failover = true
+
+    [auth]
+    token = {% file.Read "/etc/gitlab-secrets/praefect/praefect_token" | strings.TrimSpace | data.ToJSON %}
+
+    transitioning = false
+
+    [logging]
+
+    [[virtual_storage]]
+    name = 'default'
+    [[virtual_storage.node]]
+    storage = 'gitlab-gitaly-default-0'
+    address = 'tls://gitaly-0.ha.huskypup.net:8076'
+    token = {% file.Read "/etc/gitlab-secrets/praefect/gitaly_token" | strings.TrimSpace | data.ToJSON %}
+    [[virtual_storage.node]]
+    storage = 'gitlab-gitaly-default-1'
+    address = 'tls://gitaly-1.ha.huskypup.net:8076'
+    token = {% file.Read "/etc/gitlab-secrets/praefect/gitaly_token" | strings.TrimSpace | data.ToJSON %}
+    [[virtual_storage.node]]
+    storage = 'gitlab-gitaly-default-2'
+    address = 'tls://gitaly-2.ha.huskypup.net:8076'
+    token = {% file.Read "/etc/gitlab-secrets/praefect/gitaly_token" | strings.TrimSpace | data.ToJSON %}
+
+
+    [database]
+    host = 'pg-praefect-rw.gitlab.svc.cluster.local'
+    port = 5432
+    user = 'app'
+    password = {% file.Read "/etc/gitlab-secrets/praefect/db_password" | strings.TrimSpace | data.ToJSON %}
+    dbname = 'praefect_production'
+    sslmode = 'disable'

apps/gitlab/values.yaml

@@ -54,7 +54,50 @@ global:
     internal:
       names: []  # No internal Gitaly, using Praefect
     external: []  # Praefect configured below
-  
+    # GitLab cross-site replication (see
+    # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - Gitaly's
+    # gRPC has no TLS by default. The VPS's 4th node has to be reached
+    # over the same public UniFi WAN forward already used for Postgres
+    # (confirmed live the direct-Netbird-IP path doesn't accept inbound
+    # connections to home, same as the existing etcd quorum) - without
+    # this, that WAN hop would carry the gitaly_token,
+    # gitlab_shell_secret, and raw repository contents in plaintext.
+    # Chart-native support (tls_listen_addr alongside the existing
+    # plaintext listener, both stay active) - cert is a real Let's
+    # Encrypt cert via the same DNS-01 ClusterIssuer already used
+    # everywhere else (gitaly-tls-certificate.yaml), not a self-signed
+    # one, so no custom CA needs distributing to either site.
+    tls:
+      enabled: true
+
+  # Gitaly's [gitlab] url (the internal-API callback every Gitaly node
+  # uses for gitlab-shell hooks) - a real, TLS-supported override,
+  # confirmed live via gitlab/charts/gitlab/templates/_workhorse.tpl.
+  # Uses the SAME floating-hostname + CoreDNS-local-rewrite pattern
+  # already proven for Postgres and Gitaly's own node addresses: home's
+  # 3 Gitaly nodes resolve this locally (no WAN round-trip), the VPS's
+  # 4th node resolves it via the UniFi WAN forward - one consistent
+  # URL for every node regardless of site.
+  #
+  # tls.enabled REVERTED 2026-08-22 - broke the live site (HTTP 400
+  # "Client sent an HTTP request to an HTTPS server" on
+  # gitlab.kube.huskypup.net). Root cause: port 8181 is NOT internal-API
+  # -only - confirmed live via `kubectl get httproute gitlab-web` that
+  # ALL public traffic (gitlab.kube.huskypup.net, routed through Envoy/
+  # Istio edge gateway) also targets gitlab-webservice-default:8181.
+  # global.workhorse.tls.enabled replaces the plaintext listener with
+  # TLS on that SAME port chart-wide, for both the public site AND the
+  # internal API - there's no chart-native way to TLS one without the
+  # other. Envoy terminates public HTTPS and proxies to the pod in
+  # plaintext, so a TLS-only pod listener 400s every public request.
+  # host: left in place (harmless without tls.enabled - only used for
+  # the [gitlab] url string, not port selection). Internal-API-over-WAN
+  # transport security for the VPS's future Gitaly node needs a
+  # different mechanism (e.g. network-layer TLS/mTLS via Istio, or a
+  # second dedicated port if the chart ever adds one) - not solved here.
+  workhorse:
+    host: gitlab-internal-api.ha.huskypup.net
+
   # Praefect configuration
   praefect:
     enabled: true
@@ -68,6 +111,12 @@ global:
     - name: default
       gitalyReplicas: 3  # Production HA
       maxUnavailable: 1
+      # Required once global.gitaly.tls.enabled: true - the chart's own
+      # NOTES.txt validation rejects the render without it ("not
+      # specified ('default')... not supported", confirmed live).
+      # Matches gitlab.gitaly.tls.secret's default name/keys
+      # (gitaly-tls-certificate.yaml issues into this exact secret).
+      tlsSecretName: gitlab-gitaly-tls
     
     # Praefect PostgreSQL configuration
     psql:
@@ -285,14 +334,14 @@ gitlab:
     enabled: true
     minReplicas: 1
     maxReplicas: 1
-    
+
     # Note: Praefect PostgreSQL config is in global.praefect.psql
-    
+
     # Use CNPG-generated database secret
     dbSecret:
       secret: pg-praefect-app
       key: password
-    
+
     # Resources
     resources:
       requests:
@@ -301,13 +350,37 @@ gitlab:
       limits:
         cpu: 250m
         memory: 512Mi
-    
+
-    # Virtual storage configuration
+    # (dead config removed 2026-08-22: this virtualStorages block never
-    virtualStorages:
+    # actually did anything - confirmed live the chart's Praefect
-    - name: default
+    # ConfigMap template iterates global.praefect.virtualStorages, not
-      gitalyReplicas: 1  # Homelab sizing
+    # this one, so gitalyReplicas: 3 up there always won regardless of
-      maxUnavailable: 1
+    # what this said. See global.praefect.virtualStorages above for the
-  
+    # real config.)
+    #
+    # GitLab cross-site replication (see
+    # /home/scooby/.claude/plans/jiggly-snacking-iverson.md): tried a
+    # values-only CONFIG_TEMPLATE_DIRECTORY redirect (extraEnv +
+    # extraVolumes/extraVolumeMounts) to add a VPS-hosted Gitaly node the
+    # chart has no mechanism to register - confirmed live this doesn't
+    # work: extraVolumeMounts and extraEnv both render correctly, but
+    # extraVolumes is a dead values key for THIS subchart specifically -
+    # charts/gitlab/charts/praefect/templates/statefulset.yaml's own
+    # volumes: list never calls the gitlab.extraVolumes helper at all
+    # (only volumeMounts does), so the mount has nothing to mount and the
+    # pod would fail to schedule. No values-only fix exists.
+    #
+    # Real fix: argocd-apps/apps/gitlab.yaml's ignoreDifferences now
+    # covers ConfigMap gitlab-praefect's `data` field, so Helm creates
+    # the object (with its own 3-node content) but ArgoCD stops
+    # reconciling its content afterward. The actual multi-node content
+    # lives in apps/gitlab/manifests/praefect-ha-configmap.yaml as a
+    # git-tracked reference/documentation copy (deployed under its own
+    # name, praefect-ha-config, harmless and otherwise unused) - the
+    # live gitlab-praefect ConfigMap gets kubectl-patched with that same
+    # content directly, same "documented but manually-applied"
+    # convention as every other cross-cluster secret in this plan.
+
   # GitLab Exporter for Prometheus metrics
   gitlab-exporter:
     enabled: true

argocd-apps/apps/gitlab.yaml

@@ -33,6 +33,19 @@ spec:
       jsonPointers:
         - /spec/ports
         - /spec/selector
+    # GitLab cross-site replication (see
+    # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - the chart
+    # has no mechanism to register an externally-hosted (VPS) Gitaly node
+    # into Praefect's virtual storage. Lets the live ConfigMap's `data`
+    # be kubectl-patched directly (see apps/gitlab/manifests/
+    # praefect-ha-configmap.yaml for the content + patch command) without
+    # ArgoCD reverting it on the next sync - Helm still creates the
+    # object and everything else about it stays normally managed.
+    - group: ""
+      kind: ConfigMap
+      name: gitlab-praefect
+      jsonPointers:
+        - /data
   syncPolicy:
     automated:
       prune: true

infrastructure/cert-manager/manifests/letsencrypt-issuer.yaml

@@ -1,15 +1,15 @@
----
+---
-apiVersion: cert-manager.io/v1
+apiVersion: cert-manager.io/v1
-kind: ClusterIssuer
+kind: ClusterIssuer
-metadata:
+metadata:
-  name: letsencrypt-production
+  name: letsencrypt-production
-spec:
+spec:
-  acme:
+  acme:
-    server: https://acme-v02.api.letsencrypt.org/directory
+    server: https://acme-v02.api.letsencrypt.org/directory
-    email: garrettstone499@gmail.com
+    email: garrettstone499@gmail.com
-    privateKeySecretRef:
+    privateKeySecretRef:
-      name: letsencrypt-production
+      name: letsencrypt-production
-    solvers:
+    solvers:
       - dns01:
           cloudflare:
             email: garrettstone499@gmail.com
@@ -19,6 +19,17 @@ spec:
         selector:
           dnsZones:
             - "kube.huskypup.net"
+            # GitLab cross-site replication (see
+            # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) -
+            # first real Let's Encrypt cert needed for anything under
+            # ha.huskypup.net (the floating-hostname zone used for
+            # cross-site Postgres/Gitaly addressing) - everything
+            # before this used CNPG's own self-signed certs, never hit
+            # this gap. Confirmed live: DNS-01 order failed with "no
+            # configured challenge solvers can be used for this
+            # challenge" until this zone was added. Same Cloudflare
+            # account/token as kube.huskypup.net.
+            - "ha.huskypup.net"
 
 ---
 apiVersion: cert-manager.io/v1

infrastructure/coredns/manifests/coredns-configmap.yaml

@@ -67,6 +67,21 @@ data:
         rewrite name pg-gitlab.ha.huskypup.net ha-gitlab-postgres.gitlab.svc.cluster.local
         rewrite name pg-praefect.ha.huskypup.net ha-praefect-postgres.gitlab.svc.cluster.local
 
+        # GitLab cross-site replication Phase 2b (Gitaly) - home's own
+        # Praefect dials these SAME floating hostnames it hands to the
+        # VPS's Gitaly node as peer addresses (Praefect uses one address
+        # per node for everyone, not different addresses per caller), so
+        # home's own in-cluster traffic to its own local Gitaly nodes
+        # needs to resolve locally too, same reasoning as pg-*.ha above.
+        rewrite name gitaly-0.ha.huskypup.net ha-gitaly-0.gitlab.svc.cluster.local
+        rewrite name gitaly-1.ha.huskypup.net ha-gitaly-1.gitlab.svc.cluster.local
+        rewrite name gitaly-2.ha.huskypup.net ha-gitaly-2.gitlab.svc.cluster.local
+
+        # global.workhorse.host override (apps/gitlab/values.yaml) - every
+        # Gitaly node's [gitlab] url, home's own included, same local-
+        # rewrite reasoning as the entries above.
+        rewrite name gitlab-internal-api.ha.huskypup.net ha-gitlab-internal-api.gitlab.svc.cluster.local
+
         kubernetes cluster.local in-addr.arpa ip6.arpa {
             pods insecure
             fallthrough in-addr.arpa ip6.arpa

infrastructure/istio/manifests/apps/authorization-policies-apps.yaml

@@ -100,6 +100,8 @@ spec:
               - "5432"
               - "61442" # pg-gitlab floating-hostname port, see ha-postgres-nodeport.yaml
               - "61443" # pg-praefect floating-hostname port, see ha-postgres-nodeport.yaml
+              - "8075" # Gitaly gRPC, see ha-gitaly-nodeport.yaml
+              - "8181" # Rails internal API, see ha-gitlab-internal-api-nodeport.yaml
 
 ---
 # --- n8n: Allow ingress + Prometheus + intra-namespace ---

infrastructure/vps-standby/gitlab/manifests/cnpg-cluster.yaml

@@ -65,6 +65,22 @@ spec:
     recovery:
       source: home
 
+  # Stage 2 (see /home/scooby/.claude/plans/jiggly-snacking-iverson.md):
+  # connectionParameters added alongside the existing barmanObjectStore
+  # on the SAME "home" entry - the recovery-mode initial seed stays
+  # exactly as-is, ongoing replication now streams instead of relying on
+  # WAL-archive polling (the source of the crash-looping/probe-tuning
+  # fights documented for the pg-authentik precedent). Also needs a
+  # self-referencing "vps" entry - replica.self must match an
+  # externalClusters name too, even though it's this very cluster
+  # (confirmed live: "External cluster vps not found" until one was
+  # added).
+  #
+  # Reachable via home.kube.huskypup.net:61442 - a UniFi WAN port-forward
+  # to talos-cp-01's real LAN IP, source-IP-restricted to the VPS/witness
+  # public IPs (see apps/gitlab/manifests/ha-postgres-nodeport.yaml). NOT
+  # the Netbird mesh directly - UniFi has no route to that CIDR
+  # (confirmed live for the pg-authentik precedent).
   externalClusters:
     - name: home
       barmanObjectStore:
@@ -84,6 +100,45 @@ spec:
           secretAccessKey:
             name: vps-minio-secret
             key: secretkey
+      connectionParameters:
+        host: home.kube.huskypup.net
+        port: "61442"
+        dbname: gitlabhq_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-gitlab-home-replication
+        key: tls.crt
+      sslKey:
+        name: pg-gitlab-home-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-gitlab-home-ca
+        key: ca.crt
+    - name: vps
+      connectionParameters:
+        host: pg-gitlab-rw
+        port: "5432"
+        dbname: gitlabhq_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-gitlab-replication
+        key: tls.crt
+      sslKey:
+        name: pg-gitlab-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-gitlab-ca
+        key: ca.crt
+
+  # Distributed topology: both sides agree home is primary today. No
+  # replica.enabled - mutually exclusive with primary/self (confirmed
+  # live for the pg-authentik precedent).
+  replica:
+    self: vps
+    primary: home
+    source: home
 
   monitoring:
     enablePodMonitor: false  # no Prometheus on the VPS cluster

infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml

@@ -0,0 +1,62 @@
+# GitLab cross-site replication - VPS's half of the Vault pipeline
+# (apps/gitlab/manifests/ha-postgres-app-pushsecret.yaml has home's half).
+# Pulls the authoritative app-role passwords back down here.
+#
+# creationPolicy REVERTED Merge -> Owner 2026-08-22: Merge assumed CNPG
+# itself would generate a base pg-gitlab-app/pg-praefect-app secret
+# (host/dbname/username keys) for this ExternalSecret to merge a
+# password into - true for a normal owning/primary Cluster, but WRONG
+# for these VPS Clusters since Phase 1c's recreate: `pg-gitlab`/
+# `pg-praefect` here are bootstrap.recovery REPLICA clusters
+# (replica.self: vps) - the `app` role is part of the replicated
+# global role/password state streamed from home's primary, so CNPG
+# correctly never creates a local owner secret for it here. Merge had
+# nothing to merge onto -> ExternalSecret condition
+# "SecretMissing/CreationPolicy=Merge" -> pg-gitlab-app never existed
+# -> vps-gitlab-webservice crash-looped on
+# ActiveRecord::DatabaseConnectionError for ~39h before this was
+# caught (confirmed live, unrelated to any same-day change). Owner:
+# the ExternalSecret creates the whole secret itself. This chart only
+# ever reads the `password` key from it - host/port/dbname/username
+# are hardcoded directly in infrastructure/vps-standby/gitlab/
+# values.yaml's global.psql/global.praefect.psql blocks, not read from
+# this secret's other keys, so a password-only Owner-created secret is
+# sufficient (confirmed live: no other consumer of this secret's keys
+# found).
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: pg-gitlab-app-password
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: pg-gitlab-app
+    creationPolicy: Owner
+  data:
+    - secretKey: password
+      remoteRef:
+        key: vps/pg-gitlab-app
+        property: password
+---
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: pg-praefect-app-password
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: pg-praefect-app
+    creationPolicy: Owner
+  data:
+    - secretKey: password
+      remoteRef:
+        key: vps/pg-praefect-app
+        property: password

infrastructure/vps-standby/gitlab/manifests/praefect-cnpg-cluster.yaml

@@ -52,6 +52,8 @@ spec:
     recovery:
       source: home
 
+  # Stage 2 - same pattern as cnpg-cluster.yaml (pg-gitlab) in this same
+  # directory.
   externalClusters:
     - name: home
       barmanObjectStore:
@@ -65,6 +67,42 @@ spec:
           secretAccessKey:
             name: vps-minio-secret
             key: secretkey
+      connectionParameters:
+        host: home.kube.huskypup.net
+        port: "61443"
+        dbname: praefect_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-praefect-home-replication
+        key: tls.crt
+      sslKey:
+        name: pg-praefect-home-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-praefect-home-ca
+        key: ca.crt
+    - name: vps
+      connectionParameters:
+        host: pg-praefect-rw
+        port: "5432"
+        dbname: praefect_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-praefect-replication
+        key: tls.crt
+      sslKey:
+        name: pg-praefect-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-praefect-ca
+        key: ca.crt
+
+  replica:
+    self: vps
+    primary: home
+    source: home
 
   monitoring:
     enablePodMonitor: false  # no Prometheus on the VPS cluster

infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml

@@ -16,8 +16,20 @@
 #   kubectl -n nextcloud create secret generic nextcloud-restic-password \
 #     --from-literal=password=<home's nextcloud-restic-password>
 #
-# Nextcloud itself isn't running (replicas: 0, see values.yaml) so there's
+# CORRECTED 2026-08-22: the "replicas: 0" assumption below was wrong -
-# no live-write conflict risk overwriting the PVC on every run.
+# confirmed live vps-nextcloud actually runs continuously at replicas: 1
+# (no replicas key was ever set in values.yaml; the chart's own default
+# applies). This job had been silently failing every run for 2+ days as
+# a result: restic couldn't overwrite the live pod's config.php (mode
+# 0640, owned by www-data/33; this job runs as UID 65534 with only
+# group 33 via fsGroup, so it can read but not write that one file) -
+# "ignoring error for /data/config/config.php: permission denied",
+# Fatal: 1 error, even though the other 31000+ files/15GiB restored
+# fine every time. Fixed by excluding config.php from the restore
+# below rather than chasing permissions - overwriting a live, running
+# app's own config.php from a background restore job is the wrong
+# move regardless (it holds this site's actual current db credentials,
+# etc.), not just a permissions bug to route around.
 apiVersion: batch/v1
 kind: CronJob
 metadata:
@@ -64,7 +76,7 @@ spec:
                   fi
 
                   echo "==> Restoring latest snapshot into /data..."
-                  restic restore latest --tag nextcloud --host nextcloud-k8s --target /
+                  restic restore latest --tag nextcloud --host nextcloud-k8s --target / --exclude /data/config/config.php
 
                   echo "==> Done."
               env: