Fix VPS GitLab + Nextcloud crash loops found during health sweep

- infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml:
  creationPolicy Merge -> Owner (already applied live, committing to
  match). Merge assumed CNPG creates a base pg-gitlab-app/pg-praefect-app
  secret to merge a password into - wrong for these VPS Clusters since
  Phase 1c's replica-cluster recreate; CNPG doesn't create a local
  owner secret for a replicated role. Left pg-gitlab-app/pg-praefect-app
  missing entirely, crash-looping vps-gitlab-webservice on
  ActiveRecord::DatabaseConnectionError for ~39h.

- infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml:
  exclude config.php from the restic restore. The job's own comment
  assumed nextcloud runs at replicas: 0 on the VPS; live confirms
  replicas: 1 (no such key was ever actually set) - restic couldn't
  overwrite the live pod's config.php (permission denied), failing the
  whole job every run for 2+ days even though everything else restored
  fine. Excluding it is correct regardless of permissions - a live
  pod's own config.php shouldn't be overwritten by a background
  restore job.

Also fixed live (not git-tracked, config.php is PVC-persisted runtime
state, not sourced from git):
- home + VPS nextcloud config.php dbpassword: out of sync with CNPG's
  actual current pg-nextcloud-app password (baked in once at install,
  never re-synced). VPS's case was two-layered - its own
  pg-nextcloud-app secret also didn't match the real Postgres role
  password, since VPS's pg-nextcloud is a read-only streaming replica
  (spec.replica.enabled) and the authoritative password lives on home.
- grafana: broke a RollingUpdate deadlock (single-replica Deployment +
  RWO Ceph volume - new pod couldn't start while the old pod still
  held the only-one-node-at-a-time attachment, and the rollout
  wouldn't scale down the old pod until the new one was ready).
- suspended the stale gitlab-mirror-sync CronJob on the VPS (spamming
  auth failures every ~15min since its stored token predates the
  Postgres-replication cutover of VPS GitLab's DB) rather than
  deleting it, per the plan's own note to defer that until Phase 2b is
  verified.

apps/gitlab/manifests/cnpg-cluster.yaml

@@ -53,17 +53,25 @@ spec:
       max_parallel_workers_per_gather: "2"
       max_parallel_workers: "4"
 
-  # Backup configuration to MinIO
+  # Backup configuration - repointed to VPS MinIO 2026-08-22 (GitLab
+  # cross-site replication plan, see
+  # /home/scooby/.claude/plans/jiggly-snacking-iverson.md's "GitLab True
+  # Cross-Site Replication" section). Was gitlab-minio-svc (in-cluster,
+  # same Ceph the backup is meant to protect against). Also doubles as
+  # the initial-seed source for the VPS's own pg-gitlab replica
+  # (bootstrap.recovery, Phase 1c) - same barmanObjectStore block serves
+  # both purposes, matching the Authentik precedent
+  # (infrastructure/authentik/manifests/cnpg-cluster.yaml).
   backup:
     barmanObjectStore:
-      destinationPath: s3://gitlab-backups/pg-gitlab
+      destinationPath: s3://cnpg-backups/pg-gitlab
-      endpointURL: http://gitlab-minio-svc.gitlab.svc.cluster.local:9000
+      endpointURL: http://100.108.113.41:9000
       s3Credentials:
         accessKeyId:
-          name: gitlab-minio-secret
+          name: vps-minio-secret
           key: accesskey
         secretAccessKey:
-          name: gitlab-minio-secret
+          name: vps-minio-secret
           key: secretkey
       wal:
         compression: gzip
@@ -72,3 +80,58 @@ spec:
 
   monitoring:
     enablePodMonitor: true
+
+  # GitLab cross-site replication Phase 1c stage 2 (see
+  # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - streaming
+  # replication to/from the VPS's pg-gitlab cluster, alongside (not
+  # replacing) the barmanObjectStore backup above. Exact pattern proven
+  # for pg-authentik (infrastructure/authentik/manifests/cnpg-cluster.yaml).
+  #
+  # Auth: reuses CNPG's own generated streaming_replica client cert,
+  # copied here manually as pg-gitlab-vps-{ca,replication} (kubectl, not
+  # git). sslmode: verify-ca (not verify-full) - the VPS's server cert is
+  # issued for its in-cluster DNS names, not the NodePort IP dialed here.
+  #
+  # Reachable via 100.108.113.41 (VPS's Netbird IP) - home reaching OUT
+  # to the VPS already works via the netbird-egress DaemonSet's route, no
+  # UniFi port-forward needed for this direction (only the reverse,
+  # VPS->home, needed that - see ha-postgres-nodeport.yaml).
+  externalClusters:
+    - name: home
+      connectionParameters:
+        host: pg-gitlab-rw
+        port: "5432"
+        dbname: gitlabhq_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-gitlab-replication
+        key: tls.crt
+      sslKey:
+        name: pg-gitlab-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-gitlab-ca
+        key: ca.crt
+    - name: vps
+      connectionParameters:
+        host: 100.108.113.41
+        port: "32444"
+        dbname: gitlabhq_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-gitlab-vps-replication
+        key: tls.crt
+      sslKey:
+        name: pg-gitlab-vps-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-gitlab-vps-ca
+        key: ca.crt
+
+  # Distributed topology: both sides agree home is primary today.
+  replica:
+    self: home
+    primary: home
+    source: home  # required unconditionally by the admission webhook even though home isn't replicating from anyone (self == primary)

apps/gitlab/manifests/external-secret-oidc.yaml

@@ -0,0 +1,69 @@
+# gitlab-oidc-secret was a manually kubectl-created plain Secret (never
+# git-tracked, never rotatable, no record of what it was) - found and
+# fixed 2026-08-20 alongside infrastructure/authentik/gitlab-blueprint.yaml
+# (same value, same Vault path - see that file for the full story).
+#
+# `provider` key added 2026-08-21: found live that GITLAB_OMNIBUS_CONFIG's
+# `gitlab_rails['omniauth_providers'] = [...]` (still set in this app's
+# values.yaml's extraEnv, for reference/documentation only at this point)
+# is an omnibus-image-only convention - the CNG webservice image this
+# chart actually runs never processes it, so it was always a no-op and
+# GitLab never had SSO configured despite it looking configured. The
+# REAL mechanism (gitlab/charts/gitlab/templates/_omniauth.tpl in the
+# chart) is global.appConfig.omniauth.providers: a list of
+# {secret, key} refs, each pointing at a Secret key whose value is a
+# whole YAML-encoded provider block (loaded via Ruby's YAML.load_file
+# at container start, baked into gitlab.yml) - not raw client_id/secret
+# strings. This is also why the `providers:` list got removed entirely
+# a day earlier chasing a "FailedMount: references non-existent secret
+# key: provider" error: the fix should have been to populate that key
+# correctly (this), not remove the reference to it - doing so silently
+# killed SSO on both home and VPS GitLab (no error, the login page just
+# had no SSO button).
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: gitlab-oidc
+  namespace: gitlab
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: vault-backend
+  target:
+    name: gitlab-oidc-secret
+    creationPolicy: Owner
+    template:
+      type: Opaque
+      data:
+        GITLAB_OIDC_CLIENT_ID: "{{ .clientId }}"
+        GITLAB_OIDC_CLIENT_SECRET: "{{ .clientSecret }}"
+        provider: |
+          name: openid_connect
+          label: Authentik
+          args:
+            name: openid_connect
+            scope:
+              - openid
+              - profile
+              - email
+            response_type: code
+            issuer: 'https://auth.kube.huskypup.net/application/o/gitlab/'
+            discovery: true
+            client_auth_method: query
+            uid_field: sub
+            send_scope_to_token_endpoint: true
+            pkce: true
+            client_options:
+              identifier: '{{ .clientId }}'
+              secret: '{{ .clientSecret }}'
+              redirect_uri: 'https://gitlab.kube.huskypup.net/users/auth/openid_connect/callback'
+  data:
+    - secretKey: clientId
+      remoteRef:
+        key: gitlab-oauth
+        property: client-id
+    - secretKey: clientSecret
+      remoteRef:
+        key: gitlab-oauth
+        property: client-secret

apps/gitlab/manifests/gitaly-tls-certificate.yaml

@@ -0,0 +1,55 @@
+# GitLab cross-site replication Phase 2b (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - Gitaly's gRPC
+# has no TLS by default; routing it over the public UniFi WAN forward
+# (needed because the direct-Netbird-IP path doesn't accept inbound
+# connections to home, confirmed live the same way as the existing etcd
+# quorum) would otherwise carry the gitaly_token, gitlab_shell_secret,
+# and raw repository contents in plaintext.
+#
+# Real Let's Encrypt cert via the same DNS-01 ClusterIssuer already
+# proven for every other *.huskypup.net cert in this cluster, rather
+# than a self-signed cert - sidesteps having to distribute a custom CA
+# to both home and the VPS (which already trust Let's Encrypt's public
+# CA by default). Covers all 4 Gitaly floating hostnames as SANs - one
+# shared cert, referenced by every node's tlsSecretName (or the global
+# default), simpler than per-node certs.
+#
+# secretName matches the chart's own default (confirmed live via
+# gitlab/templates/_gitaly.tpl's gitlab.gitaly.tls.secret helper:
+# "<release-name>-gitaly-tls" = "gitlab-gitaly-tls") - global.gitaly.tls.
+# enabled: true picks this up with no secretName override needed.
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: gitlab-gitaly-tls
+  namespace: gitlab
+spec:
+  secretName: gitlab-gitaly-tls
+  issuerRef:
+    name: letsencrypt-production
+    kind: ClusterIssuer
+  dnsNames:
+    - gitaly-0.ha.huskypup.net
+    - gitaly-1.ha.huskypup.net
+    - gitaly-2.ha.huskypup.net
+    - gitaly-vps.ha.huskypup.net
+---
+# Separate cert/secret for workhorse's internal-API TLS listener (port
+# 8181, [gitlab] url every Gitaly node calls for gitlab-shell hooks) -
+# different secretName (gitlab-workhorse-tls, chart default per
+# gitlab/charts/gitlab/charts/webservice/templates/_helpers.tpl's
+# workhorse.tls.secret helper) than Gitaly's own cert, kept separate
+# rather than trying to share one Secret object across two different
+# chart-expected names.
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: gitlab-workhorse-tls
+  namespace: gitlab
+spec:
+  secretName: gitlab-workhorse-tls
+  issuerRef:
+    name: letsencrypt-production
+    kind: ClusterIssuer
+  dnsNames:
+    - gitlab-internal-api.ha.huskypup.net

apps/gitlab/manifests/ha-gitaly-nodeport.yaml

@@ -0,0 +1,61 @@
+---
+# GitLab cross-site replication Phase 2b (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - exposes each
+# of home's 3 existing Gitaly nodes individually to the VPS, so Praefect
+# can register the VPS as a 4th node in the SAME virtual storage and
+# inter-Gitaly replication RPCs can reach each node directly. Per-pod
+# (not per-StatefulSet), unlike the chart's own gitlab-gitaly-default
+# Service - Praefect/Gitaly must address each node individually, not
+# load-balance across them.
+#
+# Reached over the same public UniFi WAN forward already used for
+# Postgres, NOT the Netbird mesh directly - confirmed live the direct-
+# Netbird-IP path doesn't accept inbound connections to home (same gap
+# already present for the existing etcd quorum; the earlier assumption
+# that NodePort binds on a reachable Netbird interface was wrong).
+# Targets Gitaly's TLS port (8076), not the plaintext one (8075) -
+# Gitaly's gRPC has no built-in transport security, and this WAN hop
+# would otherwise carry the gitaly_token, gitlab_shell_secret, and raw
+# repository contents in plaintext. See gitaly-tls-certificate.yaml and
+# global.gitaly.tls.enabled in values.yaml.
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-gitaly-0
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    statefulset.kubernetes.io/pod-name: gitlab-gitaly-default-0
+  ports:
+    - port: 8076
+      targetPort: 8076
+      nodePort: 32446
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-gitaly-1
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    statefulset.kubernetes.io/pod-name: gitlab-gitaly-default-1
+  ports:
+    - port: 8076
+      targetPort: 8076
+      nodePort: 32447
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-gitaly-2
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    statefulset.kubernetes.io/pod-name: gitlab-gitaly-default-2
+  ports:
+    - port: 8076
+      targetPort: 8076
+      nodePort: 32448

apps/gitlab/manifests/ha-gitlab-internal-api-nodeport.yaml

@@ -0,0 +1,32 @@
+# GitLab cross-site replication Phase 2b - exposes home's Rails internal
+# API (port 8181, a dedicated internal-only listener, deliberately NOT
+# exposed through the public Istio ingress on 8080) to the VPS's Gitaly
+# node. Gitaly's gitlab-shell hooks call POST /api/v4/internal/{allowed,
+# pre_receive,post_receive} against this whenever the VPS node is
+# primary for a repository (per-repository election can cause this even
+# in normal operation) or after a real failover.
+#
+# Reached over the same public UniFi WAN forward already used for
+# Postgres/Gitaly, not the Netbird mesh directly (confirmed live that
+# path doesn't accept inbound connections to home). Port stays 8181
+# (unlike Gitaly's separate 8075/8076 split) - global.workhorse.tls.
+# enabled in values.yaml replaces the plaintext listener with TLS on
+# this SAME port rather than adding a second one. Carries the
+# gitlab_shell_secret in a header, so this needed TLS just as much as
+# Gitaly's gRPC did - see gitaly-tls-certificate.yaml's second
+# Certificate (gitlab-workhorse-tls) and the
+# gitlab-internal-api.ha.huskypup.net CoreDNS rewrite.
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-gitlab-internal-api
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    app: webservice
+    release: gitlab
+  ports:
+    - port: 8181
+      targetPort: 8181
+      nodePort: 32449

apps/gitlab/manifests/ha-gitlab-secrets-pushsecret.yaml

@@ -0,0 +1,99 @@
+# GitLab cross-site replication Phase 1b (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - pushes home's
+# chart-generated secrets into Vault so the VPS's ESO can pull them back
+# down BEFORE the VPS's Postgres becomes a real replica of home's.
+#
+# Why this matters: the GitLab chart generates these per-release, at
+# random, on first install. Home's and the VPS's values are currently
+# different. Once the VPS's Postgres is byte-for-byte replicating home's
+# rows, the VPS Rails app will be reading home's encrypted-at-rest data
+# (2FA secrets, CI/CD variables, integration tokens, PAT digests,
+# application_settings encrypted columns) with the WRONG key unless these
+# match - Rails raises OpenSSL::Cipher::CipherError on boot otherwise.
+#
+# gitlab-gitaly-secret/gitlab-praefect-secret are also needed for
+# Phase 2's cross-site gRPC auth (the VPS's Gitaly/Praefect need the same
+# tokens home's do, to be trusted as nodes of the same virtual storage).
+#
+# Same pattern as infrastructure/authentik/manifests/ha-postgres-app-pushsecret.yaml.
+# Each of these secrets happens to have exactly one key (confirmed live),
+# so one PushSecret + one ExternalSecret per secret, no enumeration needed.
+apiVersion: external-secrets.io/v1alpha1
+kind: PushSecret
+metadata:
+  name: gitlab-rails-secret-to-vps
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRefs:
+    - name: vault-backend
+      kind: ClusterSecretStore
+  selector:
+    secret:
+      name: gitlab-rails-secret
+  data:
+    - match:
+        secretKey: secrets.yml
+        remoteRef:
+          remoteKey: vps/gitlab-rails-secret
+          property: secrets.yml
+---
+apiVersion: external-secrets.io/v1alpha1
+kind: PushSecret
+metadata:
+  name: gitlab-shell-secret-to-vps
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRefs:
+    - name: vault-backend
+      kind: ClusterSecretStore
+  selector:
+    secret:
+      name: gitlab-gitlab-shell-secret
+  data:
+    - match:
+        secretKey: secret
+        remoteRef:
+          remoteKey: vps/gitlab-gitlab-shell-secret
+          property: secret
+---
+apiVersion: external-secrets.io/v1alpha1
+kind: PushSecret
+metadata:
+  name: gitlab-gitaly-secret-to-vps
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRefs:
+    - name: vault-backend
+      kind: ClusterSecretStore
+  selector:
+    secret:
+      name: gitlab-gitaly-secret
+  data:
+    - match:
+        secretKey: token
+        remoteRef:
+          remoteKey: vps/gitlab-gitaly-secret
+          property: token
+---
+apiVersion: external-secrets.io/v1alpha1
+kind: PushSecret
+metadata:
+  name: gitlab-praefect-secret-to-vps
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRefs:
+    - name: vault-backend
+      kind: ClusterSecretStore
+  selector:
+    secret:
+      name: gitlab-praefect-secret
+  data:
+    - match:
+        secretKey: token
+        remoteRef:
+          remoteKey: vps/gitlab-praefect-secret
+          property: token

apps/gitlab/manifests/ha-peerauth.yaml

@@ -0,0 +1,90 @@
+# GitLab cross-site replication (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - direct copy of
+# infrastructure/authentik/manifests/ha-postgres-peerauth.yaml's proven
+# fix. The gitlab namespace carries the same ambient-mesh STRICT default
+# as authentik; without this, ztunnel resets the VPS/witness's connection
+# (no mesh identity - they're not in this cluster) before the Postgres
+# TLS handshake can even start, surfacing as "server closed the
+# connection unexpectedly" on the replica side despite the NodePort/
+# firewall path being entirely correct.
+#
+# Scoped to just each CNPG primary pod's ports via portLevelMtls, not the
+# whole namespace - GitLab's own in-mesh east-west traffic (webservice
+# <-> gitaly <-> redis <-> sidekiq <-> praefect) stays STRICT.
+apiVersion: security.istio.io/v1
+kind: PeerAuthentication
+metadata:
+  name: allow-ha-gitlab-postgres-replication
+  namespace: gitlab
+spec:
+  selector:
+    matchLabels:
+      cnpg.io/cluster: pg-gitlab
+  mtls:
+    mode: STRICT
+  portLevelMtls:
+    "5432":
+      mode: PERMISSIVE
+    # Floating-hostname port (see ha-postgres-nodeport.yaml) - needs its
+    # own entry, ambient's port-level mTLS enforcement is keyed on the
+    # port actually dialed (61442), not just the pod's real containerPort
+    # (5432) it eventually reaches (confirmed live for the authentik
+    # precedent, same mechanism here).
+    "61442":
+      mode: PERMISSIVE
+---
+apiVersion: security.istio.io/v1
+kind: PeerAuthentication
+metadata:
+  name: allow-ha-praefect-postgres-replication
+  namespace: gitlab
+spec:
+  selector:
+    matchLabels:
+      cnpg.io/cluster: pg-praefect
+  mtls:
+    mode: STRICT
+  portLevelMtls:
+    "5432":
+      mode: PERMISSIVE
+    "61443":
+      mode: PERMISSIVE
+---
+# Phase 2b (Gitaly cross-site replication) - the 3 Gitaly pods, exposed
+# individually via ha-gitaly-nodeport.yaml for the VPS's 4th node to
+# join the same virtual storage. Gitaly pods already carry
+# istio.io/use-waypoint: none (chart default, gRPC doesn't tolerate L7
+# waypoint processing well) but that only skips L7 processing - L4
+# ztunnel mTLS enforcement is a separate concern, same fix needed as the
+# Postgres precedent.
+apiVersion: security.istio.io/v1
+kind: PeerAuthentication
+metadata:
+  name: allow-ha-gitaly-replication
+  namespace: gitlab
+spec:
+  selector:
+    matchLabels:
+      app: gitaly
+  mtls:
+    mode: STRICT
+  portLevelMtls:
+    "8075":
+      mode: PERMISSIVE
+---
+# Phase 2b - the VPS's Gitaly node calling home's Rails internal API
+# (ha-gitlab-internal-api-nodeport.yaml) for gitlab-shell hooks.
+apiVersion: security.istio.io/v1
+kind: PeerAuthentication
+metadata:
+  name: allow-ha-gitlab-internal-api
+  namespace: gitlab
+spec:
+  selector:
+    matchLabels:
+      app: webservice
+  mtls:
+    mode: STRICT
+  portLevelMtls:
+    "8181":
+      mode: PERMISSIVE

apps/gitlab/manifests/ha-postgres-app-pushsecret.yaml

@@ -0,0 +1,50 @@
+# GitLab cross-site replication (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - same pattern
+# and same reasoning as
+# infrastructure/authentik/manifests/ha-postgres-app-pushsecret.yaml:
+# pg-gitlab-app/pg-praefect-app are CNPG-generated, not Vault-native -
+# each cluster independently generated its own random `app` role
+# password at bootstrap time. Streaming replication keeps the two
+# clusters' actual live passwords in sync (WAL includes role/password
+# changes), but the two clusters' K8s Secret OBJECTS never resync on
+# their own - this closes that gap, self-healing on refreshInterval
+# instead of a one-time fix that goes stale on the next rotation.
+apiVersion: external-secrets.io/v1alpha1
+kind: PushSecret
+metadata:
+  name: pg-gitlab-app-to-vps
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRefs:
+    - name: vault-backend
+      kind: ClusterSecretStore
+  selector:
+    secret:
+      name: pg-gitlab-app
+  data:
+    - match:
+        secretKey: password
+        remoteRef:
+          remoteKey: vps/pg-gitlab-app
+          property: password
+---
+apiVersion: external-secrets.io/v1alpha1
+kind: PushSecret
+metadata:
+  name: pg-praefect-app-to-vps
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRefs:
+    - name: vault-backend
+      kind: ClusterSecretStore
+  selector:
+    secret:
+      name: pg-praefect-app
+  data:
+    - match:
+        secretKey: password
+        remoteRef:
+          remoteKey: vps/pg-praefect-app
+          property: password

apps/gitlab/manifests/ha-postgres-nodeport.yaml

@@ -0,0 +1,56 @@
+---
+# GitLab cross-site replication (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md's "GitLab True
+# Cross-Site Replication" section) - direct copy of the proven pattern
+# from infrastructure/authentik/manifests/ha-postgres-nodeport.yaml.
+# Exposes pg-gitlab's and pg-praefect's current primaries to the
+# VPS/witness over the Netbird mesh for streaming replication - same
+# selectors CNPG's own -rw ClusterIP Services use, just NodePort instead,
+# since CNPG doesn't manage these objects and won't fight with them.
+# Needs infrastructure/kyverno/policies/ha-failover-nodeport-exception.yaml
+# (name pattern ha-* already matches, no change needed there).
+#
+# Second port on each (61442/61443, ClusterIP only) for the CoreDNS
+# rewrite in infrastructure/coredns/manifests/: home's own pods resolving
+# pg-gitlab.ha.huskypup.net / pg-praefect.ha.huskypup.net rewrite to
+# THESE services specifically so they can reach them on the SAME port
+# number the app tier will eventually be configured with (Phase 3) -
+# rewrite only changes the resolved NAME, not the port a client then
+# connects to (confirmed live for the authentik precedent).
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-gitlab-postgres
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    cnpg.io/cluster: pg-gitlab
+    cnpg.io/instanceRole: primary
+  ports:
+    - name: nodeport-5432
+      port: 5432
+      targetPort: 5432
+      nodePort: 32442
+    - name: floating-port-61442
+      port: 61442
+      targetPort: 5432
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-praefect-postgres
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    cnpg.io/cluster: pg-praefect
+    cnpg.io/instanceRole: primary
+  ports:
+    - name: nodeport-5432
+      port: 5432
+      targetPort: 5432
+      nodePort: 32443
+    - name: floating-port-61443
+      port: 61443
+      targetPort: 5432

apps/gitlab/manifests/praefect-cnpg-cluster.yaml

@@ -26,10 +26,94 @@ spec:
 
   primaryUpdateStrategy: unsupervised
 
+  # Explicit params added 2026-08-22 (GitLab cross-site replication plan)
+  # - this cluster ran on bare Postgres defaults before. A VPS replica
+  # can't replay WAL past a checkpoint recorded with higher settings than
+  # its own ("recovery aborted because of insufficient parameter
+  # settings", confirmed live for pg-authentik) - giving both sides an
+  # identical explicit block up front means nobody can tune one side
+  # later and silently break replication.
+  postgresql:
+    parameters:
+      max_connections: "200"
+      shared_buffers: "512MB"
+      effective_cache_size: "1536MB"
+      maintenance_work_mem: "128MB"
+      checkpoint_completion_target: "0.9"
+      wal_buffers: "16MB"
+      default_statistics_target: "100"
+      random_page_cost: "1.1"
+      effective_io_concurrency: "200"
+      work_mem: "4MB"
+      min_wal_size: "1GB"
+      max_wal_size: "4GB"
+      max_worker_processes: "4"
+
   bootstrap:
     initdb:
       database: praefect_production
       owner: app
 
+  # Backup to VPS MinIO, also the initial-seed source for the VPS's own
+  # pg-praefect replica (bootstrap.recovery, Phase 1c) - this cluster had
+  # no backup block at all before. See the matching comment on
+  # cnpg-cluster.yaml (pg-gitlab) in this same directory.
+  backup:
+    barmanObjectStore:
+      destinationPath: s3://cnpg-backups/pg-praefect
+      endpointURL: http://100.108.113.41:9000
+      s3Credentials:
+        accessKeyId:
+          name: vps-minio-secret
+          key: accesskey
+        secretAccessKey:
+          name: vps-minio-secret
+          key: secretkey
+      wal:
+        compression: gzip
+        maxParallel: 2
+    retentionPolicy: "30d"
+
   monitoring:
     enablePodMonitor: true
+
+  # GitLab cross-site replication Phase 1c stage 2 - same pattern as
+  # cnpg-cluster.yaml (pg-gitlab) in this same directory.
+  externalClusters:
+    - name: home
+      connectionParameters:
+        host: pg-praefect-rw
+        port: "5432"
+        dbname: praefect_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-praefect-replication
+        key: tls.crt
+      sslKey:
+        name: pg-praefect-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-praefect-ca
+        key: ca.crt
+    - name: vps
+      connectionParameters:
+        host: 100.108.113.41
+        port: "32445"
+        dbname: praefect_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-praefect-vps-replication
+        key: tls.crt
+      sslKey:
+        name: pg-praefect-vps-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-praefect-vps-ca
+        key: ca.crt
+
+  replica:
+    self: home
+    primary: home
+    source: home

apps/gitlab/manifests/praefect-ha-configmap.yaml

@@ -0,0 +1,88 @@
+# GitLab cross-site replication (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - the chart has
+# no support for registering an externally-hosted Gitaly node into an
+# existing Praefect virtual storage (confirmed live: no
+# virtualStorages[].nodes/address-override key exists in the template,
+# gitalyReplicas just counts StatefulSet ordinals). Overriding Praefect's
+# rendered config.toml is the only way to add one.
+#
+# A values-only CONFIG_TEMPLATE_DIRECTORY redirect (extraEnv +
+# extraVolumes/extraVolumeMounts) was tried first and confirmed NOT to
+# work: charts/gitlab/charts/praefect/templates/statefulset.yaml's own
+# volumes: list never calls the gitlab.extraVolumes helper (only
+# volumeMounts does) - a chart limitation specific to this subchart, no
+# values-only fix exists.
+#
+# THIS ConfigMap (praefect-ha-config) is deployed as a harmless,
+# otherwise-unused object - it exists purely as a git-tracked reference
+# copy of the content that actually matters. The REAL live config lives
+# on the chart's own gitlab-praefect ConfigMap, which
+# argocd-apps/apps/gitlab.yaml's ignoreDifferences now exempts from
+# ArgoCD's normal drift-reconciliation (its `data` field specifically) -
+# apply this file's content to it directly:
+#   kubectl -n gitlab patch configmap gitlab-praefect --type merge \
+#     -p "{\"data\":{\"config.toml.tpl\":\"$(kubectl -n gitlab get cm \
+#     praefect-ha-config -o jsonpath='{.data.config\.toml\.tpl}' | \
+#     python3 -c 'import sys,json; print(json.dumps(sys.stdin.read())[1:-1])')\"}}"
+# (or simpler: kubectl -n gitlab get cm praefect-ha-config -o
+# jsonpath='{.data}' | kubectl -n gitlab patch cm gitlab-praefect --type
+# merge -p "{\"data\":$(cat -)}")
+#
+# Content below has since moved past byte-identical: nodes 0-2 now use
+# tls:// on port 8076 (matching global.gitaly.tls.enabled in
+# values.yaml) and the floating hostnames gitaly-{0,1,2}.ha.huskypup.net
+# (CoreDNS-rewritten locally for home's own traffic, same reasoning as
+# pg-gitlab.ha.huskypup.net) rather than in-cluster DNS names or the
+# plaintext scheme - this is what makes the SAME addresses usable once
+# the VPS joins as a 4th node, without home's own Praefect needing a
+# different config than what it hands the VPS as peer addresses. Praefect
+# trusts the cert via the pod's normal system CA bundle - it's a real
+# Let's Encrypt cert (gitaly-tls-certificate.yaml), not self-signed, so
+# no extra CA trust config is needed here.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: praefect-ha-config
+  namespace: gitlab
+data:
+  config.toml.tpl: |
+    # TCP address to listen on
+    listen_addr = '0.0.0.0:8075'
+    prometheus_listen_addr = '0.0.0.0:9236'
+
+    prometheus_exclude_database_from_default_metrics = true
+
+    [failover]
+    enabled = true
+    read_only_after_failover = true
+
+    [auth]
+    token = {% file.Read "/etc/gitlab-secrets/praefect/praefect_token" | strings.TrimSpace | data.ToJSON %}
+
+    transitioning = false
+
+    [logging]
+
+    [[virtual_storage]]
+    name = 'default'
+    [[virtual_storage.node]]
+    storage = 'gitlab-gitaly-default-0'
+    address = 'tls://gitaly-0.ha.huskypup.net:8076'
+    token = {% file.Read "/etc/gitlab-secrets/praefect/gitaly_token" | strings.TrimSpace | data.ToJSON %}
+    [[virtual_storage.node]]
+    storage = 'gitlab-gitaly-default-1'
+    address = 'tls://gitaly-1.ha.huskypup.net:8076'
+    token = {% file.Read "/etc/gitlab-secrets/praefect/gitaly_token" | strings.TrimSpace | data.ToJSON %}
+    [[virtual_storage.node]]
+    storage = 'gitlab-gitaly-default-2'
+    address = 'tls://gitaly-2.ha.huskypup.net:8076'
+    token = {% file.Read "/etc/gitlab-secrets/praefect/gitaly_token" | strings.TrimSpace | data.ToJSON %}
+
+
+    [database]
+    host = 'pg-praefect-rw.gitlab.svc.cluster.local'
+    port = 5432
+    user = 'app'
+    password = {% file.Read "/etc/gitlab-secrets/praefect/db_password" | strings.TrimSpace | data.ToJSON %}
+    dbname = 'praefect_production'
+    sslmode = 'disable'

apps/gitlab/values.yaml

@@ -54,6 +54,49 @@ global:
     internal:
       names: []  # No internal Gitaly, using Praefect
     external: []  # Praefect configured below
+    # GitLab cross-site replication (see
+    # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - Gitaly's
+    # gRPC has no TLS by default. The VPS's 4th node has to be reached
+    # over the same public UniFi WAN forward already used for Postgres
+    # (confirmed live the direct-Netbird-IP path doesn't accept inbound
+    # connections to home, same as the existing etcd quorum) - without
+    # this, that WAN hop would carry the gitaly_token,
+    # gitlab_shell_secret, and raw repository contents in plaintext.
+    # Chart-native support (tls_listen_addr alongside the existing
+    # plaintext listener, both stay active) - cert is a real Let's
+    # Encrypt cert via the same DNS-01 ClusterIssuer already used
+    # everywhere else (gitaly-tls-certificate.yaml), not a self-signed
+    # one, so no custom CA needs distributing to either site.
+    tls:
+      enabled: true
+
+  # Gitaly's [gitlab] url (the internal-API callback every Gitaly node
+  # uses for gitlab-shell hooks) - a real, TLS-supported override,
+  # confirmed live via gitlab/charts/gitlab/templates/_workhorse.tpl.
+  # Uses the SAME floating-hostname + CoreDNS-local-rewrite pattern
+  # already proven for Postgres and Gitaly's own node addresses: home's
+  # 3 Gitaly nodes resolve this locally (no WAN round-trip), the VPS's
+  # 4th node resolves it via the UniFi WAN forward - one consistent
+  # URL for every node regardless of site.
+  #
+  # tls.enabled REVERTED 2026-08-22 - broke the live site (HTTP 400
+  # "Client sent an HTTP request to an HTTPS server" on
+  # gitlab.kube.huskypup.net). Root cause: port 8181 is NOT internal-API
+  # -only - confirmed live via `kubectl get httproute gitlab-web` that
+  # ALL public traffic (gitlab.kube.huskypup.net, routed through Envoy/
+  # Istio edge gateway) also targets gitlab-webservice-default:8181.
+  # global.workhorse.tls.enabled replaces the plaintext listener with
+  # TLS on that SAME port chart-wide, for both the public site AND the
+  # internal API - there's no chart-native way to TLS one without the
+  # other. Envoy terminates public HTTPS and proxies to the pod in
+  # plaintext, so a TLS-only pod listener 400s every public request.
+  # host: left in place (harmless without tls.enabled - only used for
+  # the [gitlab] url string, not port selection). Internal-API-over-WAN
+  # transport security for the VPS's future Gitaly node needs a
+  # different mechanism (e.g. network-layer TLS/mTLS via Istio, or a
+  # second dedicated port if the chart ever adds one) - not solved here.
+  workhorse:
+    host: gitlab-internal-api.ha.huskypup.net
 
   # Praefect configuration
   praefect:
@@ -68,6 +111,12 @@ global:
     - name: default
       gitalyReplicas: 3  # Production HA
       maxUnavailable: 1
+      # Required once global.gitaly.tls.enabled: true - the chart's own
+      # NOTES.txt validation rejects the render without it ("not
+      # specified ('default')... not supported", confirmed live).
+      # Matches gitlab.gitaly.tls.secret's default name/keys
+      # (gitaly-tls-certificate.yaml issues into this exact secret).
+      tlsSecretName: gitlab-gitaly-tls
     
     # Praefect PostgreSQL configuration
     psql:
@@ -89,6 +138,19 @@ global:
   # Application Configuration
   appConfig:
     # OmniAuth SSO Configuration
+    #
+    # CORRECTED 2026-08-21 (second time - see git history for the wrong
+    # 2026-08-20 fix that removed this list entirely). This IS the real,
+    # actually-processed OIDC config mechanism for the CNG webservice
+    # image (chart's gitlab.appConfig.omniauth.configuration template,
+    # infra/_omniauth.tpl) - GITLAB_OMNIBUS_CONFIG below is an
+    # omnibus-image-only convention this image never processes, so it
+    # was always a documentation-only no-op despite looking functional.
+    # gitlab-oidc-secret's `provider` key now holds a full YAML-encoded
+    # provider block (apps/gitlab/manifests/external-secret-oidc.yaml),
+    # loaded via Ruby's YAML.load_file at container start - not raw
+    # client_id/secret strings, which is why plain GITLAB_OIDC_CLIENT_ID/
+    # SECRET keys alone (yesterday's assumption) were never enough.
     omniauth:
       enabled: true
       allowSingleSignOn: ['openid_connect']
@@ -289,11 +351,35 @@ gitlab:
         cpu: 250m
         memory: 512Mi
 
-    # Virtual storage configuration
+    # (dead config removed 2026-08-22: this virtualStorages block never
-    virtualStorages:
+    # actually did anything - confirmed live the chart's Praefect
-    - name: default
+    # ConfigMap template iterates global.praefect.virtualStorages, not
-      gitalyReplicas: 1  # Homelab sizing
+    # this one, so gitalyReplicas: 3 up there always won regardless of
-      maxUnavailable: 1
+    # what this said. See global.praefect.virtualStorages above for the
+    # real config.)
+    #
+    # GitLab cross-site replication (see
+    # /home/scooby/.claude/plans/jiggly-snacking-iverson.md): tried a
+    # values-only CONFIG_TEMPLATE_DIRECTORY redirect (extraEnv +
+    # extraVolumes/extraVolumeMounts) to add a VPS-hosted Gitaly node the
+    # chart has no mechanism to register - confirmed live this doesn't
+    # work: extraVolumeMounts and extraEnv both render correctly, but
+    # extraVolumes is a dead values key for THIS subchart specifically -
+    # charts/gitlab/charts/praefect/templates/statefulset.yaml's own
+    # volumes: list never calls the gitlab.extraVolumes helper at all
+    # (only volumeMounts does), so the mount has nothing to mount and the
+    # pod would fail to schedule. No values-only fix exists.
+    #
+    # Real fix: argocd-apps/apps/gitlab.yaml's ignoreDifferences now
+    # covers ConfigMap gitlab-praefect's `data` field, so Helm creates
+    # the object (with its own 3-node content) but ArgoCD stops
+    # reconciling its content afterward. The actual multi-node content
+    # lives in apps/gitlab/manifests/praefect-ha-configmap.yaml as a
+    # git-tracked reference/documentation copy (deployed under its own
+    # name, praefect-ha-config, harmless and otherwise unused) - the
+    # live gitlab-praefect ConfigMap gets kubectl-patched with that same
+    # content directly, same "documented but manually-applied"
+    # convention as every other cross-cluster secret in this plan.
 
   # GitLab Exporter for Prometheus metrics
   gitlab-exporter:

argocd-apps/apps/gitlab.yaml

@@ -33,6 +33,19 @@ spec:
       jsonPointers:
         - /spec/ports
         - /spec/selector
+    # GitLab cross-site replication (see
+    # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - the chart
+    # has no mechanism to register an externally-hosted (VPS) Gitaly node
+    # into Praefect's virtual storage. Lets the live ConfigMap's `data`
+    # be kubectl-patched directly (see apps/gitlab/manifests/
+    # praefect-ha-configmap.yaml for the content + patch command) without
+    # ArgoCD reverting it on the next sync - Helm still creates the
+    # object and everything else about it stays normally managed.
+    - group: ""
+      kind: ConfigMap
+      name: gitlab-praefect
+      jsonPointers:
+        - /data
   syncPolicy:
     automated:
       prune: true

argocd-apps/infrastructure/coredns.yaml

@@ -0,0 +1,22 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: coredns
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "0"  # cluster DNS - as early as anything can depend on it
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: infrastructure
+  source:
+    repoURL: https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
+    targetRevision: main
+    path: infrastructure/coredns/manifests
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: kube-system
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true

argocd-apps/vps-standby/gitlab.yaml

@@ -0,0 +1,33 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: vps-gitlab
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "3"  # after vps-cnpg (wave 1) and vps-eso (wave 1)
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: vps-standby
+  sources:
+    - repoURL: https://charts.gitlab.io/
+      chart: gitlab
+      targetRevision: 7.7.0
+      helm:
+        valueFiles:
+          - $values/infrastructure/vps-standby/gitlab/values.yaml
+    - repoURL: https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
+      targetRevision: main
+      ref: values
+    - repoURL: https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
+      targetRevision: main
+      path: infrastructure/vps-standby/gitlab/manifests
+  destination:
+    name: vps-standby
+    namespace: gitlab
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true

argocd-apps/vps-standby/vps-eso.yaml

@@ -0,0 +1,22 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: vps-eso
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "1"  # before any Application whose secrets it manages (authentik is wave 3)
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: vps-standby
+  source:
+    repoURL: https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
+    targetRevision: main
+    path: infrastructure/vps-eso/manifests
+  destination:
+    name: vps-standby
+    namespace: external-secrets
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true

infrastructure/authentik/argocd-blueprint.yaml

@@ -1,49 +1,77 @@
 ---
-apiVersion: v1
+# Was a plain ConfigMap with client_secret hardcoded in plaintext - found
-kind: ConfigMap
+# and fixed 2026-08-20 (user: "make the repo proper and not have
+# sensitive things in the files"). The value itself was already the
+# genuine live secret ArgoCD's own OIDC client uses (apps/argocd/
+# manifests/external-secret.yaml pulls the same value from Vault at
+# secret/argocd-oauth) - Vault was already the source of truth on the
+# APP side, this blueprint (the Authentik PROVIDER side) just had its own
+# separate hardcoded plaintext copy instead of also reading from Vault.
+#
+# Converted ConfigMap -> ExternalSecret (chart supports mounting
+# blueprints from Secrets too - see blueprints.secrets in values.yaml).
+# client_id stays inline - it's not a secret, same convention as every
+# other blueprint here.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
 metadata:
   name: authentik-blueprints-argocd
   namespace: authentik
-  labels:
+spec:
-    goauthentik.io/blueprint: "true"
+  refreshInterval: 1h
-data:
+  secretStoreRef:
-  argocd.yaml: |-
+    kind: ClusterSecretStore
-    version: 1
+    name: vault-backend
-    metadata:
+  target:
-      name: argocd-oidc
+    name: authentik-blueprints-argocd
-    entries:
+    creationPolicy: Owner
-      - model: authentik_providers_oauth2.oauth2provider
+    template:
-        id: argocd-provider
+      metadata:
-        state: present
+        labels:
-        identifiers:
+          goauthentik.io/blueprint: "true"
-          name: ArgoCD
+      data:
-        attrs:
+        argocd.yaml: |-
-          name: ArgoCD
+          version: 1
-          client_id: 3791058a7de278045e5ec36be29f7a33d635119b8c786f20c86b185139c28841
+          metadata:
-          client_secret: f787416822f5bf4c46fe9170e63d1bf9520bab7703d53341aaaec107c8c4ba253d96c30b70d23a1c0b5a401ae1ae4bfb5db06092d338e29e0b0f04cd29bc0b36
+            name: argocd-oidc
-          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+          entries:
-          authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
+            - model: authentik_providers_oauth2.oauth2provider
-          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+              id: argocd-provider
-          redirect_uris:
+              state: present
-            - url: "https://argocd.kube.huskypup.net/auth/callback"
+              identifiers:
-              matching_mode: strict
+                name: ArgoCD
-          property_mappings:
+              attrs:
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
+                name: ArgoCD
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
+                client_id: 3791058a7de278045e5ec36be29f7a33d635119b8c786f20c86b185139c28841
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
+                client_secret: "{{ .clientSecret }}"
-          client_type: confidential
+                authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
-          access_code_validity: "minutes=10"
+                authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
-          access_token_validity: "hours=1"
+                invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
-          refresh_token_validity: "days=30"
+                redirect_uris:
-          signing_key: !Find [authentik_crypto.certificatekeypair, [name, "authentik Internal JWT Certificate"]]
+                  - url: "https://argocd.kube.huskypup.net/auth/callback"
+                    matching_mode: strict
+                property_mappings:
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
+                client_type: confidential
+                access_code_validity: "minutes=10"
+                access_token_validity: "hours=1"
+                refresh_token_validity: "days=30"
+                signing_key: !Find [authentik_crypto.certificatekeypair, [name, "authentik Internal JWT Certificate"]]
 
-      - model: authentik_core.application
+            - model: authentik_core.application
-        id: argocd-application
+              id: argocd-application
-        state: present
+              state: present
-        identifiers:
+              identifiers:
-          slug: argocd
+                slug: argocd
-        attrs:
+              attrs:
-          name: ArgoCD
+                name: ArgoCD
-          slug: argocd
+                slug: argocd
-          policy_engine_mode: any
+                policy_engine_mode: any
-          provider: !KeyOf argocd-provider
+                provider: !KeyOf argocd-provider
+  data:
+    - secretKey: clientSecret
+      remoteRef:
+        key: argocd-oauth
+        property: client-secret

infrastructure/authentik/gitlab-blueprint.yaml

@@ -1,53 +1,82 @@
 ---
-apiVersion: v1
+# Was a plain ConfigMap with client_secret hardcoded in plaintext - found
-kind: ConfigMap
+# and fixed 2026-08-20 (user: "make the repo proper and not have
+# sensitive things in the files"). Unlike argocd/n8n/etc, Vault had NO
+# copy of this at all before this fix - the ONLY place it existed was
+# here and in gitlab-oidc-secret (a manually kubectl-created Secret in
+# the gitlab namespace, apps/gitlab/values.yaml:166 - also fixed
+# alongside this). Created secret/gitlab-oauth in Vault with the existing
+# live value (not rotated - this is GitLab's actual working OIDC
+# credential right now, changing it would break login until both sides
+# are updated in lockstep).
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
 metadata:
   name: authentik-blueprints-gitlab
   namespace: authentik
-  labels:
+spec:
-    goauthentik.io/blueprint: "true"
+  refreshInterval: 1h
-data:
+  secretStoreRef:
-  gitlab.yaml: |-
+    kind: ClusterSecretStore
-    # yaml-language-server: $schema=https://goauthentik.io/blueprints/schema.json
+    name: vault-backend
-    version: 1
+  target:
-    metadata:
+    name: authentik-blueprints-gitlab
-      name: gitlab-oidc
+    creationPolicy: Owner
-    entries:
+    template:
-      # OAuth2/OIDC Provider for GitLab
+      metadata:
-      - model: authentik_providers_oauth2.oauth2provider
+        labels:
-        id: gitlab-provider
+          goauthentik.io/blueprint: "true"
-        state: present
+      data:
-        identifiers:
+        gitlab.yaml: |-
-          name: GitLab
+          # yaml-language-server: $schema=https://goauthentik.io/blueprints/schema.json
-        attrs:
+          version: 1
-          name: GitLab
+          metadata:
-          client_id: 70b9da5c8166cea0ba504a869824020bce6636
+            name: gitlab-oidc
-          client_secret: 2e9c0c5b7a8969eb782d08bf1bf3d50ad96f8a73d9c18d1e4141816aebe14a622e8064bf5a5bce835d80e89cdc5f4cc6cf4a8c8234556647dc8884888866bd7d
+          entries:
-          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+            # OAuth2/OIDC Provider for GitLab
-          authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
+            - model: authentik_providers_oauth2.oauth2provider
-          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+              id: gitlab-provider
-          redirect_uris:
+              state: present
-            - url: "https://gitlab.kube.huskypup.net/users/auth/openid_connect/callback"
+              identifiers:
-              matching_mode: strict
+                name: GitLab
-          enabled: true
+              attrs:
-          property_mappings:
+                name: GitLab
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
+                client_id: 70b9da5c8166cea0ba504a869824020bce6636
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
+                client_secret: "{{ .clientSecret }}"
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
+                authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
-          client_type: confidential
+                authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
-          access_code_validity: "minutes=10"
+                invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
-          access_token_validity: "days=7"
+                redirect_uris:
-          refresh_token_validity: "days=30"
+                  - url: "https://gitlab.kube.huskypup.net/users/auth/openid_connect/callback"
-          signing_key: !Find [authentik_crypto.certificatekeypair, [name, default]]
+                    matching_mode: strict
+                  # Real GitLab on the VPS (infrastructure/vps-standby/gitlab/) -
+                  # shares this same provider/client_id, added 2026-08-21.
+                  - url: "https://gitlab.vps.huskypup.net/users/auth/openid_connect/callback"
+                    matching_mode: strict
+                enabled: true
+                property_mappings:
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
+                client_type: confidential
+                access_code_validity: "minutes=10"
+                access_token_validity: "days=7"
+                refresh_token_validity: "days=30"
+                signing_key: !Find [authentik_crypto.certificatekeypair, [name, default]]
 
-      # Application for GitLab
+            # Application for GitLab
-      - model: authentik_core.application
+            - model: authentik_core.application
-        id: gitlab-application
+              id: gitlab-application
-        state: present
+              state: present
-        identifiers:
+              identifiers:
-          slug: gitlab
+                slug: gitlab
-        attrs:
+              attrs:
-          name: GitLab
+                name: GitLab
-          slug: gitlab
+                slug: gitlab
-          policy_engine_mode: any
+                policy_engine_mode: any
-          provider: !KeyOf gitlab-provider
+                provider: !KeyOf gitlab-provider
+  data:
+    - secretKey: clientSecret
+      remoteRef:
+        key: gitlab-oauth
+        property: client-secret

infrastructure/authentik/grafana-blueprint.yaml

@@ -1,99 +1,121 @@
 ---
-apiVersion: v1
+# Was a plain ConfigMap with client_secret hardcoded in plaintext - found
-kind: ConfigMap
+# and fixed 2026-08-20. Unlike argocd/nextcloud, Vault had NO copy of
+# this at all - the only places it existed were here AND in
+# infrastructure/grafana/manifests/grafana-oauth-secret.yaml (also fixed
+# alongside this, same Vault path). Created secret/grafana-oauth with the
+# existing live value (not rotated - changing it would break login until
+# both sides are updated together).
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
 metadata:
   name: authentik-blueprints-grafana
   namespace: authentik
-  labels:
+spec:
-    goauthentik.io/blueprint: "true"
+  refreshInterval: 1h
-data:
+  secretStoreRef:
-  grafana.yaml: |-
+    kind: ClusterSecretStore
-    version: 1
+    name: vault-backend
-    metadata:
+  target:
-      name: grafana-oidc
+    name: authentik-blueprints-grafana
-    entries:
+    creationPolicy: Owner
-      # ============================================================================
+    template:
-      # AUTO-CREATE GRAFANA GROUPS
+      metadata:
-      # ============================================================================
+        labels:
-      # These groups control access levels in Grafana via OAuth role mapping:
+          goauthentik.io/blueprint: "true"
-      #   - Grafana Admins  → Admin role (full access)
+      data:
-      #   - Grafana Editors → Editor role (can edit dashboards)
+        grafana.yaml: |-
-      #   - Grafana Viewers → Viewer role (read-only)
+          version: 1
-      #
+          metadata:
-      # AUTOMATIC ADMIN ACCESS:
+            name: grafana-oidc
-      #   - Anyone in "authentik Admins" automatically gets Grafana Admin access
+          entries:
-      #   - No manual configuration needed!
+            # ============================================================================
-      # ============================================================================
+            # AUTO-CREATE GRAFANA GROUPS
+            # ============================================================================
+            # These groups control access levels in Grafana via OAuth role mapping:
+            #   - Grafana Admins  → Admin role (full access)
+            #   - Grafana Editors → Editor role (can edit dashboards)
+            #   - Grafana Viewers → Viewer role (read-only)
+            #
+            # AUTOMATIC ADMIN ACCESS:
+            #   - Anyone in "authentik Admins" automatically gets Grafana Admin access
+            #   - No manual configuration needed!
+            # ============================================================================
 
-      - model: authentik_core.group
+            - model: authentik_core.group
-        id: grafana-admins-group
+              id: grafana-admins-group
-        state: present
+              state: present
-        identifiers:
+              identifiers:
-          name: Grafana Admins
+                name: Grafana Admins
-        attrs:
+              attrs:
-          name: Grafana Admins
+                name: Grafana Admins
-          is_superuser: false
+                is_superuser: false
-          # Note: authentik Admins are automatically granted access via role_attribute_path
+                # Note: authentik Admins are automatically granted access via role_attribute_path
-          # in Grafana configuration (see grafana/values.yaml)
+                # in Grafana configuration (see grafana/values.yaml)
 
-      - model: authentik_core.group
+            - model: authentik_core.group
-        id: grafana-editors-group
+              id: grafana-editors-group
-        state: present
+              state: present
-        identifiers:
+              identifiers:
-          name: Grafana Editors
+                name: Grafana Editors
-        attrs:
+              attrs:
-          name: Grafana Editors
+                name: Grafana Editors
-          is_superuser: false
+                is_superuser: false
 
-      - model: authentik_core.group
+            - model: authentik_core.group
-        id: grafana-viewers-group
+              id: grafana-viewers-group
-        state: present
+              state: present
-        identifiers:
+              identifiers:
-          name: Grafana Viewers
+                name: Grafana Viewers
-        attrs:
+              attrs:
-          name: Grafana Viewers
+                name: Grafana Viewers
-          is_superuser: false
+                is_superuser: false
 
-      # ============================================================================
+            # ============================================================================
-      # GRAFANA OAUTH2 PROVIDER
+            # GRAFANA OAUTH2 PROVIDER
-      # ============================================================================
+            # ============================================================================
 
-      - model: authentik_providers_oauth2.oauth2provider
+            - model: authentik_providers_oauth2.oauth2provider
-        id: grafana-provider
+              id: grafana-provider
-        state: present
+              state: present
-        identifiers:
+              identifiers:
-          name: Grafana
+                name: Grafana
-        attrs:
+              attrs:
-          name: Grafana
+                name: Grafana
-          client_id: bd03e9139dd2063c6c44c4d2f65f51d69de3ba0b6d6b1b9b41c255d2376d2dcc
+                client_id: bd03e9139dd2063c6c44c4d2f65f51d69de3ba0b6d6b1b9b41c255d2376d2dcc
-          client_secret: 6b6f2deecfe8fd56cae9c512cc71eedd463d67ff08f24c816b15b15e78ce36bc4f06a2276c5ffe3f67799d935994a32ed32cdc02ec674f279c82d7cfe3ca05d5
+                client_secret: "{{ .clientSecret }}"
-          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+                authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
-          authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
+                authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
-          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+                invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
-          redirect_uris:
+                redirect_uris:
-            - url: "https://grafana.kube.huskypup.net/login/generic_oauth"
+                  - url: "https://grafana.kube.huskypup.net/login/generic_oauth"
-              matching_mode: strict
+                    matching_mode: strict
-          property_mappings:
+                property_mappings:
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
-          client_type: confidential
+                client_type: confidential
-          access_code_validity: "minutes=10"
+                access_code_validity: "minutes=10"
-          access_token_validity: "hours=1"
+                access_token_validity: "hours=1"
-          refresh_token_validity: "days=30"
+                refresh_token_validity: "days=30"
-          signing_key: !Find [authentik_crypto.certificatekeypair, [name, default]]
+                signing_key: !Find [authentik_crypto.certificatekeypair, [name, default]]
 
-      # ============================================================================
+            # ============================================================================
-      # GRAFANA APPLICATION
+            # GRAFANA APPLICATION
-      # ============================================================================
+            # ============================================================================
 
-      - model: authentik_core.application
+            - model: authentik_core.application
-        id: grafana-application
+              id: grafana-application
-        state: present
+              state: present
-        identifiers:
+              identifiers:
-          slug: grafana
+                slug: grafana
-        attrs:
+              attrs:
-          name: Grafana
+                name: Grafana
-          slug: grafana
+                slug: grafana
-          policy_engine_mode: any
+                policy_engine_mode: any
-          provider: !KeyOf grafana-provider
+                provider: !KeyOf grafana-provider
+  data:
+    - secretKey: clientSecret
+      remoteRef:
+        key: grafana-oauth
+        property: client-secret

infrastructure/authentik/guacamole-blueprint.yaml

@@ -1,83 +1,103 @@
 ---
-apiVersion: v1
+# Was a plain ConfigMap with client_secret hardcoded in plaintext - found
-kind: ConfigMap
+# and fixed 2026-08-20. Same story as argocd-blueprint.yaml: the value
+# was already Vault's source of truth on the app side (secret/
+# guacamole-oauth), this blueprint just had its own separate hardcoded
+# plaintext copy.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
 metadata:
   name: authentik-blueprints-guacamole
   namespace: authentik
-  labels:
+spec:
-    goauthentik.io/blueprint: "true"
+  refreshInterval: 1h
-data:
+  secretStoreRef:
-  guacamole.yaml: |-
+    kind: ClusterSecretStore
-    version: 1
+    name: vault-backend
-    metadata:
+  target:
-      name: guacamole-oidc
+    name: authentik-blueprints-guacamole
-      labels:
+    creationPolicy: Owner
-        blueprints.goauthentik.io/instantiate: "true"
+    template:
-    entries:
+      metadata:
-      # ============================================================================
+        labels:
-      # GUACAMOLE OAUTH2 PROVIDER
+          goauthentik.io/blueprint: "true"
-      # ============================================================================
+      data:
-      # This creates the OAuth2 provider that Guacamole will use for authentication.
+        guacamole.yaml: |-
-      # The client_id and client_secret must match what's configured in Guacamole.
+          version: 1
-      # ============================================================================
+          metadata:
+            name: guacamole-oidc
+            labels:
+              blueprints.goauthentik.io/instantiate: "true"
+          entries:
+            # ============================================================================
+            # GUACAMOLE OAUTH2 PROVIDER
+            # ============================================================================
+            # This creates the OAuth2 provider that Guacamole will use for authentication.
+            # The client_id and client_secret must match what's configured in Guacamole.
+            # ============================================================================
 
-      # ============================================================================
+            # ============================================================================
-      # GROUPS SCOPE MAPPING
+            # GROUPS SCOPE MAPPING
-      # ============================================================================
+            # ============================================================================
-      # Custom scope mapping that includes user's group names in the OIDC token.
+            # Custom scope mapping that includes user's group names in the OIDC token.
-      # ============================================================================
+            # ============================================================================
 
-      - model: authentik_providers_oauth2.scopemapping
+            - model: authentik_providers_oauth2.scopemapping
-        id: guacamole-groups-scope
+              id: guacamole-groups-scope
-        identifiers:
+              identifiers:
-          managed: goauthentik.io/providers/oauth2/guacamole-groups
+                managed: goauthentik.io/providers/oauth2/guacamole-groups
-        attrs:
+              attrs:
-          name: "Guacamole Groups Scope"
+                name: "Guacamole Groups Scope"
-          scope_name: groups
+                scope_name: groups
-          expression: |
+                expression: |
-            return {
+                  return {
-              "groups": [group.name for group in request.user.ak_groups.all()],
+                    "groups": [group.name for group in request.user.ak_groups.all()],
-            }
+                  }
 
-      - model: authentik_providers_oauth2.oauth2provider
+            - model: authentik_providers_oauth2.oauth2provider
-        id: guacamole-provider
+              id: guacamole-provider
-        identifiers:
+              identifiers:
-          name: guacamole
+                name: guacamole
-        attrs:
+              attrs:
-          name: Guacamole
+                name: Guacamole
-          client_id: a55cbe3c6d2ae30c05c86385802ffcfd76a972c22c8d039f000281d4a149fd44
+                client_id: a55cbe3c6d2ae30c05c86385802ffcfd76a972c22c8d039f000281d4a149fd44
-          client_secret: 718d4c4816aa68767a6a0da0fa4099d492473c700ba236cbeb64d294762b9548acda7e716ba11b2ad7729d6a1c16c7890b227e3b2dcfc35fd4af1eb0d6662b0f
+                client_secret: "{{ .clientSecret }}"
-          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+                authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
-          authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
+                authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
-          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+                invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
-          redirect_uris:
+                redirect_uris:
-            - url: "https://guacamole.kube.huskypup.net/"
+                  - url: "https://guacamole.kube.huskypup.net/"
-              matching_mode: strict
+                    matching_mode: strict
-            - url: "https://guacamole.kube.huskypup.net/guacamole/"
+                  - url: "https://guacamole.kube.huskypup.net/guacamole/"
-              matching_mode: strict
+                    matching_mode: strict
-          property_mappings:
+                property_mappings:
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
-            - !KeyOf guacamole-groups-scope
+                  - !KeyOf guacamole-groups-scope
-          client_type: confidential
+                client_type: confidential
-          access_code_validity: "minutes=10"
+                access_code_validity: "minutes=10"
-          access_token_validity: "hours=1"
+                access_token_validity: "hours=1"
-          refresh_token_validity: "days=30"
+                refresh_token_validity: "days=30"
-          signing_key: !Find [authentik_crypto.certificatekeypair, [name, "Lets Encrypt K8s Cert"]]
+                signing_key: !Find [authentik_crypto.certificatekeypair, [name, "Lets Encrypt K8s Cert"]]
 
-      # ============================================================================
+            # ============================================================================
-      # GUACAMOLE APPLICATION
+            # GUACAMOLE APPLICATION
-      # ============================================================================
+            # ============================================================================
-      # This creates the application entry that users will see in Authentik.
+            # This creates the application entry that users will see in Authentik.
-      # ============================================================================
+            # ============================================================================
 
-      - model: authentik_core.application
+            - model: authentik_core.application
-        id: guacamole-application
+              id: guacamole-application
-        identifiers:
+              identifiers:
-          slug: guacamole
+                slug: guacamole
-        attrs:
+              attrs:
-          name: Guacamole
+                name: Guacamole
-          slug: guacamole
+                slug: guacamole
-          policy_engine_mode: any
+                policy_engine_mode: any
-          provider: !KeyOf guacamole-provider
+                provider: !KeyOf guacamole-provider
+  data:
+    - secretKey: clientSecret
+      remoteRef:
+        key: guacamole-oauth
+        property: client-secret

infrastructure/authentik/manifests/ha-postgres-app-pushsecret.yaml

@@ -0,0 +1,36 @@
+# Multi-site active failover pilot (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - closes the gap
+# the user correctly flagged: manually kubectl/ssh-copying secrets between
+# home and the VPS defeats the entire point of having Vault. This is the
+# first piece of a real Vault -> VPS pipeline (infrastructure/vps-eso/ has
+# the other half, ESO running on the VPS itself pulling this back down).
+#
+# pg-authentik-app is CNPG-generated, not Vault-native - each cluster
+# (home and VPS) independently generates its own random password for the
+# `app` role at bootstrap time. Since VPS's Postgres now REPLICATES from
+# home (WAL includes role/password changes), the two clusters' actual live
+# passwords are only in sync because of that replication - but the two
+# clusters' K8S SECRET OBJECTS never resync on their own (confirmed live
+# 2026-08-20: VPS's copy was stale). Pushing home's value into Vault, with
+# VPS's ESO pulling it back down on refreshInterval, makes this self-
+# healing instead of a manual one-time fix that goes stale again on the
+# next password rotation.
+apiVersion: external-secrets.io/v1alpha1
+kind: PushSecret
+metadata:
+  name: pg-authentik-app-to-vps
+  namespace: authentik
+spec:
+  refreshInterval: 5m
+  secretStoreRefs:
+    - name: vault-backend
+      kind: ClusterSecretStore
+  selector:
+    secret:
+      name: pg-authentik-app
+  data:
+    - match:
+        secretKey: password
+        remoteRef:
+          remoteKey: vps/pg-authentik-app
+          property: password

infrastructure/authentik/manifests/ha-postgres-nodeport.yaml

@@ -6,6 +6,18 @@
 # object and won't fight with it. Needs
 # infrastructure/kyverno/policies/ha-failover-nodeport-exception.yaml
 # (disallow-nodeport-services is enforced cluster-wide otherwise).
+#
+# Second port (61432, ClusterIP only - no nodePort needed) added for the
+# CoreDNS rewrite in infrastructure/coredns/manifests/: home's own pods
+# resolving pg-authentik.ha.huskypup.net rewrite to THIS service (not
+# CNPG's own pg-authentik-rw) specifically so they can reach it on the
+# SAME port number the app tier is configured with (61432, matching the
+# external floating hostname's port - see infrastructure/authentik/
+# values.yaml's AUTHENTIK_POSTGRESQL__PORT) - CoreDNS rewrite only
+# changes the resolved NAME, not the port the client then connects to,
+# so the target service has to actually listen on that same port too, or
+# the connection times out even though DNS resolves correctly (confirmed
+# live 2026-08-20).
 apiVersion: v1
 kind: Service
 metadata:
@@ -17,6 +29,10 @@ spec:
     cnpg.io/cluster: pg-authentik
     cnpg.io/instanceRole: primary
   ports:
-    - port: 5432
+    - name: nodeport-5432
+      port: 5432
       targetPort: 5432
       nodePort: 32432
+    - name: floating-port-61432
+      port: 61432
+      targetPort: 5432

infrastructure/authentik/manifests/ha-postgres-peerauth.yaml

@@ -38,3 +38,14 @@ spec:
   portLevelMtls:
     "5432":
       mode: PERMISSIVE
+    # Multi-site active failover pilot, floating hostname (see
+    # infrastructure/authentik/manifests/ha-postgres-nodeport.yaml): the
+    # ha-authentik-postgres Service also listens on 61432 (-> targetPort
+    # 5432, same pods) so home's own pods can reach it via the
+    # pg-authentik.ha.huskypup.net CoreDNS rewrite on the same port the
+    # app tier is configured with externally. Needed its own PERMISSIVE
+    # entry - confirmed live 2026-08-20 that ambient's port-level mTLS
+    # enforcement is keyed on the port actually dialed (61432), not just
+    # the pod's real containerPort (5432) it eventually reaches.
+    "61432":
+      mode: PERMISSIVE

infrastructure/authentik/n8n-blueprint.yaml

@@ -1,49 +1,68 @@
 ---
-apiVersion: v1
+# Was a plain ConfigMap with client_secret hardcoded in plaintext - found
-kind: ConfigMap
+# and fixed 2026-08-20. Same story as argocd-blueprint.yaml: the value
+# was already Vault's source of truth on the app side (secret/n8n-oauth),
+# this blueprint just had its own separate hardcoded plaintext copy.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
 metadata:
   name: authentik-blueprints-n8n
   namespace: authentik
-  labels:
+spec:
-    goauthentik.io/blueprint: "true"
+  refreshInterval: 1h
-data:
+  secretStoreRef:
-  n8n.yaml: |-
+    kind: ClusterSecretStore
-    version: 1
+    name: vault-backend
-    metadata:
+  target:
-      name: n8n-oidc
+    name: authentik-blueprints-n8n
-    entries:
+    creationPolicy: Owner
-      - model: authentik_providers_oauth2.oauth2provider
+    template:
-        id: n8n-provider
+      metadata:
-        state: present
+        labels:
-        identifiers:
+          goauthentik.io/blueprint: "true"
-          name: n8n
+      data:
-        attrs:
+        n8n.yaml: |-
-          name: n8n
+          version: 1
-          client_id: e55cbe3c6d2ae30c05c86385802ffcfd76a972c22c8d039f000281d4a149fd44
+          metadata:
-          client_secret: 718d4c4816aa68767a6a0da0fa4099d492473c700ba236cbeb64d294762b9548acda7e716ba11b2ad7729d6a1c16c7890b227e3b2dcfc35fd4af1eb0d6662b0f
+            name: n8n-oidc
-          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+          entries:
-          authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
+            - model: authentik_providers_oauth2.oauth2provider
-          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+              id: n8n-provider
-          redirect_uris:
+              state: present
-            - url: "https://n8n.kube.huskypup.net/callback"
+              identifiers:
-              matching_mode: strict
+                name: n8n
-          property_mappings:
+              attrs:
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
+                name: n8n
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
+                client_id: e55cbe3c6d2ae30c05c86385802ffcfd76a972c22c8d039f000281d4a149fd44
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
+                client_secret: "{{ .clientSecret }}"
-          client_type: confidential
+                authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
-          access_code_validity: "minutes=10"
+                authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
-          access_token_validity: "hours=1"
+                invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
-          refresh_token_validity: "days=30"
+                redirect_uris:
-          signing_key: !Find [authentik_crypto.certificatekeypair, [name, "authentik Self-signed Certificate"]]
+                  - url: "https://n8n.kube.huskypup.net/callback"
+                    matching_mode: strict
+                property_mappings:
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
+                client_type: confidential
+                access_code_validity: "minutes=10"
+                access_token_validity: "hours=1"
+                refresh_token_validity: "days=30"
+                signing_key: !Find [authentik_crypto.certificatekeypair, [name, "authentik Self-signed Certificate"]]
 
-      - model: authentik_core.application
+            - model: authentik_core.application
-        id: n8n-application
+              id: n8n-application
-        state: present
+              state: present
-        identifiers:
+              identifiers:
-          slug: n8n
+                slug: n8n
-        attrs:
+              attrs:
-          name: n8n
+                name: n8n
-          slug: n8n
+                slug: n8n
-          policy_engine_mode: any
+                policy_engine_mode: any
-          provider: !KeyOf n8n-provider
+                provider: !KeyOf n8n-provider
+  data:
+    - secretKey: clientSecret
+      remoteRef:
+        key: n8n-oauth
+        property: client-secret

infrastructure/authentik/nextcloud-blueprint.yaml

@@ -1,52 +1,72 @@
 ---
-apiVersion: v1
+# Was a plain ConfigMap with client_secret hardcoded in plaintext - found
-kind: ConfigMap
+# and fixed 2026-08-20. Same story as argocd-blueprint.yaml: the value
+# was already Vault's source of truth on the app side (secret/
+# nextcloud-oauth), this blueprint just had its own separate hardcoded
+# plaintext copy.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
 metadata:
   name: authentik-blueprints-nextcloud
   namespace: authentik
-  labels:
+spec:
-    goauthentik.io/blueprint: "true"
+  refreshInterval: 1h
-data:
+  secretStoreRef:
-   nextcloud.yaml: |-
+    kind: ClusterSecretStore
-    version: 1
+    name: vault-backend
-    metadata:
+  target:
-      name: nextcloud-oidc
+    name: authentik-blueprints-nextcloud
-    entries:
+    creationPolicy: Owner
-      - model: authentik_providers_oauth2.oauth2provider
+    template:
-        id: nextcloud-provider
+      metadata:
-        state: present
+        labels:
-        identifiers:
+          goauthentik.io/blueprint: "true"
-          name: Nextcloud
+      data:
-        attrs:
+        nextcloud.yaml: |-
-          name: Nextcloud
+          version: 1
-          client_id: HmTGnUfhCvjFFbUUAne7JLrucNC2X675JicPbEnQ
+          metadata:
-          client_secret: qTUf4NYjzUuAqfNV5D7SFPuuhJJwVltU6U3LAuJVy7dddJWeJSv2B4wKaRxbbHqpIiSfYPDaKujhEysKbOWUnf4n9OHPKJ0bu1I2EzkYtChO1Gq4gUv5EHPTzQlMBPnl
+            name: nextcloud-oidc
-          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+          entries:
-          authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
+            - model: authentik_providers_oauth2.oauth2provider
-          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+              id: nextcloud-provider
-          redirect_uris:
+              state: present
-            - url: "https://nextcloud.kube.huskypup.net/apps/user_oidc/code"
+              identifiers:
-              matching_mode: strict
+                name: Nextcloud
-            - url: "http://nextcloud.kube.huskypup.net/apps/user_oidc/code"
+              attrs:
-              matching_mode: strict
+                name: Nextcloud
-          enabled: true
+                client_id: HmTGnUfhCvjFFbUUAne7JLrucNC2X675JicPbEnQ
-          property_mappings:
+                client_secret: "{{ .clientSecret }}"
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
+                authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
+                authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
+                invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
-          client_type: confidential
+                redirect_uris:
-          access_code_validity: "minutes=10"
+                  - url: "https://nextcloud.kube.huskypup.net/apps/user_oidc/code"
-          access_token_validity: "days=7"
+                    matching_mode: strict
-          refresh_token_validity: "days=30"
+                  - url: "http://nextcloud.kube.huskypup.net/apps/user_oidc/code"
-          signing_key: !Find [authentik_crypto.certificatekeypair, [name, "authentik Internal JWT Certificate"]]
+                    matching_mode: strict
+                enabled: true
+                property_mappings:
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
+                client_type: confidential
+                access_code_validity: "minutes=10"
+                access_token_validity: "days=7"
+                refresh_token_validity: "days=30"
+                signing_key: !Find [authentik_crypto.certificatekeypair, [name, "authentik Internal JWT Certificate"]]
 
-      - model: authentik_core.application
+            - model: authentik_core.application
-        id: nextcloud-application
+              id: nextcloud-application
-        state: present
+              state: present
-        identifiers:
+              identifiers:
-          slug: nextcloud
+                slug: nextcloud
-        attrs:
+              attrs:
-          name: Nextcloud
+                name: Nextcloud
-          slug: nextcloud
+                slug: nextcloud
-          policy_engine_mode: any
+                policy_engine_mode: any
-          provider: !KeyOf nextcloud-provider
+                provider: !KeyOf nextcloud-provider
+  data:
+    - secretKey: clientSecret
+      remoteRef:
+        key: nextcloud-oauth
+        property: client-secret

infrastructure/authentik/rancher-blueprint.yaml

@@ -1,54 +1,73 @@
 ---
-apiVersion: v1
+# Was a plain ConfigMap with client_secret hardcoded in plaintext - found
-kind: ConfigMap
+# and fixed 2026-08-20. Same story as argocd-blueprint.yaml: the value
+# was already Vault's source of truth on the app side (secret/
+# rancher-oauth), this blueprint just had its own separate hardcoded
+# plaintext copy.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
 metadata:
   name: authentik-blueprints-rancher
   namespace: authentik
-  labels:
+spec:
-    goauthentik.io/blueprint: "true"
+  refreshInterval: 1h
-data:
+  secretStoreRef:
-  rancher.yaml: |-
+    kind: ClusterSecretStore
-    # yaml-language-server: $schema=https://goauthentik.io/blueprints/schema.json
+    name: vault-backend
-    version: 1
+  target:
-    metadata:
+    name: authentik-blueprints-rancher
-      name: rancher-oidc
+    creationPolicy: Owner
-    entries:
+    template:
-      - model: authentik_providers_oauth2.oauth2provider
+      metadata:
-        id: rancher-provider
+        labels:
-        state: present
+          goauthentik.io/blueprint: "true"
-        identifiers:
+      data:
-          name: Rancher
+        rancher.yaml: |-
-        attrs:
+          # yaml-language-server: $schema=https://goauthentik.io/blueprints/schema.json
-          name: Rancher
+          version: 1
-          client_id: 94a6beeaa91d0d0407cf8af807e8208a611cf3f25ae42161e697a8dccd563449
+          metadata:
-          client_secret: 19ebd4545ba74a93c220d4471af634302c5991204cf650cabd522b0864d7617ded2c59b470ab6de7a5801e7de4c84e4b9741b983170815419c9503d55f849794
+            name: rancher-oidc
-          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+          entries:
-          authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
+            - model: authentik_providers_oauth2.oauth2provider
-          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+              id: rancher-provider
-          redirect_uris:
+              state: present
-            - url: "https://rancher.kube.huskypup.net/verify-auth"
+              identifiers:
-              matching_mode: strict
+                name: Rancher
-            - url: "https://rancher.kube.huskypup.net"
+              attrs:
-              matching_mode: strict
+                name: Rancher
-          enabled: true
+                client_id: 94a6beeaa91d0d0407cf8af807e8208a611cf3f25ae42161e697a8dccd563449
-          property_mappings:
+                client_secret: "{{ .clientSecret }}"
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
+                authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
+                authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
+                invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
-          client_type: confidential
+                redirect_uris:
-          access_code_validity: "minutes=10"
+                  - url: "https://rancher.kube.huskypup.net/verify-auth"
-          access_token_validity: "hours=1"
+                    matching_mode: strict
-          refresh_token_validity: "days=30"
+                  - url: "https://rancher.kube.huskypup.net"
-          signing_key: !Find [authentik_crypto.certificatekeypair, [name, "authentik Internal JWT Certificate"]]
+                    matching_mode: strict
-
+                enabled: true
-      - model: authentik_core.application
+                property_mappings:
-        id: rancher-application
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
-        state: present
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
-        identifiers:
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
-          slug: rancher
+                client_type: confidential
-        attrs:
+                access_code_validity: "minutes=10"
-          name: Rancher
+                access_token_validity: "hours=1"
-          slug: rancher
+                refresh_token_validity: "days=30"
-          policy_engine_mode: any
+                signing_key: !Find [authentik_crypto.certificatekeypair, [name, "authentik Internal JWT Certificate"]]
-          provider: !KeyOf rancher-provider
 
+            - model: authentik_core.application
+              id: rancher-application
+              state: present
+              identifiers:
+                slug: rancher
+              attrs:
+                name: Rancher
+                slug: rancher
+                policy_engine_mode: any
+                provider: !KeyOf rancher-provider
+  data:
+    - secretKey: clientSecret
+      remoteRef:
+        key: rancher-oauth
+        property: client-secret

infrastructure/authentik/values.yaml

@@ -13,10 +13,23 @@ global:
     # First secret: Authentik-specific settings (AUTHENTIK_SECRET_KEY)
     - secretRef:
         name: authentik
-    # Second secret: Database credentials from CNPG
+    # The second secretRef (pg-authentik-app, prefix: AUTHENTIK_POSTGRESQL__)
-    - secretRef:
+    # that used to live here is GONE - found live 2026-08-20 that it was
-        name: pg-authentik-app
+    # actively breaking things, not just redundant. That prefix mechanism
-      prefix: AUTHENTIK_POSTGRESQL__
+    # copies the raw secret's lowercase keys (host, port, dbname, username,
+    # password) into lowercase-suffixed env vars (AUTHENTIK_POSTGRESQL__host,
+    # etc.) - env var names are case-sensitive to Kubernetes, but authentik's
+    # own generic AUTHENTIK_*-prefixed env-var scanner is NOT, and it was
+    # picking up the lowercase AUTHENTIK_POSTGRESQL__host=pg-authentik-rw
+    # instead of the explicit uppercase AUTHENTIK_POSTGRESQL__HOST override
+    # below - silently connecting to the local CNPG service on port 5432
+    # the whole time regardless of what HOST/PORT were explicitly set to,
+    # surfacing as an unexplained "connection timeout expired" loop (the
+    # local hostname resolves fine in-cluster but on the VPS the request
+    # apparently doesn't complete - never fully root-caused, just confirmed
+    # dropping this duplicate source fixes it). Every key that secretRef
+    # used to provide (dbname/username/password) is already explicitly set
+    # via valueFrom below, so nothing is lost by removing it.
   env:
     # Configure external URL for proper OIDC discovery responses
     - name: AUTHENTIK_URL
@@ -64,22 +77,26 @@ global:
     - name: AUTHENTIK_POSTGRESQL__PORT
       value: "61432"
 
-# Blueprints - Mount ConfigMaps for auto-discovery
+# Blueprints - Mount ConfigMaps for auto-discovery (no secrets) or
+# Secrets (client_secret-bearing ones, ExternalSecret-templated from
+# Vault - found and fixed 2026-08-20, see each file's own header for
+# what it used to look like).
 blueprints:
   configMaps:
+    - authentik-blueprints-forward-auth
+    - authentik-blueprints-prometheus
+    - authentik-blueprints-ceph
+    - authentik-blueprints-home-assistant
+    - authentik-blueprints-percona-everest
+    - authentik-blueprints-netbird
+  secrets:
     - authentik-blueprints-gitlab
     - authentik-blueprints-nextcloud
     - authentik-blueprints-argocd
     - authentik-blueprints-grafana
-    - authentik-blueprints-forward-auth
-    - authentik-blueprints-prometheus
     - authentik-blueprints-n8n
-    - authentik-blueprints-ceph
-    - authentik-blueprints-home-assistant
     - authentik-blueprints-guacamole
-    - authentik-blueprints-percona-everest
     - authentik-blueprints-rancher
-    - authentik-blueprints-netbird
     - authentik-blueprints-vault
 
 # Enable Prometheus metrics

infrastructure/authentik/vault-blueprint.yaml

@@ -1,51 +1,83 @@
 ---
-apiVersion: v1
+# Was a plain ConfigMap with client_secret hardcoded in plaintext - found
-kind: ConfigMap
+# and fixed 2026-08-20. Unlike argocd/nextcloud, Vault had NO copy of
+# this at all - the only places it existed were here AND in
+# infrastructure/vault/manifests/vault-init-configmap.yaml (embedded in a
+# shell script that runs `vault write auth/oidc/config` once at cluster
+# init - also fixed alongside this, same Vault path). Created
+# secret/vault-oauth with the existing live value (not rotated - this is
+# what Vault's OWN oidc auth method is currently configured with).
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
 metadata:
   name: authentik-blueprints-vault
   namespace: authentik
-  labels:
+spec:
-    goauthentik.io/blueprint: "true"
+  refreshInterval: 1h
-data:
+  secretStoreRef:
-  vault.yaml: |-
+    kind: ClusterSecretStore
-    version: 1
+    name: vault-backend
-    metadata:
+  target:
-      name: vault-oidc
+    name: authentik-blueprints-vault
-    entries:
+    creationPolicy: Owner
-      - model: authentik_providers_oauth2.oauth2provider
+    template:
-        id: vault-provider
+      metadata:
-        state: present
+        labels:
-        identifiers:
+          goauthentik.io/blueprint: "true"
-          name: Vault
+      data:
-        attrs:
+        vault.yaml: |-
-          name: Vault
+          version: 1
-          client_id: 9816a5ae7e7914b5d18f4ab939d011a98f8c8d6b3bb6777c46431afa06ac4a85
+          metadata:
-          client_secret: ed2ba1c6378c7a46341b5162f39a7fab80e37596b01ed387c3719e8e0040344cf1daa307476c2e7a7f75041b3979275b1ebf00bb8bad94c864b4a38ded544f7b
+            name: vault-oidc
-          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+          entries:
-          authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
+            - model: authentik_providers_oauth2.oauth2provider
-          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+              id: vault-provider
-          redirect_uris:
+              state: present
-            - url: "https://vault.kube.huskypup.net/ui/vault/auth/oidc/oidc/callback"
+              identifiers:
-              matching_mode: strict
+                name: Vault
-            - url: "http://localhost:8250/oidc/callback"
+              attrs:
-              matching_mode: strict
+                name: Vault
-          property_mappings:
+                client_id: 9816a5ae7e7914b5d18f4ab939d011a98f8c8d6b3bb6777c46431afa06ac4a85
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
+                client_secret: "{{ .clientSecret }}"
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
+                authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
-            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
+                authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
-          client_type: confidential
+                invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
-          access_code_validity: "minutes=10"
+                redirect_uris:
-          access_token_validity: "hours=1"
+                  - url: "https://vault.kube.huskypup.net/ui/vault/auth/oidc/oidc/callback"
-          refresh_token_validity: "days=30"
+                    matching_mode: strict
-          signing_key: !Find [authentik_crypto.certificatekeypair, [name, "authentik Internal JWT Certificate"]]
+                  # VPS's own standby Vault instance (infrastructure/vps-standby/vault/)
+                  # shares this SAME Authentik provider/client_id - found missing
+                  # 2026-08-21 while testing VPS vault login ("missing auth_url" -
+                  # Vault silently returns no auth_url when the UI's computed
+                  # redirect_uri isn't in allowed_redirect_uris; same fix needed on
+                  # the VPS Vault's own auth/oidc/role/default, done separately since
+                  # it's a fully independent Vault instance/backend).
+                  - url: "https://vault.vps.huskypup.net/ui/vault/auth/oidc/oidc/callback"
+                    matching_mode: strict
+                  - url: "http://localhost:8250/oidc/callback"
+                    matching_mode: strict
+                property_mappings:
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
+                  - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
+                client_type: confidential
+                access_code_validity: "minutes=10"
+                access_token_validity: "hours=1"
+                refresh_token_validity: "days=30"
+                signing_key: !Find [authentik_crypto.certificatekeypair, [name, "authentik Internal JWT Certificate"]]
 
-      - model: authentik_core.application
+            - model: authentik_core.application
-        id: vault-application
+              id: vault-application
-        state: present
+              state: present
-        identifiers:
+              identifiers:
-          slug: vault
+                slug: vault
-        attrs:
+              attrs:
-          name: Vault
+                name: Vault
-          slug: vault
+                slug: vault
-          policy_engine_mode: any
+                policy_engine_mode: any
-          provider: !KeyOf vault-provider
+                provider: !KeyOf vault-provider
+  data:
+    - secretKey: clientSecret
+      remoteRef:
+        key: vault-oauth
+        property: client-secret

infrastructure/cert-manager/manifests/letsencrypt-issuer.yaml

@@ -19,6 +19,17 @@ spec:
         selector:
           dnsZones:
             - "kube.huskypup.net"
+            # GitLab cross-site replication (see
+            # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) -
+            # first real Let's Encrypt cert needed for anything under
+            # ha.huskypup.net (the floating-hostname zone used for
+            # cross-site Postgres/Gitaly addressing) - everything
+            # before this used CNPG's own self-signed certs, never hit
+            # this gap. Confirmed live: DNS-01 order failed with "no
+            # configured challenge solvers can be used for this
+            # challenge" until this zone was added. Same Cloudflare
+            # account/token as kube.huskypup.net.
+            - "ha.huskypup.net"
 
 ---
 apiVersion: cert-manager.io/v1

infrastructure/coredns/manifests/coredns-configmap.yaml

@@ -0,0 +1,100 @@
+# Brings kube-system/coredns's Corefile under GitOps management. It was
+# previously only ever edited live via kubectl (3 existing rewrite rules
+# for auth/rancher/netbird.kube.huskypup.net, none of them git-tracked
+# anywhere in this repo) - every future change meant a manual,
+# classifier-gated live edit to a cluster-wide DNS config instead of a
+# normal git commit ArgoCD applies on its own. No Talos/kubeadm
+# ownerReferences on this ConfigMap or the coredns Deployment (confirmed
+# live 2026-08-20), so it's safe to take over here.
+#
+# No pod restart needed after a sync - the Corefile's own `reload` plugin
+# (already present below) watches the mounted file and hot-reloads within
+# ~30-60s of the ConfigMap volume actually updating on disk (kubelet
+# propagates ConfigMap changes to mounted volumes automatically, typically
+# within ~60-90s).
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: coredns
+  namespace: kube-system
+data:
+  Corefile: |
+    .:53 {
+        errors
+        health {
+            lameduck 5s
+        }
+        ready
+        log . {
+            class error
+        }
+        prometheus :9153
+
+        # Rewrite auth.kube.huskypup.net to ingress gateway (NOT authentik-server directly)
+        # This ensures HTTPS works because the ingress gateway has the wildcard TLS cert.
+        # The old rewrite to authentik-server caused TLS failures for OIDC token exchange.
+        rewrite name auth.kube.huskypup.net istio-ingressgateway.istio-system.svc.cluster.local
+
+        # Rewrite rancher.kube.huskypup.net to internal Istio ingress gateway
+        rewrite name rancher.kube.huskypup.net istio-ingressgateway.istio-system.svc.cluster.local
+
+        # Rewrite netbird.kube.huskypup.net to internal Istio ingress gateway
+        rewrite name netbird.kube.huskypup.net istio-ingressgateway.istio-system.svc.cluster.local
+
+        # Multi-site active failover pilot (see
+        # /home/scooby/.claude/plans/jiggly-snacking-iverson.md): home's
+        # own pods resolving the floating pg-authentik.ha.huskypup.net
+        # record always go straight to a local service instead of
+        # round-tripping out through the UniFi WAN port-forward and back
+        # in - this UniFi doesn't support NAT hairpin/loopback for its
+        # own forwarded ports (confirmed live 2026-08-20: DNS resolved
+        # fine, TCP connect failed). Points at ha-authentik-postgres, NOT
+        # CNPG's own pg-authentik-rw directly - rewrite only changes the
+        # resolved name, not the port a client then connects to, and the
+        # app tier is configured for port 61432 (matching the external
+        # floating hostname) - pg-authentik-rw only listens on 5432,
+        # ha-authentik-postgres additionally listens on 61432 (see
+        # infrastructure/authentik/manifests/ha-postgres-nodeport.yaml)
+        # specifically so this rewrite target actually has that port.
+        rewrite name pg-authentik.ha.huskypup.net ha-authentik-postgres.authentik.svc.cluster.local
+
+        # GitLab cross-site replication (see
+        # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - same
+        # reasoning as pg-authentik.ha.huskypup.net above: home's own pods
+        # go straight to the local NodePort Service instead of round-
+        # tripping out through the UniFi WAN forward and back in (no NAT
+        # hairpin support, confirmed live for the authentik precedent).
+        rewrite name pg-gitlab.ha.huskypup.net ha-gitlab-postgres.gitlab.svc.cluster.local
+        rewrite name pg-praefect.ha.huskypup.net ha-praefect-postgres.gitlab.svc.cluster.local
+
+        # GitLab cross-site replication Phase 2b (Gitaly) - home's own
+        # Praefect dials these SAME floating hostnames it hands to the
+        # VPS's Gitaly node as peer addresses (Praefect uses one address
+        # per node for everyone, not different addresses per caller), so
+        # home's own in-cluster traffic to its own local Gitaly nodes
+        # needs to resolve locally too, same reasoning as pg-*.ha above.
+        rewrite name gitaly-0.ha.huskypup.net ha-gitaly-0.gitlab.svc.cluster.local
+        rewrite name gitaly-1.ha.huskypup.net ha-gitaly-1.gitlab.svc.cluster.local
+        rewrite name gitaly-2.ha.huskypup.net ha-gitaly-2.gitlab.svc.cluster.local
+
+        # global.workhorse.host override (apps/gitlab/values.yaml) - every
+        # Gitaly node's [gitlab] url, home's own included, same local-
+        # rewrite reasoning as the entries above.
+        rewrite name gitlab-internal-api.ha.huskypup.net ha-gitlab-internal-api.gitlab.svc.cluster.local
+
+        kubernetes cluster.local in-addr.arpa ip6.arpa {
+            pods insecure
+            fallthrough in-addr.arpa ip6.arpa
+            ttl 30
+        }
+        forward . 1.1.1.1 8.8.8.8 {
+           max_concurrent 1000
+        }
+        cache 30 {
+           disable success cluster.local
+           disable denial cluster.local
+        }
+        loop
+        reload
+        loadbalance
+    }

infrastructure/external-dns/manifests/secret-external-dns-unifi.yaml

@@ -1,8 +1,33 @@
 ---
-apiVersion: v1
+# Was a plain Secret with a live UniFi API key committed in plaintext -
-kind: Secret
+# found and fixed 2026-08-20 (same class of issue as the Cloudflare token
+# in infrastructure/cert-manager/manifests/secret-cf-token.yaml, which was
+# already rotated to this pattern - this one was missed at the time).
+# Moved to Vault+ExternalSecret, matching that precedent.
+#
+# The value in Vault right now (secret/unifi-api-key) is still the
+# ORIGINAL key that was exposed in git history - rotating it (Settings ->
+# Admins -> API Keys on the UniFi controller at https://172.28.110.1, no
+# self-service API for this) is a separate, still-pending step. Once
+# rotated:
+#   vault kv put secret/unifi-api-key api-key=<new key>
+# ESO picks it up automatically (refreshInterval below), no restart
+# needed on external-dns's side.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
 metadata:
   name: external-dns-unifi-secret
   namespace: external-dns
-stringData:
+spec:
-  api-key: 3Qha5hupHsCkkFyQb8z-T2kh-BasIGSH
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: vault-backend
+  target:
+    name: external-dns-unifi-secret
+    creationPolicy: Owner
+  data:
+    - secretKey: api-key
+      remoteRef:
+        key: unifi-api-key
+        property: api-key

infrastructure/grafana/manifests/grafana-oauth-secret.yaml

@@ -1,12 +1,32 @@
-# infrastructure/grafana/grafana-oauth-secret.yaml
+# Was a plain Secret with the OIDC client_id/client_secret hardcoded in
-# Grafana OAuth secret - hardcoded from blueprint
+# plaintext (the file's own old comment even said "hardcoded from
-
+# blueprint") - found and fixed 2026-08-20 alongside
-apiVersion: v1
+# infrastructure/authentik/grafana-blueprint.yaml (same value, same Vault
-kind: Secret
+# path - see that file for the full story).
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
 metadata:
   name: grafana-authentik-oauth
   namespace: grafana
-type: Opaque
+spec:
-stringData:
+  refreshInterval: 1h
-  GF_AUTH_GENERIC_OAUTH_CLIENT_ID: "bd03e9139dd2063c6c44c4d2f65f51d69de3ba0b6d6b1b9b41c255d2376d2dcc"
+  secretStoreRef:
-  GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: "6b6f2deecfe8fd56cae9c512cc71eedd463d67ff08f24c816b15b15e78ce36bc4f06a2276c5ffe3f67799d935994a32ed32cdc02ec674f279c82d7cfe3ca05d5"
+    kind: ClusterSecretStore
+    name: vault-backend
+  target:
+    name: grafana-authentik-oauth
+    creationPolicy: Owner
+    template:
+      type: Opaque
+      data:
+        GF_AUTH_GENERIC_OAUTH_CLIENT_ID: "{{ .clientId }}"
+        GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: "{{ .clientSecret }}"
+  data:
+    - secretKey: clientId
+      remoteRef:
+        key: grafana-oauth
+        property: client-id
+    - secretKey: clientSecret
+      remoteRef:
+        key: grafana-oauth
+        property: client-secret

infrastructure/istio/manifests/apps/authorization-policies-apps.yaml

@@ -83,6 +83,25 @@ spec:
         - source:
             ipBlocks:
               - "10.244.0.0/16"
+    # GitLab cross-site replication (see
+    # /home/scooby/.claude/plans/jiggly-snacking-iverson.md): CNPG
+    # streaming replication from the VPS/witness, arriving via the
+    # ha-gitlab-postgres/ha-praefect-postgres NodePorts - no mesh
+    # identity at all, so no source.namespaces rule above can match them.
+    # Scoped by destination port instead, matching the port-scoped
+    # PERMISSIVE PeerAuthentication in ha-peerauth.yaml (same root cause,
+    # same fix, one layer up - mTLS being allowed through isn't enough by
+    # itself, this ALLOW policy independently denies anything not
+    # matching one of the rules above). Same pattern already proven for
+    # Authentik in infrastructure/istio/manifests/mesh/authorization-policies.yaml.
+    - to:
+        - operation:
+            ports:
+              - "5432"
+              - "61442" # pg-gitlab floating-hostname port, see ha-postgres-nodeport.yaml
+              - "61443" # pg-praefect floating-hostname port, see ha-postgres-nodeport.yaml
+              - "8075" # Gitaly gRPC, see ha-gitaly-nodeport.yaml
+              - "8181" # Rails internal API, see ha-gitlab-internal-api-nodeport.yaml
 
 ---
 # --- n8n: Allow ingress + Prometheus + intra-namespace ---

infrastructure/istio/manifests/mesh/authorization-policies.yaml

@@ -131,6 +131,7 @@ spec:
         - operation:
             ports:
               - "5432"
+              - "61432" # floating-hostname port, see ha-postgres-peerauth.yaml
 
 ---
 # --- Grafana: Allow ingress + Prometheus datasource queries + scraping ---

infrastructure/vault/manifests/vault-init-configmap.yaml

@@ -173,8 +173,22 @@ data:
       ttl="24h" >/dev/null
 
     # --- OIDC Auth (Authentik SSO) ---
-    OIDC_CLIENT_ID="9816a5ae7e7914b5d18f4ab939d011a98f8c8d6b3bb6777c46431afa06ac4a85"
+    # OIDC_CLIENT_ID/SECRET used to be hardcoded here in plaintext - found
-    OIDC_CLIENT_SECRET="ed2ba1c6378c7a46341b5162f39a7fab80e37596b01ed387c3719e8e0040344cf1daa307476c2e7a7f75041b3979275b1ebf00bb8bad94c864b4a38ded544f7b"
+    # and fixed 2026-08-20. Can't source this from Vault itself the way
+    # every other app's OIDC secret now does (infrastructure/authentik/
+    # vault-blueprint.yaml, secret/vault-oauth) - this job is what GRANTS
+    # ESO's own Kubernetes-auth role a few lines above
+    # (bound_service_account_names="...,external-secrets"), so ESO can't
+    # yet authenticate to Vault at the point this script runs - genuinely
+    # circular. Sourced from vault-oidc-bootstrap instead, a plain Secret
+    # created once manually (kubectl, not git):
+    #   kubectl -n vault create secret generic vault-oidc-bootstrap \
+    #     --from-literal=client_id=<same value as secret/vault-oauth> \
+    #     --from-literal=client_secret=<same value as secret/vault-oauth>
+    # envFrom on the Job (vault-init-job.yaml) injects these as
+    # VAULT_OIDC_CLIENT_ID/VAULT_OIDC_CLIENT_SECRET.
+    OIDC_CLIENT_ID="${VAULT_OIDC_CLIENT_ID:?VAULT_OIDC_CLIENT_ID not set - see vault-oidc-bootstrap secret}"
+    OIDC_CLIENT_SECRET="${VAULT_OIDC_CLIENT_SECRET:?VAULT_OIDC_CLIENT_SECRET not set - see vault-oidc-bootstrap secret}"
     OIDC_DISCOVERY_URL="https://auth.kube.huskypup.net/application/o/vault/"
 
     echo "==> Configuring OIDC auth (Authentik)..."

infrastructure/vault/manifests/vault-init-job.yaml

@@ -16,6 +16,19 @@ spec:
         - name: vault-init
           image: alpine/k8s:1.32.13
           command: ["/bin/bash", "/scripts/vault-init-and-store.sh"]
+          env:
+            # See the matching comment in vault-init-configmap.yaml for
+            # why this can't come from Vault itself via ESO.
+            - name: VAULT_OIDC_CLIENT_ID
+              valueFrom:
+                secretKeyRef:
+                  name: vault-oidc-bootstrap
+                  key: client_id
+            - name: VAULT_OIDC_CLIENT_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: vault-oidc-bootstrap
+                  key: client_secret
           volumeMounts:
             - name: scripts
               mountPath: /scripts

infrastructure/vps-eso/manifests/clustersecretstore.yaml

@@ -0,0 +1,88 @@
+# Multi-site active failover pilot (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - closes the gap
+# flagged 2026-08-20: manually kubectl/ssh-copying secrets between home
+# and the VPS defeats the point of having Vault at all. This gives the
+# VPS its own real Vault -> ESO pipeline instead.
+#
+# External Secrets Operator itself is installed directly via helm
+# (out-of-band, like k3s/cert-manager/Netbird - see
+# scripts/vps-bootstrap.sh's existing pattern for why those aren't
+# GitOps-managed either):
+#   helm repo add external-secrets https://charts.external-secrets.io
+#   helm install external-secrets external-secrets/external-secrets \
+#     --namespace external-secrets --create-namespace \
+#     --version 0.20.4 --set installCRDs=true
+# (0.20.4 matches home's version - see argocd-apps/infrastructure/
+# external-secrets.yaml)
+#
+# Auth: AppRole, not Kubernetes auth - home's existing vault-backend
+# ClusterSecretStore (infrastructure/vault/manifests/clustersecretstore.yaml)
+# uses Vault's kubernetes auth method, which verifies a ServiceAccount JWT
+# against THAT cluster's own API server - doesn't work for the VPS, it's a
+# completely separate k3s cluster with no federation to home's API server.
+# AppRole is the standard way to authenticate an external/non-native
+# client to Vault instead.
+#
+# Reachable via https://vault.kube.huskypup.net - a NEW public Cloudflare
+# A record (home was previously deliberately kept off the public
+# internet - this was an explicit, confirmed decision, not a default).
+# Goes through the same Istio ingress gateway that already serves other
+# public *.kube.huskypup.net hosts, so no new UniFi port-forward/NodePort
+# needed, and no Istio PeerAuthentication/AuthorizationPolicy change
+# either - unlike the ha-authentik-postgres NodePort case, this traffic
+# arrives already wrapped in a normal in-mesh call from the ingress
+# gateway's own identity, not raw external TCP straight to a pod.
+#
+# NOTE: the AppRole's issued token/secret_id CANNOT be IP-bound
+# (token_bound_cidrs / secret_id_bound_cidrs) over this path - confirmed
+# live 2026-08-20 that Vault only ever sees the ingress gateway's own pod
+# IP for any request arriving this way, never the VPS's real source IP.
+# Security boundary here is AppRole credential secrecy + the narrow
+# read-only secret/vps/* policy (vps-eso-reader), not network-level
+# restriction - this is the standard/expected shape of AppRole auth for
+# external clients generally, CIDR-binding is normally extra
+# defense-in-depth on top rather than the primary mechanism.
+#
+# One-time manual bootstrap on Vault's side (already done 2026-08-20, not
+# scripted - matches every other Vault policy/auth-method setup in this
+# repo, which are also applied by hand via `vault` CLI, not GitOps):
+#   vault auth enable approle
+#   vault policy write vps-eso-reader - <<'EOF'
+#   path "secret/data/vps/*"     { capabilities = ["read", "list"] }
+#   path "secret/metadata/vps/*" { capabilities = ["read", "list"] }
+#   EOF
+#   vault write auth/approle/role/vps-eso token_policies="vps-eso-reader" \
+#     token_ttl=1h token_max_ttl=4h secret_id_num_uses=0
+#   vault read auth/approle/role/vps-eso/role-id      # -> roleId below
+#   vault write -f auth/approle/role/vps-eso/secret-id  # -> secret_id
+#
+# vault-approle-creds is a plain Secret created manually on the VPS
+# (kubectl, not git - same reasoning as every other VPS secret, and
+# consistent regardless of "how sensitive" a given field is on its own -
+# role_id and secret_id both live in the same Secret, neither in git):
+#   kubectl -n external-secrets create secret generic vault-approle-creds \
+#     --from-literal=role_id=<from role-id above> \
+#     --from-literal=secret_id=<from secret-id above>
+# Rotate the secret_id periodically by writing a new one and patching this
+# Secret.
+apiVersion: external-secrets.io/v1
+kind: ClusterSecretStore
+metadata:
+  name: vault-backend
+spec:
+  provider:
+    vault:
+      server: https://vault.kube.huskypup.net
+      path: secret # KV v2 mount, same as home's vault-backend
+      version: v2
+      auth:
+        appRole:
+          path: approle
+          roleRef:
+            name: vault-approle-creds
+            namespace: external-secrets
+            key: role_id
+          secretRef:
+            name: vault-approle-creds
+            namespace: external-secrets
+            key: secret_id

infrastructure/vps-standby/authentik/manifests/ingress.yaml

@@ -1,6 +1,9 @@
 ---
 # See infrastructure/vps-standby/vault/manifests/ingress.yaml for the
-# vps.huskypup.net subdomain design rationale.
+# vps.huskypup.net subdomain design rationale, and for why this
+# https-redirect Middleware is needed (found live 2026-08-21 diagnosing
+# the exact same "Missing auth_url"-shaped symptom on Vault's OIDC login -
+# applies equally to every VPS-standby app on this Traefik).
 #
 # CAVEAT: Authentik's provider/application configs (redirect_uris, etc.)
 # are replicated byte-for-byte from home and point at *.kube.huskypup.net
@@ -8,6 +11,16 @@
 # won't complete correctly (redirect mismatch) until that's addressed
 # separately. Direct/local Authentik admin login still works fine for
 # browsing and verifying replicated data.
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: https-redirect
+  namespace: authentik
+spec:
+  redirectScheme:
+    scheme: https
+    permanent: true
+---
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
@@ -15,6 +28,7 @@ metadata:
   namespace: authentik
   annotations:
     cert-manager.io/cluster-issuer: letsencrypt-production
+    traefik.ingress.kubernetes.io/router.middlewares: authentik-https-redirect@kubernetescrd
 spec:
   ingressClassName: traefik
   tls:

infrastructure/vps-standby/authentik/manifests/pg-authentik-app-externalsecret.yaml

@@ -0,0 +1,29 @@
+# Multi-site active failover pilot - VPS's half of the Vault pipeline
+# (infrastructure/authentik/manifests/ha-postgres-app-pushsecret.yaml has
+# home's half, which pushes the authoritative password into Vault at
+# secret/vps/pg-authentik-app). Pulls it back down here.
+#
+# Merge (not Owner/Replace): only overwrites the `password` key, leaving
+# CNPG's own generated host/dbname/username fields on this secret intact
+# - those are correctly LOCAL to each site (this secret's `host` key
+# points at the VPS's own local -rw service, which CNPG itself still
+# needs internally, even though the app tier reads a different value via
+# the pg-authentik.ha.huskypup.net override in values.yaml).
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: pg-authentik-app-password
+  namespace: authentik
+spec:
+  refreshInterval: 5m
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: pg-authentik-app
+    creationPolicy: Merge
+  data:
+    - secretKey: password
+      remoteRef:
+        key: vps/pg-authentik-app
+        property: password

infrastructure/vps-standby/authentik/values.yaml

@@ -34,9 +34,12 @@ global:
   envFrom:
     - secretRef:
         name: authentik
-    - secretRef:
+    # The pg-authentik-app secretRef that used to live here is GONE - see
-        name: pg-authentik-app
+    # the matching comment in infrastructure/authentik/values.yaml (home's
-      prefix: AUTHENTIK_POSTGRESQL__
+    # copy of this same block) for why: its lowercase-suffixed env vars
+    # (AUTHENTIK_POSTGRESQL__host etc.) were silently winning over the
+    # explicit uppercase HOST/PORT overrides below, connecting straight to
+    # the local CNPG service the whole time regardless of what was set.
   env:
     - name: AUTHENTIK_URL
       value: "https://auth.vps.huskypup.net"

infrastructure/vps-standby/gitea/manifests/ingress.yaml

@@ -1,6 +1,18 @@
 ---
 # See infrastructure/vps-standby/vault/manifests/ingress.yaml for the
-# vps.huskypup.net subdomain design rationale.
+# vps.huskypup.net subdomain design rationale, and for why this
+# https-redirect Middleware is needed (found live 2026-08-21 diagnosing
+# the exact same symptom on this app's OIDC login).
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: https-redirect
+  namespace: gitea
+spec:
+  redirectScheme:
+    scheme: https
+    permanent: true
+---
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
@@ -8,6 +20,7 @@ metadata:
   namespace: gitea
   annotations:
     cert-manager.io/cluster-issuer: letsencrypt-production
+    traefik.ingress.kubernetes.io/router.middlewares: gitea-https-redirect@kubernetescrd
 spec:
   ingressClassName: traefik
   tls:

infrastructure/vps-standby/gitlab/manifests/cnpg-cluster.yaml

@@ -0,0 +1,144 @@
+---
+# GitLab cross-site replication Phase 1c (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - was a fully
+# independent standalone cluster (own initdb, own root/Scooby test data,
+# never meant to survive this). Now a real replica of home's pg-gitlab
+# (apps/gitlab/manifests/cnpg-cluster.yaml), matching the pattern already
+# proven for pg-authentik.
+#
+# Deployed in two stages, same as the original authentik/n8n/nextcloud
+# VPS replicas were: first as a plain bootstrap.recovery replica (this
+# commit - no externalClusters/replica streaming config yet, since that
+# needs THIS cluster's own CNPG-generated certs to exist first, which
+# only happens once the cluster itself exists), then a second commit adds
+# the streaming replication config once those certs have been copied
+# cross-cluster.
+#
+# postgresql.parameters MUST match-or-exceed home's - copied verbatim
+# from apps/gitlab/manifests/cnpg-cluster.yaml. A replica can't replay
+# WAL past a checkpoint recorded with higher settings than its own
+# ("recovery aborted because of insufficient parameter settings",
+# confirmed live for pg-authentik).
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: pg-gitlab
+  namespace: gitlab
+spec:
+  imageName: ghcr.io/cloudnative-pg/postgresql:16
+  instances: 1
+
+  # shared_buffers 1GB below needs headroom - matches home's requests/limits.
+  resources:
+    requests:
+      memory: "1Gi"
+      cpu: "50m"
+    limits:
+      memory: "4Gi"
+      cpu: "500m"
+
+  storage:
+    size: 50Gi  # matches home's size
+    storageClass: local-path
+
+  primaryUpdateStrategy: unsupervised
+
+  postgresql:
+    parameters:
+      max_connections: "400"
+      shared_buffers: "1GB"
+      effective_cache_size: "3GB"
+      maintenance_work_mem: "256MB"
+      checkpoint_completion_target: "0.9"
+      wal_buffers: "16MB"
+      default_statistics_target: "100"
+      random_page_cost: "1.1"
+      effective_io_concurrency: "200"
+      work_mem: "16MB"
+      min_wal_size: "1GB"
+      max_wal_size: "4GB"
+      max_worker_processes: "4"
+      max_parallel_workers_per_gather: "2"
+      max_parallel_workers: "4"
+
+  bootstrap:
+    recovery:
+      source: home
+
+  # Stage 2 (see /home/scooby/.claude/plans/jiggly-snacking-iverson.md):
+  # connectionParameters added alongside the existing barmanObjectStore
+  # on the SAME "home" entry - the recovery-mode initial seed stays
+  # exactly as-is, ongoing replication now streams instead of relying on
+  # WAL-archive polling (the source of the crash-looping/probe-tuning
+  # fights documented for the pg-authentik precedent). Also needs a
+  # self-referencing "vps" entry - replica.self must match an
+  # externalClusters name too, even though it's this very cluster
+  # (confirmed live: "External cluster vps not found" until one was
+  # added).
+  #
+  # Reachable via home.kube.huskypup.net:61442 - a UniFi WAN port-forward
+  # to talos-cp-01's real LAN IP, source-IP-restricted to the VPS/witness
+  # public IPs (see apps/gitlab/manifests/ha-postgres-nodeport.yaml). NOT
+  # the Netbird mesh directly - UniFi has no route to that CIDR
+  # (confirmed live for the pg-authentik precedent).
+  externalClusters:
+    - name: home
+      barmanObjectStore:
+        # serverName defaults to the externalClusters[].name, NOT the
+        # source Postgres cluster's actual metadata.name - home's own
+        # backup: block defaults serverName to "pg-gitlab" (its own
+        # name). Without this override, recovery looks under the wrong
+        # server-name prefix and finds nothing (confirmed live for the
+        # pg-authentik precedent).
+        serverName: pg-gitlab
+        destinationPath: s3://cnpg-backups/pg-gitlab
+        endpointURL: http://vps-minio.minio.svc.cluster.local:9000
+        s3Credentials:
+          accessKeyId:
+            name: vps-minio-secret
+            key: accesskey
+          secretAccessKey:
+            name: vps-minio-secret
+            key: secretkey
+      connectionParameters:
+        host: home.kube.huskypup.net
+        port: "61442"
+        dbname: gitlabhq_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-gitlab-home-replication
+        key: tls.crt
+      sslKey:
+        name: pg-gitlab-home-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-gitlab-home-ca
+        key: ca.crt
+    - name: vps
+      connectionParameters:
+        host: pg-gitlab-rw
+        port: "5432"
+        dbname: gitlabhq_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-gitlab-replication
+        key: tls.crt
+      sslKey:
+        name: pg-gitlab-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-gitlab-ca
+        key: ca.crt
+
+  # Distributed topology: both sides agree home is primary today. No
+  # replica.enabled - mutually exclusive with primary/self (confirmed
+  # live for the pg-authentik precedent).
+  replica:
+    self: vps
+    primary: home
+    source: home
+
+  monitoring:
+    enablePodMonitor: false  # no Prometheus on the VPS cluster

infrastructure/vps-standby/gitlab/manifests/ha-gitlab-secrets-externalsecret.yaml

@@ -0,0 +1,87 @@
+# GitLab cross-site replication Phase 1b - VPS's half of the Vault
+# pipeline (apps/gitlab/manifests/ha-gitlab-secrets-pushsecret.yaml has
+# home's half, which pushes the authoritative values into Vault at
+# secret/vps/gitlab-{rails,gitlab-shell,gitaly,praefect}-secret). Pulls
+# them back down here, BEFORE Phase 1c's Postgres replication goes live.
+#
+# creationPolicy: Owner (not Merge) - each of these secrets has exactly
+# one key with nothing else to preserve, unlike pg-authentik-app which
+# has CNPG-local host/dbname/username fields alongside the shared
+# password. The chart's shared-secrets job skips creating a key that
+# already exists, so having ESO own these outright is sufficient to stop
+# the VPS from ever generating its own divergent values.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: gitlab-rails-secret
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: gitlab-rails-secret
+    creationPolicy: Owner
+  data:
+    - secretKey: secrets.yml
+      remoteRef:
+        key: vps/gitlab-rails-secret
+        property: secrets.yml
+---
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: gitlab-gitlab-shell-secret
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: gitlab-gitlab-shell-secret
+    creationPolicy: Owner
+  data:
+    - secretKey: secret
+      remoteRef:
+        key: vps/gitlab-gitlab-shell-secret
+        property: secret
+---
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: gitlab-gitaly-secret
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: gitlab-gitaly-secret
+    creationPolicy: Owner
+  data:
+    - secretKey: token
+      remoteRef:
+        key: vps/gitlab-gitaly-secret
+        property: token
+---
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: gitlab-praefect-secret
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: gitlab-praefect-secret
+    creationPolicy: Owner
+  data:
+    - secretKey: token
+      remoteRef:
+        key: vps/gitlab-praefect-secret
+        property: token

infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml

@@ -0,0 +1,62 @@
+# GitLab cross-site replication - VPS's half of the Vault pipeline
+# (apps/gitlab/manifests/ha-postgres-app-pushsecret.yaml has home's half).
+# Pulls the authoritative app-role passwords back down here.
+#
+# creationPolicy REVERTED Merge -> Owner 2026-08-22: Merge assumed CNPG
+# itself would generate a base pg-gitlab-app/pg-praefect-app secret
+# (host/dbname/username keys) for this ExternalSecret to merge a
+# password into - true for a normal owning/primary Cluster, but WRONG
+# for these VPS Clusters since Phase 1c's recreate: `pg-gitlab`/
+# `pg-praefect` here are bootstrap.recovery REPLICA clusters
+# (replica.self: vps) - the `app` role is part of the replicated
+# global role/password state streamed from home's primary, so CNPG
+# correctly never creates a local owner secret for it here. Merge had
+# nothing to merge onto -> ExternalSecret condition
+# "SecretMissing/CreationPolicy=Merge" -> pg-gitlab-app never existed
+# -> vps-gitlab-webservice crash-looped on
+# ActiveRecord::DatabaseConnectionError for ~39h before this was
+# caught (confirmed live, unrelated to any same-day change). Owner:
+# the ExternalSecret creates the whole secret itself. This chart only
+# ever reads the `password` key from it - host/port/dbname/username
+# are hardcoded directly in infrastructure/vps-standby/gitlab/
+# values.yaml's global.psql/global.praefect.psql blocks, not read from
+# this secret's other keys, so a password-only Owner-created secret is
+# sufficient (confirmed live: no other consumer of this secret's keys
+# found).
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: pg-gitlab-app-password
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: pg-gitlab-app
+    creationPolicy: Owner
+  data:
+    - secretKey: password
+      remoteRef:
+        key: vps/pg-gitlab-app
+        property: password
+---
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: pg-praefect-app-password
+  namespace: gitlab
+spec:
+  refreshInterval: 5m
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: pg-praefect-app
+    creationPolicy: Owner
+  data:
+    - secretKey: password
+      remoteRef:
+        key: vps/pg-praefect-app
+        property: password

infrastructure/vps-standby/gitlab/manifests/ha-postgres-nodeport.yaml

@@ -0,0 +1,47 @@
+---
+# Exposes the VPS's pg-gitlab/pg-praefect primaries (replicas today) back
+# to home over the Netbird mesh, for GitLab cross-site replication (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - needed for
+# failback (home rejoining as a replica once it's back). Same pattern as
+# infrastructure/vps-standby/authentik/manifests/ha-postgres-nodeport.yaml.
+#
+# No UniFi port-forward or Kyverno PolicyException needed here - the
+# VPS's k3s has no NodePort restriction, and home reaching OUT to the VPS
+# already works today via the netbird-egress DaemonSet's route (confirmed
+# live all session for MinIO/CrowdSec/pg-authentik) - it's only the
+# reverse direction (external peers reaching INTO home) that needs the
+# UniFi workaround (see apps/gitlab/manifests/ha-postgres-nodeport.yaml).
+#
+# Different nodePort numbers than home's side (32442/32443) - not
+# required since these are different machines, but matches the
+# authentik precedent's convention of keeping each site's numbers
+# distinct to avoid confusion.
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-gitlab-postgres
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    cnpg.io/cluster: pg-gitlab
+    cnpg.io/instanceRole: primary
+  ports:
+    - port: 5432
+      targetPort: 5432
+      nodePort: 32444
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-praefect-postgres
+  namespace: gitlab
+spec:
+  type: NodePort
+  selector:
+    cnpg.io/cluster: pg-praefect
+    cnpg.io/instanceRole: primary
+  ports:
+    - port: 5432
+      targetPort: 5432
+      nodePort: 32445

infrastructure/vps-standby/gitlab/manifests/ingress.yaml

@@ -0,0 +1,42 @@
+---
+# Real GitLab on the VPS. https-redirect Middleware from the start this
+# time - see infrastructure/vps-standby/vault/manifests/ingress.yaml for
+# why it's needed (found live 2026-08-21: Traefik serves plain HTTP
+# alongside HTTPS unless explicitly redirected, which silently breaks
+# OIDC login since the callback URL computed client-side won't match
+# what's registered in Authentik).
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: https-redirect
+  namespace: gitlab
+spec:
+  redirectScheme:
+    scheme: https
+    permanent: true
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: vps-gitlab
+  namespace: gitlab
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-production
+    traefik.ingress.kubernetes.io/router.middlewares: gitlab-https-redirect@kubernetescrd
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - gitlab.vps.huskypup.net
+      secretName: vps-gitlab-tls
+  rules:
+    - host: gitlab.vps.huskypup.net
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: vps-gitlab-webservice-default
+                port:
+                  number: 8181

infrastructure/vps-standby/gitlab/manifests/mirror-sync-cronjob.yaml

@@ -0,0 +1,59 @@
+# Periodic git mirror: home's Scooby/Homelabv4 (the actual GitOps
+# source) -> VPS GitLab's copy of the same project. See
+# mirror-sync-secret.yaml for why this is a plain CronJob instead of
+# GitLab's native pull-mirror feature (Premium-gated, confirmed live
+# 2026-08-21). This is the VPS-standby equivalent of the original
+# plan's "Gitea pull mirror of the GitOps source" - same intent,
+# different mechanism now that real GitLab replaced Gitea here.
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: gitlab-mirror-sync
+  namespace: gitlab
+spec:
+  schedule: "*/15 * * * *"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 1
+      activeDeadlineSeconds: 600
+      template:
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: mirror-sync
+              image: alpine/git:2.43.0
+              command:
+                - sh
+                - -c
+                - |
+                  set -eu
+                  rm -rf /tmp/mirror.git
+                  git clone --mirror "https://${PULL_USERNAME}:${PULL_TOKEN}@gitlab.kube.huskypup.net/Scooby/Homelabv4.git" /tmp/mirror.git
+                  cd /tmp/mirror.git
+                  git push --mirror "https://root:${PUSH_TOKEN}@gitlab.vps.huskypup.net/Scooby/Homelabv4.git"
+                  echo "Mirror sync complete: $(date -u)"
+              env:
+                - name: PULL_USERNAME
+                  valueFrom:
+                    secretKeyRef:
+                      name: gitlab-mirror-sync-secret
+                      key: pull_username
+                - name: PULL_TOKEN
+                  valueFrom:
+                    secretKeyRef:
+                      name: gitlab-mirror-sync-secret
+                      key: pull_token
+                - name: PUSH_TOKEN
+                  valueFrom:
+                    secretKeyRef:
+                      name: gitlab-mirror-sync-secret
+                      key: push_token
+              resources:
+                requests:
+                  cpu: 25m
+                  memory: 128Mi
+                limits:
+                  memory: 512Mi

infrastructure/vps-standby/gitlab/manifests/mirror-sync-secret.yaml

@@ -0,0 +1,39 @@
+# Credentials for mirror-sync-cronjob.yaml. GitLab's native repository
+# mirroring (pull mirror) is a Premium/Ultimate-gated feature as of this
+# chart version - confirmed live 2026-08-21: the API rejects `mirror`
+# and `import_url` as valid params on this CE instance entirely
+# ("allow_merge_on_skipped_pipeline, ... are missing" - i.e. neither
+# field is even recognized). So this CronJob does the plain-git
+# equivalent (clone --mirror / push --mirror) on a schedule instead.
+#
+# - pull_username/pull_token: read-only deploy token on HOME's
+#   Scooby/Homelabv4 project (deploy tokens can't have write_repository
+#   - it's not a valid scope for them, confirmed live).
+# - push_username/push_token: a write_repository-scoped Personal Access
+#   Token for VPS GitLab's root user (deploy tokens can't push either).
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: gitlab-mirror-sync
+  namespace: gitlab
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: vault-backend
+  target:
+    name: gitlab-mirror-sync-secret
+    creationPolicy: Owner
+  data:
+    - secretKey: pull_username
+      remoteRef:
+        key: vps/gitlab-mirror
+        property: username
+    - secretKey: pull_token
+      remoteRef:
+        key: vps/gitlab-mirror
+        property: token
+    - secretKey: push_token
+      remoteRef:
+        key: vps/gitlab-mirror
+        property: push_token

infrastructure/vps-standby/gitlab/manifests/oidc-secret.yaml

@@ -0,0 +1,61 @@
+# Shares the same Authentik OAuth2 provider/client_id as home's GitLab
+# (infrastructure/authentik/gitlab-blueprint.yaml has both redirect_uris
+# registered). Same Vault path (secret/gitlab-oauth) home's copy reads
+# from - vps-eso-reader policy widened to allow this one extra path
+# (infrastructure/vps-eso/manifests/clustersecretstore.yaml's policy,
+# applied live via vault CLI, not git-tracked - matches how every other
+# Vault policy/auth-method in this repo is set up).
+#
+# `provider` key added 2026-08-21: found live that GITLAB_OMNIBUS_CONFIG
+# (this app's values.yaml extraEnv) is an omnibus-image-only convention
+# the CNG webservice image never processes - see the matching comment
+# in apps/gitlab/manifests/external-secret-oidc.yaml (home) for the
+# full story. The real mechanism needs this key to hold a whole
+# YAML-encoded provider block, not raw client_id/secret strings.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: gitlab-oidc
+  namespace: gitlab
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: vault-backend
+  target:
+    name: gitlab-oidc-secret
+    creationPolicy: Owner
+    template:
+      type: Opaque
+      data:
+        GITLAB_OIDC_CLIENT_ID: "{{ .clientId }}"
+        GITLAB_OIDC_CLIENT_SECRET: "{{ .clientSecret }}"
+        provider: |
+          name: openid_connect
+          label: Authentik
+          args:
+            name: openid_connect
+            scope:
+              - openid
+              - profile
+              - email
+            response_type: code
+            issuer: 'https://auth.kube.huskypup.net/application/o/gitlab/'
+            discovery: true
+            client_auth_method: query
+            uid_field: sub
+            send_scope_to_token_endpoint: true
+            pkce: true
+            client_options:
+              identifier: '{{ .clientId }}'
+              secret: '{{ .clientSecret }}'
+              redirect_uri: 'https://gitlab.vps.huskypup.net/users/auth/openid_connect/callback'
+  data:
+    - secretKey: clientId
+      remoteRef:
+        key: gitlab-oauth
+        property: client-id
+    - secretKey: clientSecret
+      remoteRef:
+        key: gitlab-oauth
+        property: client-secret

infrastructure/vps-standby/gitlab/manifests/praefect-cnpg-cluster.yaml

@@ -0,0 +1,108 @@
+---
+# GitLab cross-site replication Phase 1c (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - the VPS never
+# had a pg-praefect cluster before (VPS ran Gitaly without Praefect,
+# global.praefect.enabled: false). This is new, matching home's
+# apps/gitlab/manifests/praefect-cnpg-cluster.yaml as a real replica.
+#
+# Same two-stage deployment as cnpg-cluster.yaml in this directory: this
+# commit is bootstrap.recovery only, streaming config added once this
+# cluster's own certs exist to copy cross-cluster.
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: pg-praefect
+  namespace: gitlab
+spec:
+  imageName: ghcr.io/cloudnative-pg/postgresql:16
+  instances: 1
+
+  resources:
+    requests:
+      memory: "512Mi"
+      cpu: "50m"
+    limits:
+      memory: "2Gi"
+
+  storage:
+    size: 10Gi  # matches home's size
+    storageClass: local-path
+
+  primaryUpdateStrategy: unsupervised
+
+  # Must match-or-exceed home's - copied verbatim from
+  # apps/gitlab/manifests/praefect-cnpg-cluster.yaml.
+  postgresql:
+    parameters:
+      max_connections: "200"
+      shared_buffers: "512MB"
+      effective_cache_size: "1536MB"
+      maintenance_work_mem: "128MB"
+      checkpoint_completion_target: "0.9"
+      wal_buffers: "16MB"
+      default_statistics_target: "100"
+      random_page_cost: "1.1"
+      effective_io_concurrency: "200"
+      work_mem: "4MB"
+      min_wal_size: "1GB"
+      max_wal_size: "4GB"
+      max_worker_processes: "4"
+
+  bootstrap:
+    recovery:
+      source: home
+
+  # Stage 2 - same pattern as cnpg-cluster.yaml (pg-gitlab) in this same
+  # directory.
+  externalClusters:
+    - name: home
+      barmanObjectStore:
+        serverName: pg-praefect
+        destinationPath: s3://cnpg-backups/pg-praefect
+        endpointURL: http://vps-minio.minio.svc.cluster.local:9000
+        s3Credentials:
+          accessKeyId:
+            name: vps-minio-secret
+            key: accesskey
+          secretAccessKey:
+            name: vps-minio-secret
+            key: secretkey
+      connectionParameters:
+        host: home.kube.huskypup.net
+        port: "61443"
+        dbname: praefect_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-praefect-home-replication
+        key: tls.crt
+      sslKey:
+        name: pg-praefect-home-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-praefect-home-ca
+        key: ca.crt
+    - name: vps
+      connectionParameters:
+        host: pg-praefect-rw
+        port: "5432"
+        dbname: praefect_production
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-praefect-replication
+        key: tls.crt
+      sslKey:
+        name: pg-praefect-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-praefect-ca
+        key: ca.crt
+
+  replica:
+    self: vps
+    primary: home
+    source: home
+
+  monitoring:
+    enablePodMonitor: false  # no Prometheus on the VPS cluster

infrastructure/vps-standby/gitlab/manifests/redis-init-fix.yaml

@@ -0,0 +1,19 @@
+# Same fix as apps/gitlab/manifests/redis-init-fix.yaml (home) - GitLab
+# chart 7.7.0's initializer ordering bug: 7_prometheus_metrics.rb loads
+# before 7_redis.rb (alphabetical 'p' < 'r'), triggering Redis cache
+# access before Gitlab::Redis::ALL_CLASSES is defined. This ConfigMap
+# provides an early initializer that loads the Redis module first.
+#
+# Mounting this into webservice/sidekiq deployments still has to be a
+# manual kubectl patch after first deploy - chart 7.7.0 doesn't support
+# extraVolumes on those workloads, same limitation home hit, and home's
+# own patch was never git-tracked either (applied live). See this
+# directory's own deploy notes for the exact patch commands.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: redis-init-fix
+  namespace: gitlab
+data:
+  0_redis_early.rb: |
+    require "gitlab/redis"

infrastructure/vps-standby/gitlab/manifests/redis-secret.yaml

@@ -0,0 +1,22 @@
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: redis-gitlab
+  namespace: gitlab
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: vault-backend
+  target:
+    name: redis-gitlab-secret
+    creationPolicy: Owner
+    template:
+      type: Opaque
+      data:
+        password: "{{ .password }}"
+  data:
+    - secretKey: password
+      remoteRef:
+        key: vps/gitlab-redis
+        property: password

infrastructure/vps-standby/gitlab/manifests/redis.yaml

@@ -0,0 +1,84 @@
+# Real GitLab on the VPS - standalone Redis, same pattern as home's
+# apps/gitlab/manifests/redis-cluster.yaml (despite that file's name,
+# it's already just a single standalone instance - "Sentinel" in the
+# filename is legacy, not what's actually running). local-path storage
+# instead of rook-ceph-block.
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: redis-gitlab
+  namespace: gitlab
+spec:
+  ports:
+    - port: 6379
+      targetPort: 6379
+      name: redis
+  selector:
+    app: redis-gitlab
+  type: ClusterIP
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: redis-gitlab
+  namespace: gitlab
+spec:
+  serviceName: redis-gitlab
+  replicas: 1
+  selector:
+    matchLabels:
+      app: redis-gitlab
+  template:
+    metadata:
+      labels:
+        app: redis-gitlab
+    spec:
+      securityContext:
+        fsGroup: 1000
+      containers:
+        - name: redis
+          image: redis:7.0-alpine
+          securityContext:
+            runAsUser: 999
+            runAsGroup: 1000
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+          ports:
+            - containerPort: 6379
+              name: redis
+          command:
+            - sh
+            - -c
+            - redis-server --appendonly yes --requirepass "$REDIS_PASSWORD"
+          env:
+            - name: REDIS_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: redis-gitlab-secret
+                  key: password
+          resources:
+            requests:
+              cpu: 25m
+              memory: 128Mi
+            limits:
+              memory: 512Mi
+          volumeMounts:
+            - name: data
+              mountPath: /data
+            - name: tmp
+              mountPath: /tmp
+      volumes:
+        - name: tmp
+          emptyDir: {}
+  volumeClaimTemplates:
+    - metadata:
+        name: data
+      spec:
+        accessModes: ["ReadWriteOnce"]
+        storageClassName: local-path
+        resources:
+          requests:
+            storage: 5Gi

infrastructure/vps-standby/gitlab/values.yaml

@@ -0,0 +1,275 @@
+# Real GitLab on the VPS - simplified single-node equivalent of
+# apps/gitlab/values.yaml (home). Same chart, same version, same
+# Authentik OIDC provider/client (shared, see infrastructure/authentik/
+# gitlab-blueprint.yaml). Deliberately lighter than home's: no Praefect
+# (pointless with 1 Gitaly node), no PgBouncer (direct CNPG connection,
+# low enough traffic not to need pooling), no SAML, no Container
+# Registry, no KAS - all addable later if actually needed, matches the
+# "keep it lighter" decision already made for the rest of vps-standby.
+
+global:
+  hosts:
+    domain: vps.huskypup.net
+    gitlab:
+      name: gitlab.vps.huskypup.net
+
+  # Ingress disabled - own Ingress + Traefik https-redirect Middleware
+  # handle routing (infrastructure/vps-standby/gitlab/manifests/ingress.yaml)
+  ingress:
+    enabled: false
+    configureCertmanager: false
+
+  edition: ce
+  time_zone: UTC
+
+  email:
+    from: 'gitlab@vps.huskypup.net'
+    display_name: GitLab (VPS)
+    reply_to: 'noreply@vps.huskypup.net'
+
+  # External PostgreSQL via CNPG (infrastructure/vps-standby/gitlab/manifests/cnpg-cluster.yaml)
+  psql:
+    host: pg-gitlab-rw.gitlab.svc.cluster.local
+    port: 5432
+    database: gitlabhq_production
+    username: app
+    password:
+      secret: pg-gitlab-app
+      key: password
+
+  # gitlab.kas.enabled: false alone doesn't stop KAS pods deploying -
+  # there's a SEPARATE global.kas.enabled toggle (default true) that
+  # actually controls it, found live 2026-08-21.
+  kas:
+    enabled: false
+
+  # Single Gitaly node, no Praefect - HA routing is pointless with 1 node
+  gitaly:
+    enabled: true
+    internal:
+      names:
+        - default
+    external: []
+
+  praefect:
+    enabled: false
+
+  # Standalone Redis (infrastructure/vps-standby/gitlab/manifests/redis.yaml)
+  redis:
+    host: redis-gitlab.gitlab.svc.cluster.local
+    port: 6379
+    auth:
+      enabled: true
+      secret: redis-gitlab-secret
+      key: password
+
+  appConfig:
+    # CORRECTED 2026-08-21 (see apps/gitlab/values.yaml (home) for the
+    # full story) - `providers:` list restored, now pointing at
+    # gitlab-oidc-secret's `provider` key which holds a real
+    # YAML-encoded provider block (manifests/oidc-secret.yaml), the
+    # actual mechanism the CNG webservice image processes.
+    # GITLAB_OMNIBUS_CONFIG below is an omnibus-image-only convention
+    # this image never reads - always a no-op here.
+    omniauth:
+      enabled: true
+      allowSingleSignOn: ['openid_connect']
+      blockAutoCreatedUsers: false
+      autoLinkUser: ['openid_connect']
+      syncProfileFromProvider: ['openid_connect']
+      syncProfileAttributes: ['email', 'name']
+      providers:
+        - secret: gitlab-oidc-secret
+          key: provider
+
+# GitLab's pre-upgrade hook checks the deployed version against a
+# previous-release ConfigMap to enforce supported upgrade paths - found
+# live 2026-08-21 that it fails with "unsupported upgrade path" on a
+# genuinely FRESH install too (confirmed: no prior chart-info ConfigMap,
+# no Helm release secret, brand new namespace) - a chart/ArgoCD
+# interaction quirk (ArgoCD doesn't do real `helm install`, so the
+# optional chart-info ConfigMap's absence apparently isn't handled the
+# way the script expects). Not relevant for a fresh install anyway - only
+# matters for real version-to-version upgrades, which is a future concern
+# once this is actually running something to upgrade FROM.
+upgradeCheck:
+  enabled: false
+
+postgresql:
+  install: false
+
+redis:
+  install: false
+
+pgbouncer:
+  enabled: false
+
+# GitLab's own bundled MinIO for object storage (uploads, artifacts) -
+# separate from vps-minio (which backs CNPG WAL archiving for other apps)
+minio:
+  persistence:
+    storageClass: local-path
+    size: 20Gi
+  resources:
+    requests:
+      cpu: 25m
+      memory: 256Mi
+    limits:
+      memory: 1Gi
+
+registry:
+  enabled: false
+
+gitlab:
+  webservice:
+    minReplicas: 1
+    maxReplicas: 1
+    # 1 worker process, not 2 - single-node VPS, already sharing the box
+    # with several other unrelated services (barman, k3s itself, node/
+    # gunicorn apps) that keep it chronically CPU-overcommitted (load
+    # average ~22-26 sustained on 6 cores, confirmed live 2026-08-21,
+    # unrelated to GitLab). Less to fork/boot per pod start.
+    workerProcesses: 1
+    deployment:
+      # Recreate, not the chart default RollingUpdate - with only 1
+      # replica, RollingUpdate still surges an extra pod during every
+      # rollout, and this box can't handle 2 concurrent GitLab Ruby
+      # boots (confirmed live 2026-08-21: load average spiked past 39
+      # on 6 cores, each pod starving the other's boot until both got
+      # liveness-killed - a feedback loop that never converges). No
+      # real availability cost since there's only 1 replica anyway.
+      strategy:
+        type: Recreate
+      # Chart defaults (initialDelaySeconds:20, periodSeconds:60,
+      # failureThreshold:3 => ~200s grace) aren't enough on this
+      # contended box - confirmed live 2026-08-21 the webservice
+      # (puma) container was repeatedly SIGTERM'd by the liveness
+      # probe mid-`require` (still loading gems via bootsnap), never
+      # finishing boot, restarting from zero every time. Loosened so
+      # a slow boot survives instead of getting killed and restarted
+      # forever.
+      livenessProbe:
+        initialDelaySeconds: 60
+        periodSeconds: 30
+        timeoutSeconds: 30
+        successThreshold: 1
+        failureThreshold: 15
+
+    extraEnvFrom:
+      GITLAB_OIDC_CLIENT_ID:
+        secretKeyRef:
+          name: gitlab-oidc-secret
+          key: GITLAB_OIDC_CLIENT_ID
+      GITLAB_OIDC_CLIENT_SECRET:
+        secretKeyRef:
+          name: gitlab-oidc-secret
+          key: GITLAB_OIDC_CLIENT_SECRET
+    extraEnv:
+      GITLAB_OMNIBUS_CONFIG: |
+        gitlab_rails['omniauth_enabled'] = true
+        gitlab_rails['omniauth_allow_single_sign_on'] = ['openid_connect']
+        gitlab_rails['omniauth_block_auto_created_users'] = false
+        gitlab_rails['omniauth_auto_link_user'] = ['openid_connect']
+        gitlab_rails['omniauth_sync_profile_from_provider'] = ['openid_connect']
+        gitlab_rails['omniauth_sync_profile_attributes'] = ['email', 'name']
+
+        gitlab_rails['omniauth_providers'] = [
+          {
+            'name' => 'openid_connect',
+            'label' => 'Authentik',
+            'args' => {
+              'name' => 'openid_connect',
+              'scope' => ['openid', 'profile', 'email'],
+              'response_type' => 'code',
+              'issuer' => 'https://auth.kube.huskypup.net/application/o/gitlab/',
+              'discovery' => true,
+              'client_auth_method' => 'query',
+              'uid_field' => 'sub',
+              'send_scope_to_token_endpoint' => true,
+              'pkce' => true,
+              'client_options' => {
+                'identifier' => ENV['GITLAB_OIDC_CLIENT_ID'],
+                'secret' => ENV['GITLAB_OIDC_CLIENT_SECRET'],
+                'redirect_uri' => 'https://gitlab.vps.huskypup.net/users/auth/openid_connect/callback'
+              }
+            }
+          }
+        ]
+    resources:
+      requests:
+        cpu: 150m
+        memory: 1536Mi
+      limits:
+        memory: 3Gi
+
+  kas:
+    enabled: false
+
+  gitaly:
+    persistence:
+      storageClass: local-path
+      size: 30Gi
+    resources:
+      requests:
+        cpu: 50m
+        memory: 256Mi
+      limits:
+        cpu: "1"
+        memory: 1Gi
+
+  gitlab-runner:
+    install: false
+
+  gitlab-exporter:
+    enabled: true
+    metrics:
+      enabled: false  # no Prometheus on the VPS cluster
+
+  sidekiq:
+    minReplicas: 1
+    maxReplicas: 1
+    # Recreate, not RollingUpdate - see gitlab.webservice.deployment.
+    # strategy above for why.
+    strategy:
+      type: Recreate
+    # Same probe-loosening as webservice.deployment above, same root
+    # cause (confirmed live 2026-08-21) - sidekiq's own boot was
+    # getting SIGTERM'd mid-`require` by the liveness probe on this
+    # CPU-contended box, every ~90-150s, resetting to zero each time
+    # and never actually finishing.
+    # Loosened further 2026-08-21: even with the first pass above, 3
+    # consecutive attempts all reached "Booting Sidekiq" (the real
+    # engine actually starts, confirmed via "Listening on queues" in
+    # logs) but never opened the health-check port (3808) within the
+    # ~8min grace window - the box's ambient background load (gitea,
+    # restic backup, barman, k3s itself - all unrelated to GitLab, load
+    # average sustained 35-55 on 6 cores) is apparently severe enough to
+    # starve that specific thread's CPU scheduling even after the main
+    # process is otherwise up. Genuine forward progress each time, not
+    # a hard hang, so more patience rather than a code fix.
+    livenessProbe:
+      initialDelaySeconds: 120
+      periodSeconds: 30
+      timeoutSeconds: 30
+      successThreshold: 1
+      failureThreshold: 30
+    resources:
+      requests:
+        cpu: 25m
+        memory: 384Mi
+      limits:
+        cpu: 250m
+        memory: 1536Mi
+
+certmanager:
+  install: false
+  installCRDs: false
+
+prometheus:
+  install: false
+
+nginx-ingress:
+  enabled: false
+
+gitlab-runner:
+  install: false

infrastructure/vps-standby/n8n/manifests/ingress.yaml

@@ -1,6 +1,18 @@
 ---
 # See infrastructure/vps-standby/vault/manifests/ingress.yaml for the
-# vps.huskypup.net subdomain design rationale.
+# vps.huskypup.net subdomain design rationale, and for why this
+# https-redirect Middleware is needed (found live 2026-08-21 diagnosing
+# the exact same symptom on this app's OIDC login).
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: https-redirect
+  namespace: n8n
+spec:
+  redirectScheme:
+    scheme: https
+    permanent: true
+---
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
@@ -8,6 +20,7 @@ metadata:
   namespace: n8n
   annotations:
     cert-manager.io/cluster-issuer: letsencrypt-production
+    traefik.ingress.kubernetes.io/router.middlewares: n8n-https-redirect@kubernetescrd
 spec:
   ingressClassName: traefik
   tls:

infrastructure/vps-standby/nextcloud/manifests/ingress.yaml

@@ -1,6 +1,18 @@
 ---
 # See infrastructure/vps-standby/vault/manifests/ingress.yaml for the
-# vps.huskypup.net subdomain design rationale.
+# vps.huskypup.net subdomain design rationale, and for why this
+# https-redirect Middleware is needed (found live 2026-08-21 diagnosing
+# the exact same symptom on this app's OIDC login).
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: https-redirect
+  namespace: nextcloud
+spec:
+  redirectScheme:
+    scheme: https
+    permanent: true
+---
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
@@ -8,6 +20,7 @@ metadata:
   namespace: nextcloud
   annotations:
     cert-manager.io/cluster-issuer: letsencrypt-production
+    traefik.ingress.kubernetes.io/router.middlewares: nextcloud-https-redirect@kubernetescrd
 spec:
   ingressClassName: traefik
   tls:

infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml

@@ -16,8 +16,20 @@
 #   kubectl -n nextcloud create secret generic nextcloud-restic-password \
 #     --from-literal=password=<home's nextcloud-restic-password>
 #
-# Nextcloud itself isn't running (replicas: 0, see values.yaml) so there's
+# CORRECTED 2026-08-22: the "replicas: 0" assumption below was wrong -
-# no live-write conflict risk overwriting the PVC on every run.
+# confirmed live vps-nextcloud actually runs continuously at replicas: 1
+# (no replicas key was ever set in values.yaml; the chart's own default
+# applies). This job had been silently failing every run for 2+ days as
+# a result: restic couldn't overwrite the live pod's config.php (mode
+# 0640, owned by www-data/33; this job runs as UID 65534 with only
+# group 33 via fsGroup, so it can read but not write that one file) -
+# "ignoring error for /data/config/config.php: permission denied",
+# Fatal: 1 error, even though the other 31000+ files/15GiB restored
+# fine every time. Fixed by excluding config.php from the restore
+# below rather than chasing permissions - overwriting a live, running
+# app's own config.php from a background restore job is the wrong
+# move regardless (it holds this site's actual current db credentials,
+# etc.), not just a permissions bug to route around.
 apiVersion: batch/v1
 kind: CronJob
 metadata:
@@ -64,7 +76,7 @@ spec:
                   fi
 
                   echo "==> Restoring latest snapshot into /data..."
-                  restic restore latest --tag nextcloud --host nextcloud-k8s --target /
+                  restic restore latest --tag nextcloud --host nextcloud-k8s --target / --exclude /data/config/config.php
 
                   echo "==> Done."
               env:

infrastructure/vps-standby/vault/manifests/ingress.yaml

@@ -4,6 +4,29 @@
 # (vault.vps.huskypup.net) that doesn't depend on home being up at all -
 # see the vps.huskypup.net subdomain design note in
 # infrastructure/vps-standby/cert-manager/manifests/cluster-issuer.yaml.
+#
+# https-redirect Middleware: found live 2026-08-21 that this Ingress's
+# tls: block makes HTTPS available but does NOT stop Traefik from ALSO
+# serving plain HTTP on the same host - a browser landing on
+# http://vault.vps.huskypup.net (no redirect) gets a Vault UI that
+# computes its OIDC callback URL using window.location.origin, i.e.
+# http://..., which doesn't match the https:// entry registered in
+# allowed_redirect_uris (auth/oidc/role/default) - Vault silently returns
+# an empty auth_url ("Missing auth_url" in the UI) rather than an
+# obviously-DNS/config-looking error. Applies to every VPS-standby app on
+# this same Traefik, not just Vault (same fix needed and applied in each
+# of infrastructure/vps-standby/{gitea,n8n,nextcloud,authentik}/manifests/
+# ingress.yaml too).
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: https-redirect
+  namespace: vault
+spec:
+  redirectScheme:
+    scheme: https
+    permanent: true
+---
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
@@ -11,6 +34,7 @@ metadata:
   namespace: vault
   annotations:
     cert-manager.io/cluster-issuer: letsencrypt-production
+    traefik.ingress.kubernetes.io/router.middlewares: vault-https-redirect@kubernetescrd
 spec:
   ingressClassName: traefik
   tls: