authentik HA: fix Istio ambient mesh blocking VPS/witness streaming replication

CNPG's new externalClusters connectionParameters were configured correctly
but streaming replication was silently failing - pg_stat_wal_receiver on
the VPS replica showed 0 rows, logs repeated 'could not connect to the
primary server: ... server closed the connection unexpectedly' every few
minutes.

Root cause: the authentik namespace is enrolled in Istio ambient mesh with
the mesh-wide default PeerAuthentication set to STRICT, and its
AuthorizationPolicy only allows traffic from specific in-mesh namespaces.
Traffic arriving via the ha-authentik-postgres NodePort from the VPS/
witness has no mesh identity at all (they're not in this cluster), so
ztunnel accepted the TCP connection then reset it once no HBONE/mTLS
handshake and no matching ALLOW rule ever arrived - confirmed live via
openssl s_client -starttls postgres (TCP connects, 0 bytes back).

Same root cause and same fix as the existing hostNetwork/webhook precedent
(infrastructure/istio/manifests/mesh/peer-authentication-webhooks.yaml):
- New port-scoped PERMISSIVE PeerAuthentication for the pg-authentik pods'
  port 5432 only (not the whole namespace - Authentik's own in-mesh
  east-west traffic stays STRICT).
- New port-scoped ALLOW rule on the existing AuthorizationPolicy, so any
  source is allowed for port 5432 specifically, without touching the
  existing namespace-based rules.

Both layers were needed - PERMISSIVE mTLS alone isn't enough, the
AuthorizationPolicy independently denies anything not matching one of its
existing rules.

Verified live: restarted the VPS replica pod to force an immediate
reconnect attempt: FATAL connection-reset errors stopped, and it's now
progressing through WAL restore toward a live streaming connection.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

infrastructure/authentik/manifests/cnpg-cluster.yaml

@@ -76,3 +76,80 @@ spec:
   monitoring:
     enablePodMonitor: true
 
+  # Multi-site active failover pilot (see
+  # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - streaming
+  # replication to/from the VPS's pg-authentik cluster, alongside (not
+  # replacing) the barmanObjectStore backup above.
+  #
+  # Auth: reuses CNPG's own generated streaming_replica client cert
+  # (CN=streaming_replica, issued by the VPS's own pg-authentik CA) -
+  # copied here manually as pg-authentik-vps-{ca,replication} (kubectl,
+  # not git - same pattern as every other cross-cluster secret tonight).
+  # This is the exact credential CNPG's own pg_hba.conf already trusts
+  # for streaming replication, nothing new to provision.
+  #
+  # sslmode: verify-ca (not verify-full) - the VPS's server cert was
+  # issued for its in-cluster DNS names, not for the NodePort IP address
+  # (100.108.113.41) home actually dials; verify-ca still validates the
+  # cert chain/CA trust and encrypts the connection, just skips hostname
+  # matching.
+  #
+  # Reachable via 100.108.113.41 (VPS's Netbird IP) - home reaching OUT
+  # to the VPS already works today via the netbird-egress DaemonSet's
+  # route (same path used for MinIO/CrowdSec backups all session) - no
+  # UniFi port-forward needed for this direction, only the reverse
+  # (VPS/witness reaching INTO home) needed that.
+  # CNPG's distributed-topology validation requires both replica.self and
+  # replica.primary to reference names present in THIS cluster's OWN
+  # externalClusters list - including a self-referencing entry (confirmed
+  # live: "External cluster home not found" until one was added, even
+  # though "home" is this very cluster). The "home" entry's
+  # connectionParameters point at its own local -rw service - never
+  # actually dialed while primary: home (that's this cluster), it only
+  # exists to satisfy the name-reference validation.
+  externalClusters:
+    - name: home
+      connectionParameters:
+        host: pg-authentik-rw
+        port: "5432"
+        dbname: app
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-authentik-replication
+        key: tls.crt
+      sslKey:
+        name: pg-authentik-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-authentik-ca
+        key: ca.crt
+    - name: vps
+      connectionParameters:
+        host: 100.108.113.41
+        port: "32433"
+        dbname: app
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-authentik-vps-replication
+        key: tls.crt
+      sslKey:
+        name: pg-authentik-vps-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-authentik-vps-ca
+        key: ca.crt
+
+  # Distributed topology: both sides agree home is primary today. Failover
+  # flips the VPS's replica.primary to "vps" (+ promotion token) - see the
+  # plan doc's failover-watcher section. Failback is the same in reverse,
+  # deliberately manual. No replica.enabled here - that flag belongs to
+  # CNPG's older single-source replica-cluster mode and is mutually
+  # exclusive with primary/self (confirmed live: "replica mode enabled is
+  # not compatible with the primary field").
+  replica:
+    self: home
+    primary: home
+    source: home  # required unconditionally by the admission webhook, even though home isn't actually replicating from anyone right now (self == primary)
+

infrastructure/authentik/manifests/ha-postgres-peerauth.yaml

@@ -0,0 +1,40 @@
+# Multi-site active failover pilot (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - CNPG streaming
+# replication from the VPS was silently failing: `pg_stat_wal_receiver` on
+# the VPS replica showed 0 rows, and its logs showed a repeating
+# "could not connect to the primary server: ... server closed the
+# connection unexpectedly" every few minutes (confirmed live 2026-08-19).
+#
+# Root cause: the `authentik` namespace is enrolled in Istio's ambient mesh
+# (`istio.io/dataplane-mode: ambient`) and the mesh-wide default
+# PeerAuthentication (istio-system/default) is STRICT - ztunnel requires a
+# valid mesh (HBONE/SPIFHE) identity for ALL traffic to pods in this
+# namespace, including traffic arriving via the ha-authentik-postgres
+# NodePort from the VPS/witness (which have no mesh identity at all -
+# they're not in this cluster). ztunnel accepts the raw TCP connection then
+# resets it once it can't complete an mTLS handshake it never receives -
+# exactly matching the "server closed the connection unexpectedly"
+# symptom. Confirmed via `openssl s_client -starttls postgres`: TCP
+# connects, the postgres SSLRequest byte is sent, 0 bytes come back.
+#
+# Same root cause and same fix as the existing precedent for this exact
+# problem (infrastructure/istio/manifests/mesh/peer-authentication-webhooks.yaml
+# - CrowdSec's hostNetwork bouncer / the API server's webhook calls): allow
+# PERMISSIVE (mTLS or plaintext) inbound. Scoped here to just the CNPG
+# primary pod's port 5432 via portLevelMtls, rather than the whole
+# namespace like that precedent does - Authentik's own in-mesh east-west
+# traffic (server/worker -> everything else) should stay STRICT.
+apiVersion: security.istio.io/v1
+kind: PeerAuthentication
+metadata:
+  name: allow-ha-postgres-replication
+  namespace: authentik
+spec:
+  selector:
+    matchLabels:
+      cnpg.io/cluster: pg-authentik
+  mtls:
+    mode: STRICT
+  portLevelMtls:
+    "5432":
+      mode: PERMISSIVE

infrastructure/ha-failover/manifests/etcd.yaml

@@ -9,11 +9,23 @@
 #
 # Pinned to a specific node (talos-cp-01) via nodeSelector so its
 # advertised peer/client URLs (which must be stable, not "whichever node
-# it landed on today") stay correct. Reachable from the VPS/witness via
+# it landed on today") stay correct.
-# that node's real IP - confirmed live 2026-08-19 that home nodes are
+#
-# directly reachable from Netbird mesh peers on their node IP (the
+# Reachable from the VPS/witness via a UniFi WAN port-forward + a
-# netbird-egress DaemonSet's route) - and exposed via a NodePort Service,
+# source-IP-restricted WAN_IN firewall rule (limited to just the VPS and
-# which needs infrastructure/kyverno/policies/ha-failover-nodeport-exception.yaml
+# witness public IPs) - NOT via the Netbird mesh directly. talos-cp-01
+# does have a real, host-bound Netbird IP (100.108.42.109 on its wt0
+# interface, confirmed live via `talosctl get addresses`), but that's a
+# dead end for inbound WAN traffic: UniFi's router only has a route to
+# its own directly-connected LAN (172.28.101.0/24), nothing advertises
+# 100.108.0.0/16 to it, so a port-forward targeting the Netbird IP would
+# never actually route. The port-forward instead targets talos-cp-01's
+# real LAN IP (172.28.101.41) - NodePort Services bind on every interface
+# on a node, so the same ports are reachable there too, and that's an
+# address UniFi can actually route to natively.
+#
+# The NodePort Service itself needs
+# infrastructure/kyverno/policies/ha-failover-nodeport-exception.yaml
 # (disallow-nodeport-services is enforced cluster-wide otherwise).
 apiVersion: v1
 kind: Namespace
@@ -56,10 +68,10 @@ spec:
             - --name=home
             - --data-dir=/var/lib/etcd/data
             - --listen-client-urls=http://0.0.0.0:2379
-            - --advertise-client-urls=http://100.108.42.109:32379
+            - --advertise-client-urls=http://home.kube.huskypup.net:61379
             - --listen-peer-urls=http://0.0.0.0:2380
-            - --initial-advertise-peer-urls=http://100.108.42.109:32380
+            - --initial-advertise-peer-urls=http://home.kube.huskypup.net:61380
-            - --initial-cluster=home=http://100.108.42.109:32380,vps=http://100.108.113.41:2380,witness=http://100.108.130.74:2380
+            - --initial-cluster=home=http://home.kube.huskypup.net:61380,vps=http://100.108.113.41:2380,witness=http://100.108.130.74:2380
             - --initial-cluster-state=new
             - --initial-cluster-token=ha-failover-quorum
           ports:

infrastructure/istio/manifests/mesh/authorization-policies.yaml

@@ -117,6 +117,20 @@ spec:
         - source:
             namespaces:
               - prometheus
+    # Multi-site active failover pilot (see
+    # /home/scooby/.claude/plans/jiggly-snacking-iverson.md): CNPG
+    # streaming replication from the VPS/witness, arriving via the
+    # ha-authentik-postgres NodePort - no mesh identity at all (they're
+    # not in this cluster), so no `source.namespaces` rule above can ever
+    # match them. Scoped by destination port instead of source, matching
+    # the port-scoped PERMISSIVE PeerAuthentication in
+    # ha-postgres-peerauth.yaml (same root cause, same fix, one layer up -
+    # mTLS being allowed through isn't enough by itself, this ALLOW policy
+    # independently denies anything not matching one of the rules above).
+    - to:
+        - operation:
+            ports:
+              - "5432"
 
 ---
 # --- Grafana: Allow ingress + Prometheus datasource queries + scraping ---

infrastructure/vps-standby/authentik/manifests/cnpg-cluster.yaml

@@ -68,13 +68,47 @@ spec:
 
   bootstrap:
     recovery:
-      source: home-backup
+      source: home
 
+  # Multi-site active failover pilot (see
+  # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - the
+  # bootstrap.recovery above (barmanObjectStore) stays exactly as-is for
+  # the initial seed; connectionParameters below adds real streaming
+  # replication for ongoing sync, replacing the old WAL-archive-polling
+  # behavior (the source of the crash-looping/probe-tuning fights from
+  # earlier tonight - streaming doesn't need to restart postgres to check
+  # for new WAL).
+  #
+  # Auth: reuses CNPG's own generated streaming_replica client cert
+  # (CN=streaming_replica, issued by home's own pg-authentik CA) - copied
+  # here manually as pg-authentik-home-{ca,replication} (kubectl, not
+  # git). Same credential CNPG's own pg_hba.conf already trusts, nothing
+  # new to provision.
+  #
+  # Reachable via home.kube.huskypup.net:61432 - a UniFi WAN port-forward
+  # to talos-cp-01's real LAN IP (172.28.101.41), source-IP-restricted to
+  # just the VPS and witness public IPs. NOT the Netbird mesh directly -
+  # UniFi has no route to that CIDR, confirmed live (see
+  # infrastructure/ha-failover/manifests/etcd.yaml for the full story).
+  # externalClusters[].name is now "home" (was "home-backup") - reused
+  # consistently for bootstrap.recovery.source, replica.primary, AND the
+  # connectionParameters below, since CNPG's distributed-topology
+  # validation requires replica.primary to match an externalClusters name
+  # exactly (confirmed live: "External cluster home not found" when it
+  # didn't). serverName inside barmanObjectStore is unaffected by this
+  # rename - it's an explicit separate override, not derived from the
+  # entry name.
+  #
+  # Also needs a self-referencing "vps" entry below (same reason -
+  # replica.self must match an externalClusters name too, even though
+  # it's this very cluster) - its connectionParameters point at VPS's own
+  # local -rw service, never actually dialed while this cluster is a
+  # replica.
   externalClusters:
-    - name: home-backup
+    - name: home
       barmanObjectStore:
-        # serverName defaults to the externalClusters[].name ("home-backup")
+        # serverName defaults to the externalClusters[].name here, NOT
-        # here, NOT the source Postgres cluster's actual name - but home's
+        # the source Postgres cluster's actual name - but home's
         # barmanObjectStore backup: block (on the pg-authentik Cluster
         # itself) defaults serverName to its own metadata.name
         # ("pg-authentik"). Without this override, recovery looks for
@@ -91,10 +125,47 @@ spec:
           secretAccessKey:
             name: vps-minio-secret
             key: secretkey
+      connectionParameters:
+        host: home.kube.huskypup.net
+        port: "61432"
+        dbname: app
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-authentik-home-replication
+        key: tls.crt
+      sslKey:
+        name: pg-authentik-home-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-authentik-home-ca
+        key: ca.crt
+    - name: vps
+      connectionParameters:
+        host: pg-authentik-rw
+        port: "5432"
+        dbname: app
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-authentik-replication
+        key: tls.crt
+      sslKey:
+        name: pg-authentik-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-authentik-ca
+        key: ca.crt
 
+  # Distributed topology: both sides agree home is primary today. No
+  # replica.enabled - that flag belongs to CNPG's older single-source
+  # replica-cluster mode and is mutually exclusive with primary/self
+  # (confirmed live: "replica mode enabled is not compatible with the
+  # primary field").
   replica:
-    enabled: true
+    self: vps
-    source: home-backup
+    primary: home
+    source: home
 
   monitoring:
     enablePodMonitor: false  # no Prometheus on the VPS cluster

infrastructure/vps-standby/authentik/manifests/ha-postgres-nodeport.yaml

@@ -0,0 +1,27 @@
+---
+# Exposes the VPS's pg-authentik primary (currently a read-only replica -
+# see cnpg-cluster.yaml's replica.enabled) to home over the Netbird mesh,
+# for the multi-site active-failover pilot. Same selector as CNPG's own
+# generated pg-authentik-rw ClusterIP Service, just NodePort instead.
+#
+# Unlike home's side of this (infrastructure/authentik/manifests/
+# ha-postgres-nodeport.yaml), no UniFi port-forward or Kyverno
+# PolicyException needed here - the VPS's k3s has no NodePort
+# restriction, and home reaching OUT to the VPS already works today via
+# the netbird-egress DaemonSet's route (confirmed live all session, same
+# path used for MinIO/CrowdSec) - it's only the reverse direction
+# (external peers reaching INTO home) that needed the UniFi workaround.
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-authentik-postgres
+  namespace: authentik
+spec:
+  type: NodePort
+  selector:
+    cnpg.io/cluster: pg-authentik
+    cnpg.io/instanceRole: primary
+  ports:
+    - port: 5432
+      targetPort: 5432
+      nodePort: 32433