authentik HA: fix Istio ambient mesh blocking VPS/witness streaming replication

CNPG's new externalClusters connectionParameters were configured correctly
but streaming replication was silently failing - pg_stat_wal_receiver on
the VPS replica showed 0 rows, logs repeated 'could not connect to the
primary server: ... server closed the connection unexpectedly' every few
minutes.

Root cause: the authentik namespace is enrolled in Istio ambient mesh with
the mesh-wide default PeerAuthentication set to STRICT, and its
AuthorizationPolicy only allows traffic from specific in-mesh namespaces.
Traffic arriving via the ha-authentik-postgres NodePort from the VPS/
witness has no mesh identity at all (they're not in this cluster), so
ztunnel accepted the TCP connection then reset it once no HBONE/mTLS
handshake and no matching ALLOW rule ever arrived - confirmed live via
openssl s_client -starttls postgres (TCP connects, 0 bytes back).

Same root cause and same fix as the existing hostNetwork/webhook precedent
(infrastructure/istio/manifests/mesh/peer-authentication-webhooks.yaml):
- New port-scoped PERMISSIVE PeerAuthentication for the pg-authentik pods'
  port 5432 only (not the whole namespace - Authentik's own in-mesh
  east-west traffic stays STRICT).
- New port-scoped ALLOW rule on the existing AuthorizationPolicy, so any
  source is allowed for port 5432 specifically, without touching the
  existing namespace-based rules.

Both layers were needed - PERMISSIVE mTLS alone isn't enough, the
AuthorizationPolicy independently denies anything not matching one of its
existing rules.

Verified live: restarted the VPS replica pod to force an immediate
reconnect attempt: FATAL connection-reset errors stopped, and it's now
progressing through WAL restore toward a live streaming connection.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Scooby Husky
2026-08-19 00:47:34 -05:00
co-authored by Claude Sonnet 5
parent a21a8601f3
commit 3e7643e67e
2 changed files with 54 additions and 0 deletions
@@ -117,6 +117,20 @@ spec:
- source:
namespaces:
- prometheus
# Multi-site active failover pilot (see
# /home/scooby/.claude/plans/jiggly-snacking-iverson.md): CNPG
# streaming replication from the VPS/witness, arriving via the
# ha-authentik-postgres NodePort - no mesh identity at all (they're
# not in this cluster), so no `source.namespaces` rule above can ever
# match them. Scoped by destination port instead of source, matching
# the port-scoped PERMISSIVE PeerAuthentication in
# ha-postgres-peerauth.yaml (same root cause, same fix, one layer up -
# mTLS being allowed through isn't enough by itself, this ALLOW policy
# independently denies anything not matching one of the rules above).
- to:
- operation:
ports:
- "5432"
---
# --- Grafana: Allow ingress + Prometheus datasource queries + scraping ---