mirror of
https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
synced 2026-08-20 23:16:49 +00:00
authentik HA: fix Istio ambient mesh blocking VPS/witness streaming replication
CNPG's new externalClusters connectionParameters were configured correctly but streaming replication was silently failing - pg_stat_wal_receiver on the VPS replica showed 0 rows, logs repeated 'could not connect to the primary server: ... server closed the connection unexpectedly' every few minutes. Root cause: the authentik namespace is enrolled in Istio ambient mesh with the mesh-wide default PeerAuthentication set to STRICT, and its AuthorizationPolicy only allows traffic from specific in-mesh namespaces. Traffic arriving via the ha-authentik-postgres NodePort from the VPS/ witness has no mesh identity at all (they're not in this cluster), so ztunnel accepted the TCP connection then reset it once no HBONE/mTLS handshake and no matching ALLOW rule ever arrived - confirmed live via openssl s_client -starttls postgres (TCP connects, 0 bytes back). Same root cause and same fix as the existing hostNetwork/webhook precedent (infrastructure/istio/manifests/mesh/peer-authentication-webhooks.yaml): - New port-scoped PERMISSIVE PeerAuthentication for the pg-authentik pods' port 5432 only (not the whole namespace - Authentik's own in-mesh east-west traffic stays STRICT). - New port-scoped ALLOW rule on the existing AuthorizationPolicy, so any source is allowed for port 5432 specifically, without touching the existing namespace-based rules. Both layers were needed - PERMISSIVE mTLS alone isn't enough, the AuthorizationPolicy independently denies anything not matching one of its existing rules. Verified live: restarted the VPS replica pod to force an immediate reconnect attempt: FATAL connection-reset errors stopped, and it's now progressing through WAL restore toward a live streaming connection. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
a21a8601f3
commit
3e7643e67e
@@ -0,0 +1,40 @@
|
||||
# Multi-site active failover pilot (see
|
||||
# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - CNPG streaming
|
||||
# replication from the VPS was silently failing: `pg_stat_wal_receiver` on
|
||||
# the VPS replica showed 0 rows, and its logs showed a repeating
|
||||
# "could not connect to the primary server: ... server closed the
|
||||
# connection unexpectedly" every few minutes (confirmed live 2026-08-19).
|
||||
#
|
||||
# Root cause: the `authentik` namespace is enrolled in Istio's ambient mesh
|
||||
# (`istio.io/dataplane-mode: ambient`) and the mesh-wide default
|
||||
# PeerAuthentication (istio-system/default) is STRICT - ztunnel requires a
|
||||
# valid mesh (HBONE/SPIFHE) identity for ALL traffic to pods in this
|
||||
# namespace, including traffic arriving via the ha-authentik-postgres
|
||||
# NodePort from the VPS/witness (which have no mesh identity at all -
|
||||
# they're not in this cluster). ztunnel accepts the raw TCP connection then
|
||||
# resets it once it can't complete an mTLS handshake it never receives -
|
||||
# exactly matching the "server closed the connection unexpectedly"
|
||||
# symptom. Confirmed via `openssl s_client -starttls postgres`: TCP
|
||||
# connects, the postgres SSLRequest byte is sent, 0 bytes come back.
|
||||
#
|
||||
# Same root cause and same fix as the existing precedent for this exact
|
||||
# problem (infrastructure/istio/manifests/mesh/peer-authentication-webhooks.yaml
|
||||
# - CrowdSec's hostNetwork bouncer / the API server's webhook calls): allow
|
||||
# PERMISSIVE (mTLS or plaintext) inbound. Scoped here to just the CNPG
|
||||
# primary pod's port 5432 via portLevelMtls, rather than the whole
|
||||
# namespace like that precedent does - Authentik's own in-mesh east-west
|
||||
# traffic (server/worker -> everything else) should stay STRICT.
|
||||
apiVersion: security.istio.io/v1
|
||||
kind: PeerAuthentication
|
||||
metadata:
|
||||
name: allow-ha-postgres-replication
|
||||
namespace: authentik
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
cnpg.io/cluster: pg-authentik
|
||||
mtls:
|
||||
mode: STRICT
|
||||
portLevelMtls:
|
||||
"5432":
|
||||
mode: PERMISSIVE
|
||||
Reference in New Issue
Block a user