mirror of
https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
synced 2026-08-21 05:26:49 +00:00
Scooby HuskyandClaude Sonnet 5
3e7643e67e
CNPG's new externalClusters connectionParameters were configured correctly but streaming replication was silently failing - pg_stat_wal_receiver on the VPS replica showed 0 rows, logs repeated 'could not connect to the primary server: ... server closed the connection unexpectedly' every few minutes. Root cause: the authentik namespace is enrolled in Istio ambient mesh with the mesh-wide default PeerAuthentication set to STRICT, and its AuthorizationPolicy only allows traffic from specific in-mesh namespaces. Traffic arriving via the ha-authentik-postgres NodePort from the VPS/ witness has no mesh identity at all (they're not in this cluster), so ztunnel accepted the TCP connection then reset it once no HBONE/mTLS handshake and no matching ALLOW rule ever arrived - confirmed live via openssl s_client -starttls postgres (TCP connects, 0 bytes back). Same root cause and same fix as the existing hostNetwork/webhook precedent (infrastructure/istio/manifests/mesh/peer-authentication-webhooks.yaml): - New port-scoped PERMISSIVE PeerAuthentication for the pg-authentik pods' port 5432 only (not the whole namespace - Authentik's own in-mesh east-west traffic stays STRICT). - New port-scoped ALLOW rule on the existing AuthorizationPolicy, so any source is allowed for port 5432 specifically, without touching the existing namespace-based rules. Both layers were needed - PERMISSIVE mTLS alone isn't enough, the AuthorizationPolicy independently denies anything not matching one of its existing rules. Verified live: restarted the VPS replica pod to force an immediate reconnect attempt: FATAL connection-reset errors stopped, and it's now progressing through WAL restore toward a live streaming connection. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
41 lines
1.9 KiB
YAML
41 lines
1.9 KiB
YAML
# Multi-site active failover pilot (see
|
|
# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - CNPG streaming
|
|
# replication from the VPS was silently failing: `pg_stat_wal_receiver` on
|
|
# the VPS replica showed 0 rows, and its logs showed a repeating
|
|
# "could not connect to the primary server: ... server closed the
|
|
# connection unexpectedly" every few minutes (confirmed live 2026-08-19).
|
|
#
|
|
# Root cause: the `authentik` namespace is enrolled in Istio's ambient mesh
|
|
# (`istio.io/dataplane-mode: ambient`) and the mesh-wide default
|
|
# PeerAuthentication (istio-system/default) is STRICT - ztunnel requires a
|
|
# valid mesh (HBONE/SPIFHE) identity for ALL traffic to pods in this
|
|
# namespace, including traffic arriving via the ha-authentik-postgres
|
|
# NodePort from the VPS/witness (which have no mesh identity at all -
|
|
# they're not in this cluster). ztunnel accepts the raw TCP connection then
|
|
# resets it once it can't complete an mTLS handshake it never receives -
|
|
# exactly matching the "server closed the connection unexpectedly"
|
|
# symptom. Confirmed via `openssl s_client -starttls postgres`: TCP
|
|
# connects, the postgres SSLRequest byte is sent, 0 bytes come back.
|
|
#
|
|
# Same root cause and same fix as the existing precedent for this exact
|
|
# problem (infrastructure/istio/manifests/mesh/peer-authentication-webhooks.yaml
|
|
# - CrowdSec's hostNetwork bouncer / the API server's webhook calls): allow
|
|
# PERMISSIVE (mTLS or plaintext) inbound. Scoped here to just the CNPG
|
|
# primary pod's port 5432 via portLevelMtls, rather than the whole
|
|
# namespace like that precedent does - Authentik's own in-mesh east-west
|
|
# traffic (server/worker -> everything else) should stay STRICT.
|
|
apiVersion: security.istio.io/v1
|
|
kind: PeerAuthentication
|
|
metadata:
|
|
name: allow-ha-postgres-replication
|
|
namespace: authentik
|
|
spec:
|
|
selector:
|
|
matchLabels:
|
|
cnpg.io/cluster: pg-authentik
|
|
mtls:
|
|
mode: STRICT
|
|
portLevelMtls:
|
|
"5432":
|
|
mode: PERMISSIVE
|