home's own authentik pods reach ha-authentik-postgres via the CoreDNS
rewrite on port 61432 (not just external traffic via NodePort on 5432/
61432) - discovered live that Istio ambient's port-level mTLS/L4
authorization enforcement is keyed on the port actually dialed (61432
here), not just the pod's real containerPort (5432) traffic eventually
reaches after Service translation. The existing port-5432-only rules
(from the streaming-replication fix) didn't cover this in-cluster path,
surfacing as 'server closed the connection unexpectedly' from home's own
authentik-worker pod.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CNPG's new externalClusters connectionParameters were configured correctly
but streaming replication was silently failing - pg_stat_wal_receiver on
the VPS replica showed 0 rows, logs repeated 'could not connect to the
primary server: ... server closed the connection unexpectedly' every few
minutes.
Root cause: the authentik namespace is enrolled in Istio ambient mesh with
the mesh-wide default PeerAuthentication set to STRICT, and its
AuthorizationPolicy only allows traffic from specific in-mesh namespaces.
Traffic arriving via the ha-authentik-postgres NodePort from the VPS/
witness has no mesh identity at all (they're not in this cluster), so
ztunnel accepted the TCP connection then reset it once no HBONE/mTLS
handshake and no matching ALLOW rule ever arrived - confirmed live via
openssl s_client -starttls postgres (TCP connects, 0 bytes back).
Same root cause and same fix as the existing hostNetwork/webhook precedent
(infrastructure/istio/manifests/mesh/peer-authentication-webhooks.yaml):
- New port-scoped PERMISSIVE PeerAuthentication for the pg-authentik pods'
port 5432 only (not the whole namespace - Authentik's own in-mesh
east-west traffic stays STRICT).
- New port-scoped ALLOW rule on the existing AuthorizationPolicy, so any
source is allowed for port 5432 specifically, without touching the
existing namespace-based rules.
Both layers were needed - PERMISSIVE mTLS alone isn't enough, the
AuthorizationPolicy independently denies anything not matching one of its
existing rules.
Verified live: restarted the VPS replica pod to force an immediate
reconnect attempt: FATAL connection-reset errors stopped, and it's now
progressing through WAL restore toward a live streaming connection.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
n8n's allow-n8n-access AuthorizationPolicy only allowed traffic from the
n8n/istio-system/prometheus namespaces - unlike nextcloud/authentik's
policies, it was missing the broader ipBlocks: 10.244.0.0/16 fallback
rule. The CNPG operator (cnpg-system namespace) polling pg-n8n instances'
status API (port 8000) wasn't covered by any rule, so Istio's L4
authorization enforcement (via ztunnel - this traffic is direct pod-IP,
not routed through a Service, so waypoint/L7 never sees it) rejected the
connection outright, surfacing as a TCP reset to the operator.
This was the actual cause of pg-n8n's long-recurring 'Instance Status
Extraction Error' - confirmed 100% reproducible tonight (every single
operator poll attempt failed identically), not the intermittent
HBONE-tunnel issue it resembled from the outside. Also labeled the n8n
namespace istio.io/use-waypoint=waypoint (was in the enrollment script's
namespace list but the label was missing live - namespace parity fix,
though not itself the fix for this specific issue since this traffic
bypasses the waypoint).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
allow-netbird-operator-access (AuthorizationPolicy) and the netbird-operator
PeerAuthentication both targeted namespace 'netbird-operator', which has
never existed - the actual netbird-operator pod runs in the 'netbird'
namespace. This permanently blocked istio-mesh-config from going Synced
(kubectl apply failed: namespaces "netbird-operator" not found on every
sync attempt).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause (confirmed via ztunnel logs): the argo-cd chart's default
NetworkPolicies only allow each component's application port (e.g. 6379
for redis), but Istio ambient mode routes ALL pod-to-pod traffic through
ztunnel's HBONE tunnel on port 15008 first - so the tunnel itself was
being blocked even though the "real" port was allowed. Every inter-pod
connection in the argocd namespace hung for exactly 10s then reset;
ztunnel's own log named it directly ("connection timed out, maybe a
NetworkPolicy is blocking HBONE port 15008"). This broke argocd-server's
Redis-backed session/cluster-info caching cluster-wide and was silently
preventing the root Application from picking up new child Applications.
Fix: additive NetworkPolicy allowing ingress on 15008 for all argocd
pods (NetworkPolicies union across multiple policies selecting a pod,
so this doesn't touch/replace the chart's rendered ones - safe across
Helm upgrades).
Also: dropped argocd from the namespace-enrollment job's waypoint list.
argocd's only AuthorizationPolicy (allow-argocd-access) is a plain
source-namespace/IP match with no L7 rules - its own status shows
"attached to ztunnel", not waypoint - so forcing L7 waypoint processing
onto the namespace was unnecessary overhead, not a security requirement.
ztunnel's mTLS still fully covers it. (This was a red herring for the
HBONE bug itself, not the fix, but a valid simplification found along
the way.)
The operator watches for netbird.io/expose (presence-based), not
netbird.io/resource which was from the old Helmfile bootstrap script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The global-authentik-authz AuthorizationPolicy uses *.kube.huskypup.net
as a host match, but Kiali flagged KIA0104 because the wildcard had no
matching entry in Istio's service registry. Adding a MESH_EXTERNAL
ServiceEntry with resolution NONE registers the hostname without
affecting routing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The netbird-operator webhook service runs in the netbird namespace and
receives admission webhook calls from kube-apiserver using node IPs
(172.28.101.0/24). Without this, ztunnel rejects the connections with
"allow policies exist, but none allowed".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The ztunnel container exposes metrics on port named 'ztunnel-stats' (15020),
not 'http-monitoring'. This was preventing Prometheus from scraping ztunnel
L4 metrics, causing missing traffic in Kiali.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bitnami/kubectl has removed all version tags from Docker Hub,
leaving only :latest which is blocked by Kyverno disallow-latest-tag
policy. Switch to alpine/k8s:1.32.13 which has kubectl, bash, jq,
and uses proper version tags.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The :1.31 tag is gone from Docker Hub and :latest is blocked by Kyverno
disallow-latest-tag policy. The 1.29 image is cached on cluster nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The bitnami/kubectl:1.31 tag doesn't exist on Docker Hub. Updated all
hook jobs to use bitnami/kubectl:latest.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PreSync/PostSync hook Jobs reference argocd-hook-sa but it only existed
in the argocd namespace. Jobs running in app namespaces (crowdsec,
authentik, netbird, vault, istio-system) need the SA in their own
namespace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
