Build real pod-egress routing to the VPS, fix everything blocked by its absence

Root cause of tonight's earlier CrowdSec/CNPG-backup workarounds: no node
in the cluster had any route into the Netbird mesh CIDR (100.108.0.0/16)
for pod-originated traffic. The per-namespace netbird 'router' pods are
inbound-only infrastructure (external peers reaching K8s services); their
own architecture has no reverse path.

Fix, in two parts:

1. infrastructure/netbird/manifests/egress-daemonset.yaml - one netbird
   client per node, hostNetwork so its wt0 interface lives in the node's
   real network namespace, plus a sidecar that adds a host route sending
   100.108.0.0/16 out via it. hostNetwork requires a scoped Kyverno
   PolicyException (infrastructure/kyverno/policies/netbird-egress-exception.yaml)
   to the disallow-host-namespaces STIG policy - narrowly for this one
   DaemonSet by name, not a namespace-wide exclusion.

2. Discovered the route alone wasn't enough for k3s NodePort traffic
   (vps-minio:30900): Netbird manages its own nftables ACLs independent of
   iptables/Kyverno, and its forward chain (netbird-rt-fwd) only permits
   *established* connections through a peer acting as a router - never new
   ones, by design, unless a Netbird 'Network Route' policy is explicitly
   configured (it isn't, for this VPS). Locally-terminated connections
   (tinyproxy) go through a separate, already-permissive ACL chain, which
   is why the CrowdSec proxy fix from earlier tonight worked. Replicated
   that working pattern for MinIO: minio-forward.service on the VPS host
   (systemd, socat) forwards 100.108.113.41:9000 -> MinIO's ClusterIP,
   avoiding the NodePort path entirely.

Re-enabled everything that was disabled/suspended earlier tonight because
of this gap, pointed at the new endpoint:
- CrowdSec CAPI/console-enroll (removed DISABLE_ONLINE_API, restored the
  VPS proxy env vars)
- n8n/nextcloud/authentik CNPG backup.barmanObjectStore
- vault-raft-snapshot CronJob (unsuspended)
- nextcloud PVC content sync CronJob endpoint

vps-minio.netbird.internal is retired everywhere - it was never actually
resolvable (Netbird has no DNS configured) even before today's routing
fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

apps/n8n/manifests/cnpg-cluster.yaml

@@ -47,29 +47,31 @@ spec:
       database: n8n
       owner: n8n
 
-  # Backup to VPS MinIO - DISABLED 2026-08-17. No node in the cluster has
+  # Backup to VPS MinIO - RE-ENABLED 2026-08-18. Real pod-egress routing to
-  # any route into the Netbird mesh for pod-originated (egress) traffic;
+  # the VPS now exists (netbird-egress DaemonSet, hostNetwork + per-node
-  # the per-namespace netbird "router" pods are inbound-only infrastructure.
+  # route into 100.108.0.0/16 - see infrastructure/netbird/manifests/
-  # continuousArchiving kept failing to connect to vps-minio.netbird.internal,
+  # egress-daemonset.yaml). Endpoint changed from the NodePort
-  # which held Ready=False permanently (real condition, not cosmetic - this
+  # (vps-minio.netbird.internal:30900 - never resolvable anyway, Netbird has
-  # is what drove n8n's CNPG health flapping in ArgoCD). Re-enable once real
+  # no DNS configured, and separately blocked by Netbird's own ACL model,
-  # pod-egress routing to the VPS exists (tracked as a separate task) - no
+  # which only permits *established* forwarded connections through a peer,
-  # other change needed, this block is otherwise complete/correct.
+  # never new ones) to a locally-terminated socat forward on the VPS host
-  # backup:
+  # itself (100.108.113.41:9000 - see minio-forward.service on the VPS),
-  #   barmanObjectStore:
+  # mirroring the pattern that already worked for the CrowdSec CAPI proxy.
-  #     destinationPath: s3://cnpg-backups/pg-n8n
+  backup:
-  #     endpointURL: http://vps-minio.netbird.internal:30900
+    barmanObjectStore:
-  #     s3Credentials:
+      destinationPath: s3://cnpg-backups/pg-n8n
-  #       accessKeyId:
+      endpointURL: http://100.108.113.41:9000
-  #         name: vps-minio-secret
+      s3Credentials:
-  #         key: accesskey
+        accessKeyId:
-  #       secretAccessKey:
+          name: vps-minio-secret
-  #         name: vps-minio-secret
+          key: accesskey
-  #         key: secretkey
+        secretAccessKey:
-  #     wal:
+          name: vps-minio-secret
-  #       compression: gzip
+          key: secretkey
-  #       maxParallel: 2
+      wal:
-  #   retentionPolicy: "30d"
+        compression: gzip
+        maxParallel: 2
+    retentionPolicy: "30d"
 
   monitoring:
     enablePodMonitor: true

apps/nextcloud/manifests/cnpg-cluster.yaml

@@ -44,30 +44,33 @@ spec:
       database: nextcloud
       owner: nextcloud
 
-  # Backup to VPS MinIO - DISABLED 2026-08-17. No node in the cluster has
+  # Backup to VPS MinIO - RE-ENABLED 2026-08-18. Real pod-egress routing to
-  # any route into the Netbird mesh for pod-originated (egress) traffic;
+  # the VPS now exists (netbird-egress DaemonSet, hostNetwork + per-node
-  # the per-namespace netbird "router" pods are inbound-only infrastructure.
+  # route into 100.108.0.0/16 - see infrastructure/netbird/manifests/
-  # continuousArchiving kept failing to connect to vps-minio.netbird.internal,
+  # egress-daemonset.yaml). Endpoint changed from the NodePort
-  # which held Ready=False permanently (real condition, not cosmetic).
+  # (vps-minio.netbird.internal:30900 - never resolvable anyway, Netbird has
-  # Re-enable once real pod-egress routing to the VPS exists (tracked as a
+  # no DNS configured, and separately blocked by Netbird's own ACL model,
-  # separate task) - no other change needed, this block is otherwise
+  # which only permits *established* forwarded connections through a peer,
-  # complete/correct. This covers the DB only - file PVC content is
+  # never new ones) to a locally-terminated socat forward on the VPS host
-  # separate, see nextcloud-pvc-sync-cronjob.yaml (same underlying gap).
+  # itself (100.108.113.41:9000 - see minio-forward.service on the VPS),
-  # backup:
+  # mirroring the pattern that already worked for the CrowdSec CAPI proxy.
-  #   barmanObjectStore:
+  # This covers the DB only - file PVC content is separate, see
-  #     destinationPath: s3://cnpg-backups/pg-nextcloud
+  # nextcloud-pvc-sync-cronjob.yaml (same fix applies there too).
-  #     endpointURL: http://vps-minio.netbird.internal:30900
+  backup:
-  #     s3Credentials:
+    barmanObjectStore:
-  #       accessKeyId:
+      destinationPath: s3://cnpg-backups/pg-nextcloud
-  #         name: vps-minio-secret
+      endpointURL: http://100.108.113.41:9000
-  #         key: accesskey
+      s3Credentials:
-  #       secretAccessKey:
+        accessKeyId:
-  #         name: vps-minio-secret
+          name: vps-minio-secret
-  #         key: secretkey
+          key: accesskey
-  #     wal:
+        secretAccessKey:
-  #       compression: gzip
+          name: vps-minio-secret
-  #       maxParallel: 2
+          key: secretkey
-  #   retentionPolicy: "30d"
+      wal:
+        compression: gzip
+        maxParallel: 2
+    retentionPolicy: "30d"
 
   monitoring:
     enablePodMonitor: true

apps/nextcloud/manifests/pvc-sync-cronjob.yaml

@@ -100,10 +100,13 @@ spec:
 
                   echo "==> Done."
               env:
-                # VPS's Netbird address - replace once bootstrapped, matches
+                # Locally-terminated socat forward on the VPS host to
+                # MinIO's ClusterIP, not the NodePort - see
                 # infrastructure/vault/manifests/raft-snapshot-cronjob.yaml
+                # for why (Netbird has no DNS, and blocks new forwarded
+                # connections through a peer by default).
                 - name: VPS_MINIO_ENDPOINT
-                  value: "vps-minio.netbird.internal:30900"
+                  value: "100.108.113.41:9000"
                 - name: MINIO_ACCESS_KEY
                   valueFrom:
                     secretKeyRef:

infrastructure/authentik/manifests/cnpg-cluster.yaml

@@ -47,29 +47,31 @@ spec:
       database: app
       owner: app
 
-  # Backup to VPS MinIO - DISABLED 2026-08-17. No node in the cluster has
+  # Backup to VPS MinIO - RE-ENABLED 2026-08-18. Real pod-egress routing to
-  # any route into the Netbird mesh for pod-originated (egress) traffic;
+  # the VPS now exists (netbird-egress DaemonSet, hostNetwork + per-node
-  # the per-namespace netbird "router" pods are inbound-only infrastructure.
+  # route into 100.108.0.0/16 - see infrastructure/netbird/manifests/
-  # continuousArchiving kept failing to connect to vps-minio.netbird.internal,
+  # egress-daemonset.yaml). Endpoint changed from the NodePort
-  # which held Ready=False permanently (real condition, not cosmetic).
+  # (vps-minio.netbird.internal:30900 - never resolvable anyway, Netbird has
-  # Re-enable once real pod-egress routing to the VPS exists (tracked as a
+  # no DNS configured, and separately blocked by Netbird's own ACL model,
-  # separate task) - no other change needed, this block is otherwise
+  # which only permits *established* forwarded connections through a peer,
-  # complete/correct.
+  # never new ones) to a locally-terminated socat forward on the VPS host
-  # backup:
+  # itself (100.108.113.41:9000 - see minio-forward.service on the VPS),
-  #   barmanObjectStore:
+  # mirroring the pattern that already worked for the CrowdSec CAPI proxy.
-  #     destinationPath: s3://cnpg-backups/pg-authentik
+  backup:
-  #     endpointURL: http://vps-minio.netbird.internal:30900
+    barmanObjectStore:
-  #     s3Credentials:
+      destinationPath: s3://cnpg-backups/pg-authentik
-  #       accessKeyId:
+      endpointURL: http://100.108.113.41:9000
-  #         name: vps-minio-secret
+      s3Credentials:
-  #         key: accesskey
+        accessKeyId:
-  #       secretAccessKey:
+          name: vps-minio-secret
-  #         name: vps-minio-secret
+          key: accesskey
-  #         key: secretkey
+        secretAccessKey:
-  #     wal:
+          name: vps-minio-secret
-  #       compression: gzip
+          key: secretkey
-  #       maxParallel: 2
+      wal:
-  #   retentionPolicy: "30d"
+        compression: gzip
+        maxParallel: 2
+    retentionPolicy: "30d"
 
   monitoring:
     enablePodMonitor: true

infrastructure/crowdsec/values.yaml

@@ -34,29 +34,20 @@ lapi:
           name: crowdsec-bouncer-key
           key: api-key
           optional: true
-    # CAPI/console-enroll disabled for now (2026-08-17). Root causes found:
+    # CAPI/console-enroll RE-ENABLED 2026-08-18. Home's public IP still gets
-    # 1) Home's public IP gets a 403 from AWS WAF in front of api.crowdsec.net
+    # a 403 from AWS WAF in front of api.crowdsec.net (an IP-level block,
-    #    (even on a bare unauthenticated request) - an IP-level block
+    # unrelated to credentials), so this routes through a tinyproxy on the
-    #    unrelated to credentials; ENROLL_KEY above is valid and correctly
+    # VPS over Netbird instead - confirmed working end-to-end now that real
-    #    wired, but can't get past this from home.
+    # pod-egress routing to the VPS exists (netbird-egress DaemonSet, see
-    # 2) Tried routing around it via a proxy on the VPS over Netbird -
+    # infrastructure/netbird/manifests/egress-daemonset.yaml). This was
-    #    confirmed no node in this cluster has any route into the Netbird
+    # tried once before and failed because that egress route didn't exist
-    #    mesh CIDR for pod-originated (egress) traffic at all. The
+    # yet - not a proxy or credentials problem.
-    #    per-namespace netbird "router" pods (gitlab, vault, argocd, etc.)
+    - name: HTTPS_PROXY
-    #    are inbound-only infrastructure (let external Netbird peers reach
+      value: "http://100.108.113.41:8888"
-    #    K8s services) - nothing provides the reverse path. Building that
+    - name: HTTP_PROXY
-    #    properly needs either a Talos-native static route (fragile - would
+      value: "http://100.108.113.41:8888"
-    #    point at a pod IP that changes on every restart) or a deliberate
+    - name: NO_PROXY
-    #    exception to the disallow-host-namespaces STIG policy that
+      value: "localhost,127.0.0.1,.svc,.svc.cluster.local,10.244.0.0/16,10.96.0.0/16"
-    #    correctly blocks the hostNetwork-based approach. Out of scope for
-    #    a same-night fix - tracked as a separate future task.
-    # LAPI treats a failed CAPI/enroll attempt as fatal at startup (crashes
-    # the whole server, taking the agent + bouncer down with it via their
-    # dependency on LAPI), so disable it entirely rather than crash-loop.
-    # ENROLL_KEY above stays set - flip this back to false (or remove it)
-    # once real pod-egress routing to the VPS exists, no other change needed.
-    - name: DISABLE_ONLINE_API
-      value: "true"
   persistentVolume:
     data:
       enabled: false

infrastructure/kyverno/policies/netbird-egress-exception.yaml

@@ -0,0 +1,38 @@
+# Scoped exception to the disallow-host-namespaces STIG policy (V-242386),
+# narrowly for the netbird-egress DaemonSet only - not a namespace-wide
+# exclusion like crowdsec's (see disallow-host-namespaces ClusterPolicy).
+#
+# Why hostNetwork is genuinely needed here: no node in this cluster has any
+# route into the Netbird mesh CIDR (100.108.0.0/16) for pod-originated
+# egress traffic - confirmed 2026-08-17 while debugging CrowdSec/CNPG
+# backups reaching the VPS. The existing per-namespace "router" pods
+# (NBRoutingPeer-based) are inbound-only and run fine in ordinary pod
+# network namespaces. An EGRESS gateway is architecturally different: its
+# wt0 interface must live in each node's real network namespace so the
+# node's own kernel routing table can send matched pod traffic out through
+# it directly - a pod-netns router would need SNAT/forwarding indirection
+# on top, and per-pod route injection isn't possible without hostPID
+# (also STIG-blocked) or a Cilium egress-gateway feature that doesn't
+# actually solve this (it selects source IP for already-routable traffic,
+# it doesn't create the missing route).
+apiVersion: kyverno.io/v2
+kind: PolicyException
+metadata:
+  name: netbird-egress-exemption
+  namespace: kyverno
+spec:
+  exceptions:
+    - policyName: disallow-host-namespaces
+      ruleNames:
+        - host-namespaces
+        - autogen-host-namespaces
+  match:
+    any:
+      - resources:
+          kinds:
+            - DaemonSet
+            - Pod
+          namespaces:
+            - netbird
+          names:
+            - netbird-egress*

infrastructure/netbird/manifests/egress-daemonset.yaml

@@ -0,0 +1,96 @@
+# Pod-egress routing into the Netbird mesh.
+#
+# The existing per-namespace `router` Deployments (NBRoutingPeer CRs -
+# gitlab, vault, argocd, nextcloud, etc.) are INBOUND-only infrastructure:
+# they let external Netbird peers reach into those namespaces' services.
+# Nothing programs the reverse - no node ever gets a route sending pod
+# traffic OUT through any of those router pods' wt0 interfaces. Confirmed
+# 2026-08-17 while debugging CrowdSec's CAPI enrollment: `ip route get
+# 100.108.113.41` on a node running a pod that needed to reach the VPS's
+# Netbird IP just showed the plain LAN default gateway - none of the
+# "connected" router pods were ever actually in the path, and a tcpdump on
+# their wt0 during live attempts showed zero packets.
+#
+# This DaemonSet is the missing outbound half: one netbird client per node,
+# running with hostNetwork so its wt0 interface lives directly in the node's
+# real network namespace (avoiding all the SNAT/forwarding complexity a
+# pod-netns subnet router would need), plus a sidecar that adds a host route
+# sending 100.108.0.0/16 out via that interface. Once packets leave a node
+# via wt0 with a real Netbird-mesh source identity, return routing already
+# works via the same route-advertisement mechanism the inbound routers use.
+#
+# hostNetwork requires infrastructure/kyverno/policies/netbird-egress-exception.yaml
+# - a scoped PolicyException to the disallow-host-namespaces STIG policy,
+# not a broad exclusion. See that file for the full justification.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: netbird-egress
+  namespace: netbird
+  labels:
+    app.kubernetes.io/name: netbird-egress
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: netbird-egress
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: netbird-egress
+      annotations:
+        # Same reasoning as netbird-cluster-router: ztunnel iptables rules
+        # interfere with WireGuard packet forwarding.
+        ambient.istio.io/redirection: disabled
+    spec:
+      hostNetwork: true
+      dnsPolicy: ClusterFirstWithHostNet
+      containers:
+        - name: netbird
+          image: netbirdio/netbird:0.66.0
+          imagePullPolicy: IfNotPresent
+          env:
+            - name: NB_SETUP_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: netbird-router-setup-key
+                  key: setup-key
+            - name: NB_MANAGEMENT_URL
+              value: "https://netbird.kube.huskypup.net"
+          securityContext:
+            capabilities:
+              add:
+                - NET_ADMIN
+          resources:
+            requests:
+              cpu: 25m
+              memory: 64Mi
+            limits:
+              memory: 128Mi
+        # Adds the host route sending mesh-bound traffic out via wt0 once
+        # the netbird container brings the interface up. Re-asserts on a
+        # loop since wt0 can be recreated on reconnect (route otherwise
+        # silently disappears with it).
+        - name: route-manager
+          image: alpine:3.20
+          command:
+            - sh
+            - -c
+            - |
+              apk add --no-cache iproute2 >/dev/null 2>&1
+              while true; do
+                if ip link show wt0 >/dev/null 2>&1; then
+                  ip route replace 100.108.0.0/16 dev wt0 2>/dev/null
+                fi
+                sleep 10
+              done
+          securityContext:
+            capabilities:
+              add:
+                - NET_ADMIN
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 64Mi
+      terminationGracePeriodSeconds: 30

infrastructure/vault/manifests/raft-snapshot-cronjob.yaml

@@ -45,7 +45,14 @@ data:
 
     # VPS's Netbird address - replace with the actual peer IP/hostname once
     # the VPS is bootstrapped and joined to the mesh (scripts/vps-bootstrap.sh).
-    VPS_MINIO_ENDPOINT="${VPS_MINIO_ENDPOINT:-vps-minio.netbird.internal:30900}"
+    # 100.108.113.41:9000 is a locally-terminated socat forward on the VPS
+    # host (minio-forward.service) to MinIO's ClusterIP, not the NodePort.
+    # vps-minio.netbird.internal was never resolvable (Netbird has no DNS
+    # configured) and the NodePort itself is unreachable from other peers -
+    # Netbird's own ACL model only permits established forwarded
+    # connections through a peer, never new ones. See egress-daemonset.yaml
+    # for the pod-egress route this now travels over.
+    VPS_MINIO_ENDPOINT="${VPS_MINIO_ENDPOINT:-100.108.113.41:9000}"
     BUCKET="vault-raft-snapshots"
     SNAP_NAME="vault-raft-$(date -u +%Y%m%dT%H%M%SZ).snap"
 
@@ -102,13 +109,9 @@ metadata:
   namespace: vault
 spec:
   schedule: "0 */6 * * *"  # every 6 hours
-  # Suspended 2026-08-17: the upload step targets vps-minio.netbird.internal,
+  # Re-enabled 2026-08-18: real pod-egress routing to the VPS now exists
-  # which no pod in this cluster can currently reach - no node has an
+  # (netbird-egress DaemonSet) and the endpoint above was fixed to use the
-  # egress route into the Netbird mesh (see commit history around this
+  # locally-terminated socat forward instead of the unreachable NodePort.
-  # date for the full investigation). The leader-detection fix above is
-  # real and independent of this, so it's kept fixed rather than reverted.
-  # Unsuspend once real pod-egress routing to the VPS exists.
-  suspend: true
   concurrencyPolicy: Forbid
   successfulJobsHistoryLimit: 3
   failedJobsHistoryLimit: 3