Build real pod-egress routing to the VPS, fix everything blocked by its absence

Root cause of tonight's earlier CrowdSec/CNPG-backup workarounds: no node
in the cluster had any route into the Netbird mesh CIDR (100.108.0.0/16)
for pod-originated traffic. The per-namespace netbird 'router' pods are
inbound-only infrastructure (external peers reaching K8s services); their
own architecture has no reverse path.

Fix, in two parts:

1. infrastructure/netbird/manifests/egress-daemonset.yaml - one netbird
   client per node, hostNetwork so its wt0 interface lives in the node's
   real network namespace, plus a sidecar that adds a host route sending
   100.108.0.0/16 out via it. hostNetwork requires a scoped Kyverno
   PolicyException (infrastructure/kyverno/policies/netbird-egress-exception.yaml)
   to the disallow-host-namespaces STIG policy - narrowly for this one
   DaemonSet by name, not a namespace-wide exclusion.

2. Discovered the route alone wasn't enough for k3s NodePort traffic
   (vps-minio:30900): Netbird manages its own nftables ACLs independent of
   iptables/Kyverno, and its forward chain (netbird-rt-fwd) only permits
   *established* connections through a peer acting as a router - never new
   ones, by design, unless a Netbird 'Network Route' policy is explicitly
   configured (it isn't, for this VPS). Locally-terminated connections
   (tinyproxy) go through a separate, already-permissive ACL chain, which
   is why the CrowdSec proxy fix from earlier tonight worked. Replicated
   that working pattern for MinIO: minio-forward.service on the VPS host
   (systemd, socat) forwards 100.108.113.41:9000 -> MinIO's ClusterIP,
   avoiding the NodePort path entirely.

Re-enabled everything that was disabled/suspended earlier tonight because
of this gap, pointed at the new endpoint:
- CrowdSec CAPI/console-enroll (removed DISABLE_ONLINE_API, restored the
  VPS proxy env vars)
- n8n/nextcloud/authentik CNPG backup.barmanObjectStore
- vault-raft-snapshot CronJob (unsuspended)
- nextcloud PVC content sync CronJob endpoint

vps-minio.netbird.internal is retired everywhere - it was never actually
resolvable (Netbird has no DNS configured) even before today's routing
fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Scooby Husky
2026-08-17 20:36:13 -05:00
co-authored by Claude Sonnet 5
parent 07a59d3b34
commit 018eb3570f
8 changed files with 241 additions and 103 deletions
+27 -24
View File
@@ -44,30 +44,33 @@ spec:
database: nextcloud
owner: nextcloud
# Backup to VPS MinIO - DISABLED 2026-08-17. No node in the cluster has
# any route into the Netbird mesh for pod-originated (egress) traffic;
# the per-namespace netbird "router" pods are inbound-only infrastructure.
# continuousArchiving kept failing to connect to vps-minio.netbird.internal,
# which held Ready=False permanently (real condition, not cosmetic).
# Re-enable once real pod-egress routing to the VPS exists (tracked as a
# separate task) - no other change needed, this block is otherwise
# complete/correct. This covers the DB only - file PVC content is
# separate, see nextcloud-pvc-sync-cronjob.yaml (same underlying gap).
# backup:
# barmanObjectStore:
# destinationPath: s3://cnpg-backups/pg-nextcloud
# endpointURL: http://vps-minio.netbird.internal:30900
# s3Credentials:
# accessKeyId:
# name: vps-minio-secret
# key: accesskey
# secretAccessKey:
# name: vps-minio-secret
# key: secretkey
# wal:
# compression: gzip
# maxParallel: 2
# retentionPolicy: "30d"
# Backup to VPS MinIO - RE-ENABLED 2026-08-18. Real pod-egress routing to
# the VPS now exists (netbird-egress DaemonSet, hostNetwork + per-node
# route into 100.108.0.0/16 - see infrastructure/netbird/manifests/
# egress-daemonset.yaml). Endpoint changed from the NodePort
# (vps-minio.netbird.internal:30900 - never resolvable anyway, Netbird has
# no DNS configured, and separately blocked by Netbird's own ACL model,
# which only permits *established* forwarded connections through a peer,
# never new ones) to a locally-terminated socat forward on the VPS host
# itself (100.108.113.41:9000 - see minio-forward.service on the VPS),
# mirroring the pattern that already worked for the CrowdSec CAPI proxy.
# This covers the DB only - file PVC content is separate, see
# nextcloud-pvc-sync-cronjob.yaml (same fix applies there too).
backup:
barmanObjectStore:
destinationPath: s3://cnpg-backups/pg-nextcloud
endpointURL: http://100.108.113.41:9000
s3Credentials:
accessKeyId:
name: vps-minio-secret
key: accesskey
secretAccessKey:
name: vps-minio-secret
key: secretkey
wal:
compression: gzip
maxParallel: 2
retentionPolicy: "30d"
monitoring:
enablePodMonitor: true
@@ -100,10 +100,13 @@ spec:
echo "==> Done."
env:
# VPS's Netbird address - replace once bootstrapped, matches
# Locally-terminated socat forward on the VPS host to
# MinIO's ClusterIP, not the NodePort - see
# infrastructure/vault/manifests/raft-snapshot-cronjob.yaml
# for why (Netbird has no DNS, and blocks new forwarded
# connections through a peer by default).
- name: VPS_MINIO_ENDPOINT
value: "vps-minio.netbird.internal:30900"
value: "100.108.113.41:9000"
- name: MINIO_ACCESS_KEY
valueFrom:
secretKeyRef: