Build real pod-egress routing to the VPS, fix everything blocked by its absence

Root cause of tonight's earlier CrowdSec/CNPG-backup workarounds: no node
in the cluster had any route into the Netbird mesh CIDR (100.108.0.0/16)
for pod-originated traffic. The per-namespace netbird 'router' pods are
inbound-only infrastructure (external peers reaching K8s services); their
own architecture has no reverse path.

Fix, in two parts:

1. infrastructure/netbird/manifests/egress-daemonset.yaml - one netbird
   client per node, hostNetwork so its wt0 interface lives in the node's
   real network namespace, plus a sidecar that adds a host route sending
   100.108.0.0/16 out via it. hostNetwork requires a scoped Kyverno
   PolicyException (infrastructure/kyverno/policies/netbird-egress-exception.yaml)
   to the disallow-host-namespaces STIG policy - narrowly for this one
   DaemonSet by name, not a namespace-wide exclusion.

2. Discovered the route alone wasn't enough for k3s NodePort traffic
   (vps-minio:30900): Netbird manages its own nftables ACLs independent of
   iptables/Kyverno, and its forward chain (netbird-rt-fwd) only permits
   *established* connections through a peer acting as a router - never new
   ones, by design, unless a Netbird 'Network Route' policy is explicitly
   configured (it isn't, for this VPS). Locally-terminated connections
   (tinyproxy) go through a separate, already-permissive ACL chain, which
   is why the CrowdSec proxy fix from earlier tonight worked. Replicated
   that working pattern for MinIO: minio-forward.service on the VPS host
   (systemd, socat) forwards 100.108.113.41:9000 -> MinIO's ClusterIP,
   avoiding the NodePort path entirely.

Re-enabled everything that was disabled/suspended earlier tonight because
of this gap, pointed at the new endpoint:
- CrowdSec CAPI/console-enroll (removed DISABLE_ONLINE_API, restored the
  VPS proxy env vars)
- n8n/nextcloud/authentik CNPG backup.barmanObjectStore
- vault-raft-snapshot CronJob (unsuspended)
- nextcloud PVC content sync CronJob endpoint

vps-minio.netbird.internal is retired everywhere - it was never actually
resolvable (Netbird has no DNS configured) even before today's routing
fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Scooby Husky
2026-08-17 20:36:13 -05:00
co-authored by Claude Sonnet 5
parent 07a59d3b34
commit 018eb3570f
8 changed files with 241 additions and 103 deletions
+14 -23
View File
@@ -34,29 +34,20 @@ lapi:
name: crowdsec-bouncer-key
key: api-key
optional: true
# CAPI/console-enroll disabled for now (2026-08-17). Root causes found:
# 1) Home's public IP gets a 403 from AWS WAF in front of api.crowdsec.net
# (even on a bare unauthenticated request) - an IP-level block
# unrelated to credentials; ENROLL_KEY above is valid and correctly
# wired, but can't get past this from home.
# 2) Tried routing around it via a proxy on the VPS over Netbird -
# confirmed no node in this cluster has any route into the Netbird
# mesh CIDR for pod-originated (egress) traffic at all. The
# per-namespace netbird "router" pods (gitlab, vault, argocd, etc.)
# are inbound-only infrastructure (let external Netbird peers reach
# K8s services) - nothing provides the reverse path. Building that
# properly needs either a Talos-native static route (fragile - would
# point at a pod IP that changes on every restart) or a deliberate
# exception to the disallow-host-namespaces STIG policy that
# correctly blocks the hostNetwork-based approach. Out of scope for
# a same-night fix - tracked as a separate future task.
# LAPI treats a failed CAPI/enroll attempt as fatal at startup (crashes
# the whole server, taking the agent + bouncer down with it via their
# dependency on LAPI), so disable it entirely rather than crash-loop.
# ENROLL_KEY above stays set - flip this back to false (or remove it)
# once real pod-egress routing to the VPS exists, no other change needed.
- name: DISABLE_ONLINE_API
value: "true"
# CAPI/console-enroll RE-ENABLED 2026-08-18. Home's public IP still gets
# a 403 from AWS WAF in front of api.crowdsec.net (an IP-level block,
# unrelated to credentials), so this routes through a tinyproxy on the
# VPS over Netbird instead - confirmed working end-to-end now that real
# pod-egress routing to the VPS exists (netbird-egress DaemonSet, see
# infrastructure/netbird/manifests/egress-daemonset.yaml). This was
# tried once before and failed because that egress route didn't exist
# yet - not a proxy or credentials problem.
- name: HTTPS_PROXY
value: "http://100.108.113.41:8888"
- name: HTTP_PROXY
value: "http://100.108.113.41:8888"
- name: NO_PROXY
value: "localhost,127.0.0.1,.svc,.svc.cluster.local,10.244.0.0/16,10.96.0.0/16"
persistentVolume:
data:
enabled: false