mirror of
https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
synced 2026-08-21 05:26:49 +00:00
Build real pod-egress routing to the VPS, fix everything blocked by its absence
Root cause of tonight's earlier CrowdSec/CNPG-backup workarounds: no node in the cluster had any route into the Netbird mesh CIDR (100.108.0.0/16) for pod-originated traffic. The per-namespace netbird 'router' pods are inbound-only infrastructure (external peers reaching K8s services); their own architecture has no reverse path. Fix, in two parts: 1. infrastructure/netbird/manifests/egress-daemonset.yaml - one netbird client per node, hostNetwork so its wt0 interface lives in the node's real network namespace, plus a sidecar that adds a host route sending 100.108.0.0/16 out via it. hostNetwork requires a scoped Kyverno PolicyException (infrastructure/kyverno/policies/netbird-egress-exception.yaml) to the disallow-host-namespaces STIG policy - narrowly for this one DaemonSet by name, not a namespace-wide exclusion. 2. Discovered the route alone wasn't enough for k3s NodePort traffic (vps-minio:30900): Netbird manages its own nftables ACLs independent of iptables/Kyverno, and its forward chain (netbird-rt-fwd) only permits *established* connections through a peer acting as a router - never new ones, by design, unless a Netbird 'Network Route' policy is explicitly configured (it isn't, for this VPS). Locally-terminated connections (tinyproxy) go through a separate, already-permissive ACL chain, which is why the CrowdSec proxy fix from earlier tonight worked. Replicated that working pattern for MinIO: minio-forward.service on the VPS host (systemd, socat) forwards 100.108.113.41:9000 -> MinIO's ClusterIP, avoiding the NodePort path entirely. Re-enabled everything that was disabled/suspended earlier tonight because of this gap, pointed at the new endpoint: - CrowdSec CAPI/console-enroll (removed DISABLE_ONLINE_API, restored the VPS proxy env vars) - n8n/nextcloud/authentik CNPG backup.barmanObjectStore - vault-raft-snapshot CronJob (unsuspended) - nextcloud PVC content sync CronJob endpoint vps-minio.netbird.internal is retired everywhere - it was never actually resolvable (Netbird has no DNS configured) even before today's routing fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
07a59d3b34
commit
018eb3570f
@@ -34,29 +34,20 @@ lapi:
|
||||
name: crowdsec-bouncer-key
|
||||
key: api-key
|
||||
optional: true
|
||||
# CAPI/console-enroll disabled for now (2026-08-17). Root causes found:
|
||||
# 1) Home's public IP gets a 403 from AWS WAF in front of api.crowdsec.net
|
||||
# (even on a bare unauthenticated request) - an IP-level block
|
||||
# unrelated to credentials; ENROLL_KEY above is valid and correctly
|
||||
# wired, but can't get past this from home.
|
||||
# 2) Tried routing around it via a proxy on the VPS over Netbird -
|
||||
# confirmed no node in this cluster has any route into the Netbird
|
||||
# mesh CIDR for pod-originated (egress) traffic at all. The
|
||||
# per-namespace netbird "router" pods (gitlab, vault, argocd, etc.)
|
||||
# are inbound-only infrastructure (let external Netbird peers reach
|
||||
# K8s services) - nothing provides the reverse path. Building that
|
||||
# properly needs either a Talos-native static route (fragile - would
|
||||
# point at a pod IP that changes on every restart) or a deliberate
|
||||
# exception to the disallow-host-namespaces STIG policy that
|
||||
# correctly blocks the hostNetwork-based approach. Out of scope for
|
||||
# a same-night fix - tracked as a separate future task.
|
||||
# LAPI treats a failed CAPI/enroll attempt as fatal at startup (crashes
|
||||
# the whole server, taking the agent + bouncer down with it via their
|
||||
# dependency on LAPI), so disable it entirely rather than crash-loop.
|
||||
# ENROLL_KEY above stays set - flip this back to false (or remove it)
|
||||
# once real pod-egress routing to the VPS exists, no other change needed.
|
||||
- name: DISABLE_ONLINE_API
|
||||
value: "true"
|
||||
# CAPI/console-enroll RE-ENABLED 2026-08-18. Home's public IP still gets
|
||||
# a 403 from AWS WAF in front of api.crowdsec.net (an IP-level block,
|
||||
# unrelated to credentials), so this routes through a tinyproxy on the
|
||||
# VPS over Netbird instead - confirmed working end-to-end now that real
|
||||
# pod-egress routing to the VPS exists (netbird-egress DaemonSet, see
|
||||
# infrastructure/netbird/manifests/egress-daemonset.yaml). This was
|
||||
# tried once before and failed because that egress route didn't exist
|
||||
# yet - not a proxy or credentials problem.
|
||||
- name: HTTPS_PROXY
|
||||
value: "http://100.108.113.41:8888"
|
||||
- name: HTTP_PROXY
|
||||
value: "http://100.108.113.41:8888"
|
||||
- name: NO_PROXY
|
||||
value: "localhost,127.0.0.1,.svc,.svc.cluster.local,10.244.0.0/16,10.96.0.0/16"
|
||||
persistentVolume:
|
||||
data:
|
||||
enabled: false
|
||||
|
||||
Reference in New Issue
Block a user