mirror of
https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
synced 2026-08-21 05:26:49 +00:00
Build real pod-egress routing to the VPS, fix everything blocked by its absence
Root cause of tonight's earlier CrowdSec/CNPG-backup workarounds: no node in the cluster had any route into the Netbird mesh CIDR (100.108.0.0/16) for pod-originated traffic. The per-namespace netbird 'router' pods are inbound-only infrastructure (external peers reaching K8s services); their own architecture has no reverse path. Fix, in two parts: 1. infrastructure/netbird/manifests/egress-daemonset.yaml - one netbird client per node, hostNetwork so its wt0 interface lives in the node's real network namespace, plus a sidecar that adds a host route sending 100.108.0.0/16 out via it. hostNetwork requires a scoped Kyverno PolicyException (infrastructure/kyverno/policies/netbird-egress-exception.yaml) to the disallow-host-namespaces STIG policy - narrowly for this one DaemonSet by name, not a namespace-wide exclusion. 2. Discovered the route alone wasn't enough for k3s NodePort traffic (vps-minio:30900): Netbird manages its own nftables ACLs independent of iptables/Kyverno, and its forward chain (netbird-rt-fwd) only permits *established* connections through a peer acting as a router - never new ones, by design, unless a Netbird 'Network Route' policy is explicitly configured (it isn't, for this VPS). Locally-terminated connections (tinyproxy) go through a separate, already-permissive ACL chain, which is why the CrowdSec proxy fix from earlier tonight worked. Replicated that working pattern for MinIO: minio-forward.service on the VPS host (systemd, socat) forwards 100.108.113.41:9000 -> MinIO's ClusterIP, avoiding the NodePort path entirely. Re-enabled everything that was disabled/suspended earlier tonight because of this gap, pointed at the new endpoint: - CrowdSec CAPI/console-enroll (removed DISABLE_ONLINE_API, restored the VPS proxy env vars) - n8n/nextcloud/authentik CNPG backup.barmanObjectStore - vault-raft-snapshot CronJob (unsuspended) - nextcloud PVC content sync CronJob endpoint vps-minio.netbird.internal is retired everywhere - it was never actually resolvable (Netbird has no DNS configured) even before today's routing fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
07a59d3b34
commit
018eb3570f
@@ -45,7 +45,14 @@ data:
|
||||
|
||||
# VPS's Netbird address - replace with the actual peer IP/hostname once
|
||||
# the VPS is bootstrapped and joined to the mesh (scripts/vps-bootstrap.sh).
|
||||
VPS_MINIO_ENDPOINT="${VPS_MINIO_ENDPOINT:-vps-minio.netbird.internal:30900}"
|
||||
# 100.108.113.41:9000 is a locally-terminated socat forward on the VPS
|
||||
# host (minio-forward.service) to MinIO's ClusterIP, not the NodePort.
|
||||
# vps-minio.netbird.internal was never resolvable (Netbird has no DNS
|
||||
# configured) and the NodePort itself is unreachable from other peers -
|
||||
# Netbird's own ACL model only permits established forwarded
|
||||
# connections through a peer, never new ones. See egress-daemonset.yaml
|
||||
# for the pod-egress route this now travels over.
|
||||
VPS_MINIO_ENDPOINT="${VPS_MINIO_ENDPOINT:-100.108.113.41:9000}"
|
||||
BUCKET="vault-raft-snapshots"
|
||||
SNAP_NAME="vault-raft-$(date -u +%Y%m%dT%H%M%SZ).snap"
|
||||
|
||||
@@ -102,13 +109,9 @@ metadata:
|
||||
namespace: vault
|
||||
spec:
|
||||
schedule: "0 */6 * * *" # every 6 hours
|
||||
# Suspended 2026-08-17: the upload step targets vps-minio.netbird.internal,
|
||||
# which no pod in this cluster can currently reach - no node has an
|
||||
# egress route into the Netbird mesh (see commit history around this
|
||||
# date for the full investigation). The leader-detection fix above is
|
||||
# real and independent of this, so it's kept fixed rather than reverted.
|
||||
# Unsuspend once real pod-egress routing to the VPS exists.
|
||||
suspend: true
|
||||
# Re-enabled 2026-08-18: real pod-egress routing to the VPS now exists
|
||||
# (netbird-egress DaemonSet) and the endpoint above was fixed to use the
|
||||
# locally-terminated socat forward instead of the unreachable NodePort.
|
||||
concurrencyPolicy: Forbid
|
||||
successfulJobsHistoryLimit: 3
|
||||
failedJobsHistoryLimit: 3
|
||||
|
||||
Reference in New Issue
Block a user