mirror of
https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
synced 2026-08-20 23:16:49 +00:00
Scooby HuskyandClaude Sonnet 5
018eb3570f
Root cause of tonight's earlier CrowdSec/CNPG-backup workarounds: no node in the cluster had any route into the Netbird mesh CIDR (100.108.0.0/16) for pod-originated traffic. The per-namespace netbird 'router' pods are inbound-only infrastructure (external peers reaching K8s services); their own architecture has no reverse path. Fix, in two parts: 1. infrastructure/netbird/manifests/egress-daemonset.yaml - one netbird client per node, hostNetwork so its wt0 interface lives in the node's real network namespace, plus a sidecar that adds a host route sending 100.108.0.0/16 out via it. hostNetwork requires a scoped Kyverno PolicyException (infrastructure/kyverno/policies/netbird-egress-exception.yaml) to the disallow-host-namespaces STIG policy - narrowly for this one DaemonSet by name, not a namespace-wide exclusion. 2. Discovered the route alone wasn't enough for k3s NodePort traffic (vps-minio:30900): Netbird manages its own nftables ACLs independent of iptables/Kyverno, and its forward chain (netbird-rt-fwd) only permits *established* connections through a peer acting as a router - never new ones, by design, unless a Netbird 'Network Route' policy is explicitly configured (it isn't, for this VPS). Locally-terminated connections (tinyproxy) go through a separate, already-permissive ACL chain, which is why the CrowdSec proxy fix from earlier tonight worked. Replicated that working pattern for MinIO: minio-forward.service on the VPS host (systemd, socat) forwards 100.108.113.41:9000 -> MinIO's ClusterIP, avoiding the NodePort path entirely. Re-enabled everything that was disabled/suspended earlier tonight because of this gap, pointed at the new endpoint: - CrowdSec CAPI/console-enroll (removed DISABLE_ONLINE_API, restored the VPS proxy env vars) - n8n/nextcloud/authentik CNPG backup.barmanObjectStore - vault-raft-snapshot CronJob (unsuspended) - nextcloud PVC content sync CronJob endpoint vps-minio.netbird.internal is retired everywhere - it was never actually resolvable (Netbird has no DNS configured) even before today's routing fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
97 lines
3.6 KiB
YAML
97 lines
3.6 KiB
YAML
# Pod-egress routing into the Netbird mesh.
|
|
#
|
|
# The existing per-namespace `router` Deployments (NBRoutingPeer CRs -
|
|
# gitlab, vault, argocd, nextcloud, etc.) are INBOUND-only infrastructure:
|
|
# they let external Netbird peers reach into those namespaces' services.
|
|
# Nothing programs the reverse - no node ever gets a route sending pod
|
|
# traffic OUT through any of those router pods' wt0 interfaces. Confirmed
|
|
# 2026-08-17 while debugging CrowdSec's CAPI enrollment: `ip route get
|
|
# 100.108.113.41` on a node running a pod that needed to reach the VPS's
|
|
# Netbird IP just showed the plain LAN default gateway - none of the
|
|
# "connected" router pods were ever actually in the path, and a tcpdump on
|
|
# their wt0 during live attempts showed zero packets.
|
|
#
|
|
# This DaemonSet is the missing outbound half: one netbird client per node,
|
|
# running with hostNetwork so its wt0 interface lives directly in the node's
|
|
# real network namespace (avoiding all the SNAT/forwarding complexity a
|
|
# pod-netns subnet router would need), plus a sidecar that adds a host route
|
|
# sending 100.108.0.0/16 out via that interface. Once packets leave a node
|
|
# via wt0 with a real Netbird-mesh source identity, return routing already
|
|
# works via the same route-advertisement mechanism the inbound routers use.
|
|
#
|
|
# hostNetwork requires infrastructure/kyverno/policies/netbird-egress-exception.yaml
|
|
# - a scoped PolicyException to the disallow-host-namespaces STIG policy,
|
|
# not a broad exclusion. See that file for the full justification.
|
|
apiVersion: apps/v1
|
|
kind: DaemonSet
|
|
metadata:
|
|
name: netbird-egress
|
|
namespace: netbird
|
|
labels:
|
|
app.kubernetes.io/name: netbird-egress
|
|
spec:
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: netbird-egress
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: netbird-egress
|
|
annotations:
|
|
# Same reasoning as netbird-cluster-router: ztunnel iptables rules
|
|
# interfere with WireGuard packet forwarding.
|
|
ambient.istio.io/redirection: disabled
|
|
spec:
|
|
hostNetwork: true
|
|
dnsPolicy: ClusterFirstWithHostNet
|
|
containers:
|
|
- name: netbird
|
|
image: netbirdio/netbird:0.66.0
|
|
imagePullPolicy: IfNotPresent
|
|
env:
|
|
- name: NB_SETUP_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: netbird-router-setup-key
|
|
key: setup-key
|
|
- name: NB_MANAGEMENT_URL
|
|
value: "https://netbird.kube.huskypup.net"
|
|
securityContext:
|
|
capabilities:
|
|
add:
|
|
- NET_ADMIN
|
|
resources:
|
|
requests:
|
|
cpu: 25m
|
|
memory: 64Mi
|
|
limits:
|
|
memory: 128Mi
|
|
# Adds the host route sending mesh-bound traffic out via wt0 once
|
|
# the netbird container brings the interface up. Re-asserts on a
|
|
# loop since wt0 can be recreated on reconnect (route otherwise
|
|
# silently disappears with it).
|
|
- name: route-manager
|
|
image: alpine:3.20
|
|
command:
|
|
- sh
|
|
- -c
|
|
- |
|
|
apk add --no-cache iproute2 >/dev/null 2>&1
|
|
while true; do
|
|
if ip link show wt0 >/dev/null 2>&1; then
|
|
ip route replace 100.108.0.0/16 dev wt0 2>/dev/null
|
|
fi
|
|
sleep 10
|
|
done
|
|
securityContext:
|
|
capabilities:
|
|
add:
|
|
- NET_ADMIN
|
|
resources:
|
|
requests:
|
|
cpu: 10m
|
|
memory: 32Mi
|
|
limits:
|
|
memory: 64Mi
|
|
terminationGracePeriodSeconds: 30
|