Disable CrowdSec online API - CAPI unreachable from home, egress routing gap found

LAPI treated a failed CAPI/console-enroll attempt as fatal at startup,
crashing the whole server (and taking agent + bouncer down with it via
their LAPI dependency) on every restart. Root cause investigation tonight
found two separate real issues:

1. Home's public IP gets a 403 from AWS WAF in front of api.crowdsec.net
   on any request, unrelated to credentials - ENROLL_KEY is valid and
   correctly wired but can't get past this.
2. Attempted routing around it via a VPS-side proxy over Netbird; found no
   node in the cluster has any route into the Netbird mesh CIDR for
   pod-originated egress traffic - the per-namespace netbird router pods
   (gitlab, vault, argocd, etc.) are inbound-only infrastructure. A proper
   fix needs either a Talos-native static route (fragile against pod IP
   churn) or a deliberate STIG exception for hostNetwork - out of scope
   for tonight, tracked separately.

DISABLE_ONLINE_API=true lets LAPI start cleanly with local-only detection
(agent + bouncer functional again). ENROLL_KEY stays set in Vault/values -
flip this back once real egress routing exists.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Scooby Husky
2026-08-17 19:37:33 -05:00
co-authored by Claude Sonnet 5
parent c8643ca025
commit ff638250d6
+23 -13
View File
@@ -34,19 +34,29 @@ lapi:
name: crowdsec-bouncer-key
key: api-key
optional: true
# Route CAPI/console-enroll HTTPS calls (api.crowdsec.net) through a
# tinyproxy on the VPS, reachable over Netbird. Confirmed 2026-08-17:
# home's public IP gets a 403 from AWS WAF in front of CrowdSec's API
# (even on a bare unauthenticated request), while the VPS's IP gets a
# normal 401 - this is an IP-level block unrelated to credentials, so a
# fresh enrollment key alone doesn't help without changing egress IP.
# cscli's Go HTTP client honors these standard proxy env vars.
- name: HTTPS_PROXY
value: "http://100.108.113.41:8888"
- name: HTTP_PROXY
value: "http://100.108.113.41:8888"
- name: NO_PROXY
value: "localhost,127.0.0.1,.svc,.svc.cluster.local,10.244.0.0/16,10.96.0.0/16"
# CAPI/console-enroll disabled for now (2026-08-17). Root causes found:
# 1) Home's public IP gets a 403 from AWS WAF in front of api.crowdsec.net
# (even on a bare unauthenticated request) - an IP-level block
# unrelated to credentials; ENROLL_KEY above is valid and correctly
# wired, but can't get past this from home.
# 2) Tried routing around it via a proxy on the VPS over Netbird -
# confirmed no node in this cluster has any route into the Netbird
# mesh CIDR for pod-originated (egress) traffic at all. The
# per-namespace netbird "router" pods (gitlab, vault, argocd, etc.)
# are inbound-only infrastructure (let external Netbird peers reach
# K8s services) - nothing provides the reverse path. Building that
# properly needs either a Talos-native static route (fragile - would
# point at a pod IP that changes on every restart) or a deliberate
# exception to the disallow-host-namespaces STIG policy that
# correctly blocks the hostNetwork-based approach. Out of scope for
# a same-night fix - tracked as a separate future task.
# LAPI treats a failed CAPI/enroll attempt as fatal at startup (crashes
# the whole server, taking the agent + bouncer down with it via their
# dependency on LAPI), so disable it entirely rather than crash-loop.
# ENROLL_KEY above stays set - flip this back to false (or remove it)
# once real pod-egress routing to the VPS exists, no other change needed.
- name: DISABLE_ONLINE_API
value: "true"
persistentVolume:
data:
enabled: false