LAPI treated a failed CAPI/console-enroll attempt as fatal at startup,
crashing the whole server (and taking agent + bouncer down with it via
their LAPI dependency) on every restart. Root cause investigation tonight
found two separate real issues:
1. Home's public IP gets a 403 from AWS WAF in front of api.crowdsec.net
on any request, unrelated to credentials - ENROLL_KEY is valid and
correctly wired but can't get past this.
2. Attempted routing around it via a VPS-side proxy over Netbird; found no
node in the cluster has any route into the Netbird mesh CIDR for
pod-originated egress traffic - the per-namespace netbird router pods
(gitlab, vault, argocd, etc.) are inbound-only infrastructure. A proper
fix needs either a Talos-native static route (fragile against pod IP
churn) or a deliberate STIG exception for hostNetwork - out of scope
for tonight, tracked separately.
DISABLE_ONLINE_API=true lets LAPI start cleanly with local-only detection
(agent + bouncer functional again). ENROLL_KEY stays set in Vault/values -
flip this back once real egress routing exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed 2026-08-17: home's public IP gets a 403 ForbiddenException from
AWS WAF in front of api.crowdsec.net, even on a bare unauthenticated test
request - the VPS's IP gets a normal 401 on the identical request. This is
an IP-level block unrelated to credentials, which is why LAPI kept hitting
403 on both anonymous auto-registration and console enroll with a fresh,
valid enrollment key.
Fix: HTTPS_PROXY/HTTP_PROXY pointed at a tinyproxy instance on the VPS
(bound to its Netbird interface only, not internet-reachable), reachable
from home over the existing Netbird mesh via netbird-cluster-router.
cscli/crowdsec's Go HTTP client honors standard proxy env vars.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bitnami/kubectl has removed all version tags from Docker Hub,
leaving only :latest which is blocked by Kyverno disallow-latest-tag
policy. Switch to alpine/k8s:1.32.13 which has kubectl, bash, jq,
and uses proper version tags.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The :1.31 tag is gone from Docker Hub and :latest is blocked by Kyverno
disallow-latest-tag policy. The 1.29 image is cached on cluster nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The bitnami/kubectl:1.31 tag doesn't exist on Docker Hub. Updated all
hook jobs to use bitnami/kubectl:latest.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PreSync/PostSync hook Jobs reference argocd-hook-sa but it only existed
in the argocd namespace. Jobs running in app namespaces (crowdsec,
authentik, netbird, vault, istio-system) need the SA in their own
namespace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
