vault-init-and-store.sh minted the auth/kubernetes/config reviewer JWT via
'kubectl create token vault-auth' with no --duration, defaulting to a 1hr
TTL, then wrote it into Vault as a static token_reviewer_jwt. ~1hr after
every cluster boot/hook rerun it silently expired, breaking TokenReview
for every kubernetes-auth login (including ESO's) with a generic 403 that
logged nothing at INFO/ERROR. This was the actual root cause of today's
broad ArgoCD Degraded wave across ~14 apps - not Vault's seal state, which
was fine the whole time.
Fix: clear token_reviewer_jwt and rely on Vault's local-JWT auto-detection
(disable_local_ca_jwt=false default), which reads the vault pod's own
kubelet-refreshed SA token from disk on every call instead of a static
copy. The vault SA already has system:auth-delegator via the existing
vault-server-binding ClusterRoleBinding.
Applied live directly against Vault to unblock immediately; confirmed
ClusterSecretStore vault-backend flipped to Ready=True and previously
broken ExternalSecrets (guacamole, teslamate, unpoller, netbird x3) all
resynced successfully.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both pre-existing (not from today's work), found while cleaning up a
recurring stale-pod pattern (a fresh vault-init Job failing on every
sync of the vault Application):
1. Init-detection was always broken: `vault status -format=json`
pretty-prints with a space after colons ("initialized": true), but
the parser (grep -o '"initialized":[a-z]*' | cut -d: -f2) required
no space - it always matched empty, so `initialized`/`sealed` were
always blank regardless of real state. Every run concluded "not
initialized" and tried to re-init, which correctly fails once Vault
already has been ("Vault is already initialized"). Switched to jq.
Separately, the old `|| echo '{}'` fallback also discarded valid
JSON on vault status's normal non-zero exit codes (2 = sealed) -
fixed too, though the parsing bug was the actual blocker.
2. `vault_exec` wraps `kubectl exec` without `-i`, so the heredoc piped
into `vault policy write NAME -` never reached the remote command -
vault saw an empty policy body. Never previously reached in
practice since bug #1 always failed the script earlier. Added -i.
Verified end-to-end with a one-off Job run: script now correctly
detects the already-initialized state and completes successfully,
including the policy/role writes that were previously unreachable.
