Completes the 'make the repo proper' cleanup from earlier this session -
these were flagged but deliberately not touched in 185e9c2 given the
blast radius (live SSO for 8 apps). User confirmed: fix all 8 now.
infrastructure/authentik/{argocd,gitlab,nextcloud,grafana,n8n,guacamole,
rancher,vault}-blueprint.yaml: converted from plain ConfigMap (client_secret
hardcoded) to ExternalSecret with a templated blueprint body
(client_secret: "{{ .clientSecret }}") pulling from Vault. Chart already
supports mounting blueprints from Secrets (blueprints.secrets, alongside
blueprints.configMaps) - infrastructure/authentik/values.yaml updated to
route these 8 there instead.
For argocd/nextcloud/n8n/guacamole/rancher: Vault already had the matching
value at secret/<app>-oauth (the APP side was already Vault-backed via its
own ExternalSecret) - the blueprint was the only remaining plaintext copy.
For gitlab/grafana/vault: Vault had no copy at all yet - created
secret/{gitlab,grafana,vault}-oauth with the EXISTING live values (not
rotated - these are the actual working credentials right now, rotating
would break login until every consumer is updated in lockstep, which is
out of scope for a cleanup pass). Also fixed the OTHER plaintext copies
that existed for these three specifically:
- apps/gitlab/manifests/external-secret-oidc.yaml (new): replaces a
manually kubectl-created, never-git-tracked gitlab-oidc-secret.
- infrastructure/grafana/manifests/grafana-oauth-secret.yaml: was a
plain Secret whose own comment said 'hardcoded from blueprint'.
- infrastructure/vault/manifests/vault-init-{configmap,job}.yaml: this
one COULDN'T be converted to the same ExternalSecret-from-Vault
pattern - it's the PostSync Job that grants ESO's own Kubernetes-auth
role in Vault, so ESO can't yet authenticate to pull anything from
Vault at the point this script runs (genuinely circular). Sourced
from a new vault-oidc-bootstrap Secret instead - created once
manually (kubectl, not git, matching how Vault's own root/unseal
material is already handled), independent of the ESO pipeline.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
vault-init-and-store.sh minted the auth/kubernetes/config reviewer JWT via
'kubectl create token vault-auth' with no --duration, defaulting to a 1hr
TTL, then wrote it into Vault as a static token_reviewer_jwt. ~1hr after
every cluster boot/hook rerun it silently expired, breaking TokenReview
for every kubernetes-auth login (including ESO's) with a generic 403 that
logged nothing at INFO/ERROR. This was the actual root cause of today's
broad ArgoCD Degraded wave across ~14 apps - not Vault's seal state, which
was fine the whole time.
Fix: clear token_reviewer_jwt and rely on Vault's local-JWT auto-detection
(disable_local_ca_jwt=false default), which reads the vault pod's own
kubelet-refreshed SA token from disk on every call instead of a static
copy. The vault SA already has system:auth-delegator via the existing
vault-server-binding ClusterRoleBinding.
Applied live directly against Vault to unblock immediately; confirmed
ClusterSecretStore vault-backend flipped to Ready=True and previously
broken ExternalSecrets (guacamole, teslamate, unpoller, netbird x3) all
resynced successfully.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both pre-existing (not from today's work), found while cleaning up a
recurring stale-pod pattern (a fresh vault-init Job failing on every
sync of the vault Application):
1. Init-detection was always broken: `vault status -format=json`
pretty-prints with a space after colons ("initialized": true), but
the parser (grep -o '"initialized":[a-z]*' | cut -d: -f2) required
no space - it always matched empty, so `initialized`/`sealed` were
always blank regardless of real state. Every run concluded "not
initialized" and tried to re-init, which correctly fails once Vault
already has been ("Vault is already initialized"). Switched to jq.
Separately, the old `|| echo '{}'` fallback also discarded valid
JSON on vault status's normal non-zero exit codes (2 = sealed) -
fixed too, though the parsing bug was the actual blocker.
2. `vault_exec` wraps `kubectl exec` without `-i`, so the heredoc piped
into `vault policy write NAME -` never reached the remote command -
vault saw an empty policy body. Never previously reached in
practice since bug #1 always failed the script earlier. Added -i.
Verified end-to-end with a one-off Job run: script now correctly
detects the already-initialized state and completes successfully,
including the policy/role writes that were previously unreachable.
