From 8372887ce1f1b0aec229dbf14ac9300948e28b1f Mon Sep 17 00:00:00 2001 From: Scooby Husky Date: Mon, 17 Aug 2026 16:13:48 -0500 Subject: [PATCH] Fix two long-standing bugs in the vault-init PreSync hook script Both pre-existing (not from today's work), found while cleaning up a recurring stale-pod pattern (a fresh vault-init Job failing on every sync of the vault Application): 1. Init-detection was always broken: `vault status -format=json` pretty-prints with a space after colons ("initialized": true), but the parser (grep -o '"initialized":[a-z]*' | cut -d: -f2) required no space - it always matched empty, so `initialized`/`sealed` were always blank regardless of real state. Every run concluded "not initialized" and tried to re-init, which correctly fails once Vault already has been ("Vault is already initialized"). Switched to jq. Separately, the old `|| echo '{}'` fallback also discarded valid JSON on vault status's normal non-zero exit codes (2 = sealed) - fixed too, though the parsing bug was the actual blocker. 2. `vault_exec` wraps `kubectl exec` without `-i`, so the heredoc piped into `vault policy write NAME -` never reached the remote command - vault saw an empty policy body. Never previously reached in practice since bug #1 always failed the script earlier. Added -i. Verified end-to-end with a one-off Job run: script now correctly detects the already-initialized state and completes successfully, including the policy/role writes that were previously unreachable. --- .../vault/manifests/vault-init-configmap.yaml | 29 ++++++++++++++++--- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/infrastructure/vault/manifests/vault-init-configmap.yaml b/infrastructure/vault/manifests/vault-init-configmap.yaml index 6c8b83b..8274f51 100644 --- a/infrastructure/vault/manifests/vault-init-configmap.yaml +++ b/infrastructure/vault/manifests/vault-init-configmap.yaml @@ -26,7 +26,12 @@ data: # Helper: run vault CLI inside the vault pod vault_exec() { - kubectl -n "$VAULT_NS" exec "$VAULT_POD" -- env VAULT_ADDR=http://127.0.0.1:8200 vault "$@" + # -i forwards stdin - needed for the heredoc-piped `vault policy write + # NAME -` call below. Without it, `kubectl exec` never passes stdin + # through and vault sees an empty policy body. This was unreachable + # until the init-detection fix above (the script always failed earlier, + # every run, before ever getting this far). + kubectl -n "$VAULT_NS" exec -i "$VAULT_POD" -- env VAULT_ADDR=http://127.0.0.1:8200 vault "$@" } # --- Wait for vault-0 pod to be ready --- @@ -45,10 +50,26 @@ data: sleep 5 # --- Check init/seal status --- + # `vault status` legitimately exits non-zero for normal states (2 = sealed, + # still prints valid JSON) as well as real failures (1 = can't connect, no + # output). The old `|| echo '{}'` fallback couldn't tell those apart - it + # discarded valid "already initialized, just sealed" JSON on exit 2 too, + # which made this script wrongly conclude "not initialized" and attempt to + # re-init an already-initialized Vault (always fails: "Vault is already + # initialized"). Only fall back to {} when there's truly no output. echo "==> Checking Vault status..." - status_json="$(vault_exec status -format=json 2>/dev/null || echo '{}')" - initialized="$(echo "$status_json" | grep -o '"initialized":[a-z]*' | cut -d: -f2 || echo "")" - sealed="$(echo "$status_json" | grep -o '"sealed":[a-z]*' | cut -d: -f2 || echo "")" + status_json="$(vault_exec status -format=json 2>/dev/null)" + if [ -z "$status_json" ]; then + status_json="{}" + fi + # jq, not grep/cut - `vault status -format=json` pretty-prints with a space + # after each colon ("initialized": true), which the old grep -o + # '"initialized":[a-z]*' pattern never matched (no space in the pattern) - + # initialized/sealed silently parsed as empty strings on every run + # regardless of actual Vault state, which is the real reason this script + # always tried to re-init an already-initialized Vault. + initialized="$(echo "$status_json" | jq -r '.initialized // empty')" + sealed="$(echo "$status_json" | jq -r '.sealed // empty')" root_token="" unseal_key=""