Both pre-existing (not from today's work), found while cleaning up a
recurring stale-pod pattern (a fresh vault-init Job failing on every
sync of the vault Application):
1. Init-detection was always broken: `vault status -format=json`
pretty-prints with a space after colons ("initialized": true), but
the parser (grep -o '"initialized":[a-z]*' | cut -d: -f2) required
no space - it always matched empty, so `initialized`/`sealed` were
always blank regardless of real state. Every run concluded "not
initialized" and tried to re-init, which correctly fails once Vault
already has been ("Vault is already initialized"). Switched to jq.
Separately, the old `|| echo '{}'` fallback also discarded valid
JSON on vault status's normal non-zero exit codes (2 = sealed) -
fixed too, though the parsing bug was the actual blocker.
2. `vault_exec` wraps `kubectl exec` without `-i`, so the heredoc piped
into `vault policy write NAME -` never reached the remote command -
vault saw an empty policy body. Never previously reached in
practice since bug #1 always failed the script earlier. Added -i.
Verified end-to-end with a one-off Job run: script now correctly
detects the already-initialized state and completes successfully,
including the policy/role writes that were previously unreachable.
- ZONE_NAME was "kube.huskypup.net" in both the home-ip-ddns CronJob and
the VPS failover watcher - that's a record, not a Cloudflare zone (the
actual zone is the parent "huskypup.net"). Caused home-ip-ddns to fail
every run (zone lookup returned zero results, curl -f exit 22) -
confirmed live and fixed.
- Added seccompProfile/non-root/dropped-capabilities securityContext to
the three CronJobs added this session that were missing it (flagged by
the cluster's "restricted" PodSecurity admission). Repointed the raft
snapshot job's mc binary install from /usr/local/bin to /tmp so it
still works running as non-root.
Foundation for a DR/backup path using an always-on VPS as a second
ArgoCD-managed cluster, plus DB/backup standardization work that fell
out of it:
- vps-standby ArgoCD cluster destination + AppProject, MinIO backup
receiver, VPS bootstrap script (k3s, Netbird, cert-manager)
- Dual-site DNS failover watcher + home-IP DDNS CronJob, Cloudflare
token moved out of git into Vault+ExternalSecret
- Nextcloud migrated from ad-hoc MariaDB to CNPG + redis-operator
(matches n8n/Authentik/GitLab's backup-native pattern)
- Authentik's CNPG manifests moved into the actual ArgoCD-synced
manifests/ path (were present but never wired into the sync path)
- Vault raft-snapshot CronJob, CNPG barmanObjectStore backups
(Authentik/n8n/Nextcloud), Nextcloud file-PVC restic sync - all
targeting the new VPS MinIO receiver
See VPS Warm-Standby plan doc for full design rationale.
bitnami/kubectl has removed all version tags from Docker Hub,
leaving only :latest which is blocked by Kyverno disallow-latest-tag
policy. Switch to alpine/k8s:1.32.13 which has kubectl, bash, jq,
and uses proper version tags.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The :1.31 tag is gone from Docker Hub and :latest is blocked by Kyverno
disallow-latest-tag policy. The 1.29 image is cached on cluster nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The bitnami/kubectl:1.31 tag doesn't exist on Docker Hub. Updated all
hook jobs to use bitnami/kubectl:latest.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PreSync/PostSync hook Jobs reference argocd-hook-sa but it only existed
in the argocd namespace. Jobs running in app namespaces (crowdsec,
authentik, netbird, vault, istio-system) need the SA in their own
namespace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scooby Husky