mirror of
https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
synced 2026-08-21 05:26:49 +00:00
Scooby HuskyandClaude Sonnet 5
185e9c292e
Root cause of the 'PostgreSQL connection failed, retrying... (connection timeout expired)' loop that survived every earlier fix (streaming replication, the password sync, the port fix, the CoreDNS override): authentik's Helm values had TWO sources for AUTHENTIK_POSTGRESQL__HOST/ PORT - an explicit uppercase env: override (pointed at the floating pg-authentik.ha.huskypup.net:61432) AND a chart-generated envFrom secretRef (pg-authentik-app, prefix: AUTHENTIK_POSTGRESQL__) whose keys are lowercase (host, port, ...), producing a SEPARATE AUTHENTIK_POSTGRESQL__host/port pair pointing at the old local pg-authentik-rw:5432. Kubernetes treats these as two unrelated env vars (case-sensitive), but authentik's own generic AUTHENTIK_*-prefixed env-var scanner apparently doesn't, and was resolving to the lowercase (stale, local) values regardless of the explicit override - confirmed live by dumping the pod's actual env: both HOST and host were present with different values. Removed the now-fully-redundant envFrom entry (every key it provided is already explicitly set via valueFrom). Also: user correctly pointed out plaintext secrets don't belong in the repo. Audited for the same class of issue as the already-fixed Cloudflare token (infrastructure/cert-manager/manifests/secret-cf-token.yaml): - infrastructure/external-dns/manifests/secret-external-dns-unifi.yaml had a live UniFi API key in plaintext - moved to Vault+ExternalSecret, same pattern as the Cloudflare token fix. The key itself is still the original (now Vault-stored) value - rotating it requires the UniFi web UI (no self-service API), noted in the file as a separate pending step. - infrastructure/vps-eso/manifests/clustersecretstore.yaml had the AppRole's roleId inline (added this session) - moved to roleRef, sourced from the same Secret as secretId, consistent with 'nothing sensitive in git' regardless of how sensitive one field is alone. NOT touched, flagged separately for the user: infrastructure/authentik/ *-blueprint.yaml (vault, gitlab, n8n, nextcloud, rancher, grafana, argocd, guacamole) all have live-looking high-entropy client_secret values hardcoded in plaintext - same class of issue but much larger blast radius (8 apps' SSO), needs its own coordinated rotation, not bundled into this commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
89 lines
4.3 KiB
YAML
89 lines
4.3 KiB
YAML
# Multi-site active failover pilot (see
|
|
# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - closes the gap
|
|
# flagged 2026-08-20: manually kubectl/ssh-copying secrets between home
|
|
# and the VPS defeats the point of having Vault at all. This gives the
|
|
# VPS its own real Vault -> ESO pipeline instead.
|
|
#
|
|
# External Secrets Operator itself is installed directly via helm
|
|
# (out-of-band, like k3s/cert-manager/Netbird - see
|
|
# scripts/vps-bootstrap.sh's existing pattern for why those aren't
|
|
# GitOps-managed either):
|
|
# helm repo add external-secrets https://charts.external-secrets.io
|
|
# helm install external-secrets external-secrets/external-secrets \
|
|
# --namespace external-secrets --create-namespace \
|
|
# --version 0.20.4 --set installCRDs=true
|
|
# (0.20.4 matches home's version - see argocd-apps/infrastructure/
|
|
# external-secrets.yaml)
|
|
#
|
|
# Auth: AppRole, not Kubernetes auth - home's existing vault-backend
|
|
# ClusterSecretStore (infrastructure/vault/manifests/clustersecretstore.yaml)
|
|
# uses Vault's kubernetes auth method, which verifies a ServiceAccount JWT
|
|
# against THAT cluster's own API server - doesn't work for the VPS, it's a
|
|
# completely separate k3s cluster with no federation to home's API server.
|
|
# AppRole is the standard way to authenticate an external/non-native
|
|
# client to Vault instead.
|
|
#
|
|
# Reachable via https://vault.kube.huskypup.net - a NEW public Cloudflare
|
|
# A record (home was previously deliberately kept off the public
|
|
# internet - this was an explicit, confirmed decision, not a default).
|
|
# Goes through the same Istio ingress gateway that already serves other
|
|
# public *.kube.huskypup.net hosts, so no new UniFi port-forward/NodePort
|
|
# needed, and no Istio PeerAuthentication/AuthorizationPolicy change
|
|
# either - unlike the ha-authentik-postgres NodePort case, this traffic
|
|
# arrives already wrapped in a normal in-mesh call from the ingress
|
|
# gateway's own identity, not raw external TCP straight to a pod.
|
|
#
|
|
# NOTE: the AppRole's issued token/secret_id CANNOT be IP-bound
|
|
# (token_bound_cidrs / secret_id_bound_cidrs) over this path - confirmed
|
|
# live 2026-08-20 that Vault only ever sees the ingress gateway's own pod
|
|
# IP for any request arriving this way, never the VPS's real source IP.
|
|
# Security boundary here is AppRole credential secrecy + the narrow
|
|
# read-only secret/vps/* policy (vps-eso-reader), not network-level
|
|
# restriction - this is the standard/expected shape of AppRole auth for
|
|
# external clients generally, CIDR-binding is normally extra
|
|
# defense-in-depth on top rather than the primary mechanism.
|
|
#
|
|
# One-time manual bootstrap on Vault's side (already done 2026-08-20, not
|
|
# scripted - matches every other Vault policy/auth-method setup in this
|
|
# repo, which are also applied by hand via `vault` CLI, not GitOps):
|
|
# vault auth enable approle
|
|
# vault policy write vps-eso-reader - <<'EOF'
|
|
# path "secret/data/vps/*" { capabilities = ["read", "list"] }
|
|
# path "secret/metadata/vps/*" { capabilities = ["read", "list"] }
|
|
# EOF
|
|
# vault write auth/approle/role/vps-eso token_policies="vps-eso-reader" \
|
|
# token_ttl=1h token_max_ttl=4h secret_id_num_uses=0
|
|
# vault read auth/approle/role/vps-eso/role-id # -> roleId below
|
|
# vault write -f auth/approle/role/vps-eso/secret-id # -> secret_id
|
|
#
|
|
# vault-approle-creds is a plain Secret created manually on the VPS
|
|
# (kubectl, not git - same reasoning as every other VPS secret, and
|
|
# consistent regardless of "how sensitive" a given field is on its own -
|
|
# role_id and secret_id both live in the same Secret, neither in git):
|
|
# kubectl -n external-secrets create secret generic vault-approle-creds \
|
|
# --from-literal=role_id=<from role-id above> \
|
|
# --from-literal=secret_id=<from secret-id above>
|
|
# Rotate the secret_id periodically by writing a new one and patching this
|
|
# Secret.
|
|
apiVersion: external-secrets.io/v1
|
|
kind: ClusterSecretStore
|
|
metadata:
|
|
name: vault-backend
|
|
spec:
|
|
provider:
|
|
vault:
|
|
server: https://vault.kube.huskypup.net
|
|
path: secret # KV v2 mount, same as home's vault-backend
|
|
version: v2
|
|
auth:
|
|
appRole:
|
|
path: approle
|
|
roleRef:
|
|
name: vault-approle-creds
|
|
namespace: external-secrets
|
|
key: role_id
|
|
secretRef:
|
|
name: vault-approle-creds
|
|
namespace: external-secrets
|
|
key: secret_id
|