mirror of
https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
synced 2026-08-21 05:26:49 +00:00
Scooby HuskyandClaude Sonnet 5
cbf7c07711
User correctly flagged this: manually copying secrets between home and the VPS defeats the entire point of having Vault. Every VPS secret this session (MinIO creds, Cloudflare token, CNPG certs, the pg-authentik-app password) was a one-off kubectl create secret copy-paste, because the VPS's k3s cluster had no Vault/ESO pipeline at all - this builds one. - infrastructure/authentik/manifests/ha-postgres-app-pushsecret.yaml: home pushes pg-authentik-app's password into Vault at secret/vps/pg-authentik-app (PushSecret, not just Get - this password is CNPG-generated, not Vault-native, so it has to originate from a push). - infrastructure/vps-eso/manifests/clustersecretstore.yaml: ESO on the VPS (installed via helm, out-of-band like k3s/cert-manager - see the file's own header) authenticates to home's Vault via AppRole (not Kubernetes auth - the VPS is a separate cluster with no federation to home's API server). Reachable via a new public https://vault.kube.huskypup.net record - Vault was deliberately kept off the public internet before this, explicitly confirmed with the user before opening it. Traffic goes through the same Istio ingress gateway already serving other public hosts, so unlike the ha-authentik-postgres NodePort case, no PeerAuthentication/AuthorizationPolicy change was needed - it arrives as a normal in-mesh call from the gateway's own identity, not raw external TCP to a pod. Found and fixed a real Vault gotcha while wiring this up: tried to remove the AppRole's token_bound_cidrs restriction (added first, before discovering Vault can't see the VPS's real source IP through the gateway - it only ever sees the gateway's own pod IP) by omitting the field from a follow-up - that does NOT clear it, the AppRole role endpoint preserves omitted fields rather than resetting them to default. Had to explicitly write token_bound_cidrs=. Spent a while chasing a misleading 403 permission denied on auth/token/ lookup-self before finding this - vault token capabilities said read was allowed (policy was fine), the actual rejection was IP-bound token use from an unbound context. - infrastructure/vps-standby/authentik/manifests/ pg-authentik-app-externalsecret.yaml: pulls it back down, Merge policy (only overwrites the password key - host/dbname/username stay local, CNPG still needs its own local -rw hostname for internal use). - argocd-apps/vps-standby/vps-eso.yaml: new Application for the ClusterSecretStore. Verified end-to-end: password now matches between home and VPS's pg-authentik-app secrets via this pipeline (not the earlier manual patch), confirmed by comparing both live secret values after ESO's sync. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
85 lines
4.2 KiB
YAML
85 lines
4.2 KiB
YAML
# Multi-site active failover pilot (see
|
|
# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - closes the gap
|
|
# flagged 2026-08-20: manually kubectl/ssh-copying secrets between home
|
|
# and the VPS defeats the point of having Vault at all. This gives the
|
|
# VPS its own real Vault -> ESO pipeline instead.
|
|
#
|
|
# External Secrets Operator itself is installed directly via helm
|
|
# (out-of-band, like k3s/cert-manager/Netbird - see
|
|
# scripts/vps-bootstrap.sh's existing pattern for why those aren't
|
|
# GitOps-managed either):
|
|
# helm repo add external-secrets https://charts.external-secrets.io
|
|
# helm install external-secrets external-secrets/external-secrets \
|
|
# --namespace external-secrets --create-namespace \
|
|
# --version 0.20.4 --set installCRDs=true
|
|
# (0.20.4 matches home's version - see argocd-apps/infrastructure/
|
|
# external-secrets.yaml)
|
|
#
|
|
# Auth: AppRole, not Kubernetes auth - home's existing vault-backend
|
|
# ClusterSecretStore (infrastructure/vault/manifests/clustersecretstore.yaml)
|
|
# uses Vault's kubernetes auth method, which verifies a ServiceAccount JWT
|
|
# against THAT cluster's own API server - doesn't work for the VPS, it's a
|
|
# completely separate k3s cluster with no federation to home's API server.
|
|
# AppRole is the standard way to authenticate an external/non-native
|
|
# client to Vault instead.
|
|
#
|
|
# Reachable via https://vault.kube.huskypup.net - a NEW public Cloudflare
|
|
# A record (home was previously deliberately kept off the public
|
|
# internet - this was an explicit, confirmed decision, not a default).
|
|
# Goes through the same Istio ingress gateway that already serves other
|
|
# public *.kube.huskypup.net hosts, so no new UniFi port-forward/NodePort
|
|
# needed, and no Istio PeerAuthentication/AuthorizationPolicy change
|
|
# either - unlike the ha-authentik-postgres NodePort case, this traffic
|
|
# arrives already wrapped in a normal in-mesh call from the ingress
|
|
# gateway's own identity, not raw external TCP straight to a pod.
|
|
#
|
|
# NOTE: the AppRole's issued token/secret_id CANNOT be IP-bound
|
|
# (token_bound_cidrs / secret_id_bound_cidrs) over this path - confirmed
|
|
# live 2026-08-20 that Vault only ever sees the ingress gateway's own pod
|
|
# IP for any request arriving this way, never the VPS's real source IP.
|
|
# Security boundary here is AppRole credential secrecy + the narrow
|
|
# read-only secret/vps/* policy (vps-eso-reader), not network-level
|
|
# restriction - this is the standard/expected shape of AppRole auth for
|
|
# external clients generally, CIDR-binding is normally extra
|
|
# defense-in-depth on top rather than the primary mechanism.
|
|
#
|
|
# One-time manual bootstrap on Vault's side (already done 2026-08-20, not
|
|
# scripted - matches every other Vault policy/auth-method setup in this
|
|
# repo, which are also applied by hand via `vault` CLI, not GitOps):
|
|
# vault auth enable approle
|
|
# vault policy write vps-eso-reader - <<'EOF'
|
|
# path "secret/data/vps/*" { capabilities = ["read", "list"] }
|
|
# path "secret/metadata/vps/*" { capabilities = ["read", "list"] }
|
|
# EOF
|
|
# vault write auth/approle/role/vps-eso token_policies="vps-eso-reader" \
|
|
# token_ttl=1h token_max_ttl=4h secret_id_num_uses=0
|
|
# vault read auth/approle/role/vps-eso/role-id # -> roleId below
|
|
# vault write -f auth/approle/role/vps-eso/secret-id # -> secret_id
|
|
#
|
|
# vault-approle-creds is a plain Secret created manually on the VPS
|
|
# (kubectl, not git - same reasoning as every other VPS secret):
|
|
# kubectl -n external-secrets create secret generic vault-approle-creds \
|
|
# --from-literal=role_id=<from role-id above> \
|
|
# --from-literal=secret_id=<from secret-id above>
|
|
# Rotate the secret_id periodically by writing a new one and patching this
|
|
# Secret - role_id is stable and not sensitive on its own (useless without
|
|
# a valid secret_id).
|
|
apiVersion: external-secrets.io/v1
|
|
kind: ClusterSecretStore
|
|
metadata:
|
|
name: vault-backend
|
|
spec:
|
|
provider:
|
|
vault:
|
|
server: https://vault.kube.huskypup.net
|
|
path: secret # KV v2 mount, same as home's vault-backend
|
|
version: v2
|
|
auth:
|
|
appRole:
|
|
path: approle
|
|
roleId: ecaf2eda-d922-f7cf-1143-690bbbb4d8ea # not sensitive alone, see note above
|
|
secretRef:
|
|
name: vault-approle-creds
|
|
namespace: external-secrets
|
|
key: secret_id
|