Fix real cause of VPS authentik's DB timeout loop + repo secrets cleanup

Root cause of the 'PostgreSQL connection failed, retrying... (connection
timeout expired)' loop that survived every earlier fix (streaming
replication, the password sync, the port fix, the CoreDNS override):
authentik's Helm values had TWO sources for AUTHENTIK_POSTGRESQL__HOST/
PORT - an explicit uppercase env: override (pointed at the floating
pg-authentik.ha.huskypup.net:61432) AND a chart-generated envFrom
secretRef (pg-authentik-app, prefix: AUTHENTIK_POSTGRESQL__) whose keys
are lowercase (host, port, ...), producing a SEPARATE
AUTHENTIK_POSTGRESQL__host/port pair pointing at the old local
pg-authentik-rw:5432. Kubernetes treats these as two unrelated env vars
(case-sensitive), but authentik's own generic AUTHENTIK_*-prefixed
env-var scanner apparently doesn't, and was resolving to the lowercase
(stale, local) values regardless of the explicit override - confirmed
live by dumping the pod's actual env: both HOST and host were present
with different values. Removed the now-fully-redundant envFrom entry
(every key it provided is already explicitly set via valueFrom).

Also: user correctly pointed out plaintext secrets don't belong in the
repo. Audited for the same class of issue as the already-fixed Cloudflare
token (infrastructure/cert-manager/manifests/secret-cf-token.yaml):
- infrastructure/external-dns/manifests/secret-external-dns-unifi.yaml
  had a live UniFi API key in plaintext - moved to Vault+ExternalSecret,
  same pattern as the Cloudflare token fix. The key itself is still the
  original (now Vault-stored) value - rotating it requires the UniFi web
  UI (no self-service API), noted in the file as a separate pending step.
- infrastructure/vps-eso/manifests/clustersecretstore.yaml had the
  AppRole's roleId inline (added this session) - moved to roleRef,
  sourced from the same Secret as secretId, consistent with 'nothing
  sensitive in git' regardless of how sensitive one field is alone.

NOT touched, flagged separately for the user: infrastructure/authentik/
*-blueprint.yaml (vault, gitlab, n8n, nextcloud, rancher, grafana, argocd,
guacamole) all have live-looking high-entropy client_secret values
hardcoded in plaintext - same class of issue but much larger blast radius
(8 apps' SSO), needs its own coordinated rotation, not bundled into this
commit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Scooby Husky
2026-08-20 18:47:50 -05:00
co-authored by Claude Sonnet 5
parent cfbe844ecd
commit 185e9c292e
4 changed files with 60 additions and 15 deletions
@@ -1,8 +1,33 @@
---
apiVersion: v1
kind: Secret
# Was a plain Secret with a live UniFi API key committed in plaintext -
# found and fixed 2026-08-20 (same class of issue as the Cloudflare token
# in infrastructure/cert-manager/manifests/secret-cf-token.yaml, which was
# already rotated to this pattern - this one was missed at the time).
# Moved to Vault+ExternalSecret, matching that precedent.
#
# The value in Vault right now (secret/unifi-api-key) is still the
# ORIGINAL key that was exposed in git history - rotating it (Settings ->
# Admins -> API Keys on the UniFi controller at https://172.28.110.1, no
# self-service API for this) is a separate, still-pending step. Once
# rotated:
# vault kv put secret/unifi-api-key api-key=<new key>
# ESO picks it up automatically (refreshInterval below), no restart
# needed on external-dns's side.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: external-dns-unifi-secret
namespace: external-dns
stringData:
api-key: 3Qha5hupHsCkkFyQb8z-T2kh-BasIGSH
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: vault-backend
target:
name: external-dns-unifi-secret
creationPolicy: Owner
data:
- secretKey: api-key
remoteRef:
key: unifi-api-key
property: api-key