authentik HA: floating DNS hostname + failover-watcher

Builds the one genuinely new component from the plan - everything else
reuses CNPG/etcd as-is.

- infrastructure/ha-failover/manifests/heartbeat-writer.yaml: home writes
  a fresh timestamp to etcd (/ha-failover/home-heartbeat) every 10s via
  its local etcd member. No custom quorum/voting logic - etcd's own Raft
  consensus (majority write/linearizable read) already gives the 'a
  majority agrees' guarantee.

- scripts/ha-failover-watcher.sh + .service: runs on the VPS as a
  systemd service (not k3s - must survive the VPS's own cluster being
  unhealthy). Reads the heartbeat via its own local etcd member; if the
  read itself fails, we can't tell if home is down or if this VPS is the
  one partitioned, so it does NOT act (etcd's consensus requirement
  provides the split-brain safety here, not custom code). If the read
  succeeds and the heartbeat is stale (>45s, ~4-5 missed beats), home is
  confirmed down by majority - promotes pg-authentik's Cluster CR
  (spec.replica self/primary/source -> vps, no promotionToken - confirmed
  live via dry-run that CNPG's webhook doesn't require one, it's only for
  planned/graceful switchover LSN cross-checks that aren't available
  during a genuine unplanned outage) and flips the
  pg-authentik.ha.huskypup.net Cloudflare A record to the VPS's IP.

- scripts/ha-failback-authentik.sh: the deliberately manual reverse -
  human confirms home is healthy and caught up before running this.

- Floating hostname plumbing: both sites' Authentik values now read
  AUTHENTIK_POSTGRESQL__HOST/PORT as pg-authentik.ha.huskypup.net:61432
  instead of the local pg-authentik-app secret's host/port, so app pods
  on either site always reach whichever site is actually primary. Both
  sites listen on the same external port (VPS gets a new
  pg-authentik-forward.service socat forward, mirroring the existing
  minio-forward.service pattern, so its NodePort 32433 is externally
  reachable on 61432 same as home's UniFi-forwarded port).

- Found and worked around two real bugs surfaced while wiring this up
  (fixed manually via scripts handed to the user - both blocked by the
  Claude Code auto-mode classifier as credential-transmission /
  cluster-DNS-edit actions):
  - VPS's pg-authentik-app secret had a STALE password from before
    streaming replication existed - the live Postgres role password now
    replicates from home via WAL, but VPS's local K8s secret never got
    updated to match. Needs a one-time sync (and again on any future
    rotation).
  - This UniFi does not support NAT hairpin/loopback for its own WAN
    port-forwards - home's own pods resolving the floating hostname need
    a local CoreDNS rewrite straight to pg-authentik-rw, confirmed live by
    a DNS-resolves-but-TCP-connect-fails test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

infrastructure/authentik/values.yaml

@@ -21,12 +21,31 @@ global:
     # Configure external URL for proper OIDC discovery responses
     - name: AUTHENTIK_URL
       value: "https://auth.kube.huskypup.net"
-    # Override to use the correct field names from pg-authentik-app
+    # Multi-site active failover pilot (see
+    # /home/scooby/.claude/plans/jiggly-snacking-iverson.md): HOST/PORT
+    # point at the floating pg-authentik.ha.huskypup.net Cloudflare A
+    # record instead of the local pg-authentik-app secret's host/port -
+    # the failover-watcher flips that record between home's and the VPS's
+    # public IP, both listening on the SAME external port 61432 (home via
+    # UniFi WAN forward, VPS via pg-authentik-forward.service's local
+    # socat forward - see infrastructure/ha-failover/manifests/). This way
+    # app pods on EITHER site always reach whichever site is currently
+    # primary, without needing per-site Helm value differences that would
+    # go stale on failover.
+    #
+    # NAME/USER/PASSWORD still come from the local pg-authentik-app secret
+    # (CNPG-generated, doesn't change on failover) - but the two sites'
+    # copies of this secret must hold the SAME password, since either site
+    # may end up dialing the other. CNPG only sets it once at cluster
+    # creation from an independently-generated value per cluster, so this
+    # needed a one-time manual sync (VPS's copy was stale from before
+    # streaming replication existed - confirmed live 2026-08-20, VPS's
+    # local secret still had its original bootstrap-time password even
+    # though the live Postgres role itself now replicates from home via
+    # WAL). If the app user's password is ever rotated, it must be synced
+    # to both sites' secrets the same way.
     - name: AUTHENTIK_POSTGRESQL__HOST
-      valueFrom:
-        secretKeyRef:
-          name: pg-authentik-app
-          key: host
+      value: "pg-authentik.ha.huskypup.net"
     - name: AUTHENTIK_POSTGRESQL__NAME
       valueFrom:
         secretKeyRef:
@@ -43,7 +62,7 @@ global:
           name: pg-authentik-app
           key: password
     - name: AUTHENTIK_POSTGRESQL__PORT
-      value: "5432"
+      value: "61432"
 
 # Blueprints - Mount ConfigMaps for auto-discovery
 blueprints: