authentik HA: floating DNS hostname + failover-watcher

Builds the one genuinely new component from the plan - everything else
reuses CNPG/etcd as-is.

- infrastructure/ha-failover/manifests/heartbeat-writer.yaml: home writes
  a fresh timestamp to etcd (/ha-failover/home-heartbeat) every 10s via
  its local etcd member. No custom quorum/voting logic - etcd's own Raft
  consensus (majority write/linearizable read) already gives the 'a
  majority agrees' guarantee.

- scripts/ha-failover-watcher.sh + .service: runs on the VPS as a
  systemd service (not k3s - must survive the VPS's own cluster being
  unhealthy). Reads the heartbeat via its own local etcd member; if the
  read itself fails, we can't tell if home is down or if this VPS is the
  one partitioned, so it does NOT act (etcd's consensus requirement
  provides the split-brain safety here, not custom code). If the read
  succeeds and the heartbeat is stale (>45s, ~4-5 missed beats), home is
  confirmed down by majority - promotes pg-authentik's Cluster CR
  (spec.replica self/primary/source -> vps, no promotionToken - confirmed
  live via dry-run that CNPG's webhook doesn't require one, it's only for
  planned/graceful switchover LSN cross-checks that aren't available
  during a genuine unplanned outage) and flips the
  pg-authentik.ha.huskypup.net Cloudflare A record to the VPS's IP.

- scripts/ha-failback-authentik.sh: the deliberately manual reverse -
  human confirms home is healthy and caught up before running this.

- Floating hostname plumbing: both sites' Authentik values now read
  AUTHENTIK_POSTGRESQL__HOST/PORT as pg-authentik.ha.huskypup.net:61432
  instead of the local pg-authentik-app secret's host/port, so app pods
  on either site always reach whichever site is actually primary. Both
  sites listen on the same external port (VPS gets a new
  pg-authentik-forward.service socat forward, mirroring the existing
  minio-forward.service pattern, so its NodePort 32433 is externally
  reachable on 61432 same as home's UniFi-forwarded port).

- Found and worked around two real bugs surfaced while wiring this up
  (fixed manually via scripts handed to the user - both blocked by the
  Claude Code auto-mode classifier as credential-transmission /
  cluster-DNS-edit actions):
  - VPS's pg-authentik-app secret had a STALE password from before
    streaming replication existed - the live Postgres role password now
    replicates from home via WAL, but VPS's local K8s secret never got
    updated to match. Needs a one-time sync (and again on any future
    rotation).
  - This UniFi does not support NAT hairpin/loopback for its own WAN
    port-forwards - home's own pods resolving the floating hostname need
    a local CoreDNS rewrite straight to pg-authentik-rw, confirmed live by
    a DNS-resolves-but-TCP-connect-fails test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

scripts/ha-failover-watcher.service

@@ -0,0 +1,25 @@
+# Multi-site active failover pilot (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md). Install on the
+# VPS (172.93.53.139) alongside ha-failover-watcher.sh:
+#   scp scripts/ha-failover-watcher.sh root@172.93.53.139:/usr/local/bin/
+#   ssh root@172.93.53.139 chmod +x /usr/local/bin/ha-failover-watcher.sh
+#   mkdir -p /etc/ha-failover on the VPS, put the Cloudflare DNS-edit
+#     token (same one cert-manager's cloudflare-token-secret uses) in
+#     /etc/ha-failover/cloudflare-token, chmod 600, chown root
+#   scp this file to /etc/systemd/system/ha-failover-watcher.service
+#   systemctl daemon-reload && systemctl enable --now ha-failover-watcher
+#
+# Requires: curl, jq, kubectl, base64 - all already present on the VPS
+# from earlier in this session.
+[Unit]
+Description=HA failover watcher - promotes pg-authentik to primary if home's etcd heartbeat goes stale (majority-confirmed via etcd quorum)
+After=network.target k3s.service
+
+[Service]
+ExecStart=/usr/local/bin/ha-failover-watcher.sh
+Restart=always
+RestartSec=5
+User=root
+
+[Install]
+WantedBy=multi-user.target