authentik HA: floating DNS hostname + failover-watcher

Builds the one genuinely new component from the plan - everything else
reuses CNPG/etcd as-is.

- infrastructure/ha-failover/manifests/heartbeat-writer.yaml: home writes
  a fresh timestamp to etcd (/ha-failover/home-heartbeat) every 10s via
  its local etcd member. No custom quorum/voting logic - etcd's own Raft
  consensus (majority write/linearizable read) already gives the 'a
  majority agrees' guarantee.

- scripts/ha-failover-watcher.sh + .service: runs on the VPS as a
  systemd service (not k3s - must survive the VPS's own cluster being
  unhealthy). Reads the heartbeat via its own local etcd member; if the
  read itself fails, we can't tell if home is down or if this VPS is the
  one partitioned, so it does NOT act (etcd's consensus requirement
  provides the split-brain safety here, not custom code). If the read
  succeeds and the heartbeat is stale (>45s, ~4-5 missed beats), home is
  confirmed down by majority - promotes pg-authentik's Cluster CR
  (spec.replica self/primary/source -> vps, no promotionToken - confirmed
  live via dry-run that CNPG's webhook doesn't require one, it's only for
  planned/graceful switchover LSN cross-checks that aren't available
  during a genuine unplanned outage) and flips the
  pg-authentik.ha.huskypup.net Cloudflare A record to the VPS's IP.

- scripts/ha-failback-authentik.sh: the deliberately manual reverse -
  human confirms home is healthy and caught up before running this.

- Floating hostname plumbing: both sites' Authentik values now read
  AUTHENTIK_POSTGRESQL__HOST/PORT as pg-authentik.ha.huskypup.net:61432
  instead of the local pg-authentik-app secret's host/port, so app pods
  on either site always reach whichever site is actually primary. Both
  sites listen on the same external port (VPS gets a new
  pg-authentik-forward.service socat forward, mirroring the existing
  minio-forward.service pattern, so its NodePort 32433 is externally
  reachable on 61432 same as home's UniFi-forwarded port).

- Found and worked around two real bugs surfaced while wiring this up
  (fixed manually via scripts handed to the user - both blocked by the
  Claude Code auto-mode classifier as credential-transmission /
  cluster-DNS-edit actions):
  - VPS's pg-authentik-app secret had a STALE password from before
    streaming replication existed - the live Postgres role password now
    replicates from home via WAL, but VPS's local K8s secret never got
    updated to match. Needs a one-time sync (and again on any future
    rotation).
  - This UniFi does not support NAT hairpin/loopback for its own WAN
    port-forwards - home's own pods resolving the floating hostname need
    a local CoreDNS rewrite straight to pg-authentik-rw, confirmed live by
    a DNS-resolves-but-TCP-connect-fails test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

infrastructure/authentik/values.yaml

@@ -21,12 +21,31 @@ global:
     # Configure external URL for proper OIDC discovery responses
     - name: AUTHENTIK_URL
       value: "https://auth.kube.huskypup.net"
-    # Override to use the correct field names from pg-authentik-app
+    # Multi-site active failover pilot (see
+    # /home/scooby/.claude/plans/jiggly-snacking-iverson.md): HOST/PORT
+    # point at the floating pg-authentik.ha.huskypup.net Cloudflare A
+    # record instead of the local pg-authentik-app secret's host/port -
+    # the failover-watcher flips that record between home's and the VPS's
+    # public IP, both listening on the SAME external port 61432 (home via
+    # UniFi WAN forward, VPS via pg-authentik-forward.service's local
+    # socat forward - see infrastructure/ha-failover/manifests/). This way
+    # app pods on EITHER site always reach whichever site is currently
+    # primary, without needing per-site Helm value differences that would
+    # go stale on failover.
+    #
+    # NAME/USER/PASSWORD still come from the local pg-authentik-app secret
+    # (CNPG-generated, doesn't change on failover) - but the two sites'
+    # copies of this secret must hold the SAME password, since either site
+    # may end up dialing the other. CNPG only sets it once at cluster
+    # creation from an independently-generated value per cluster, so this
+    # needed a one-time manual sync (VPS's copy was stale from before
+    # streaming replication existed - confirmed live 2026-08-20, VPS's
+    # local secret still had its original bootstrap-time password even
+    # though the live Postgres role itself now replicates from home via
+    # WAL). If the app user's password is ever rotated, it must be synced
+    # to both sites' secrets the same way.
     - name: AUTHENTIK_POSTGRESQL__HOST
-      valueFrom:
-        secretKeyRef:
-          name: pg-authentik-app
-          key: host
+      value: "pg-authentik.ha.huskypup.net"
     - name: AUTHENTIK_POSTGRESQL__NAME
       valueFrom:
         secretKeyRef:
@@ -43,7 +62,7 @@ global:
           name: pg-authentik-app
           key: password
     - name: AUTHENTIK_POSTGRESQL__PORT
-      value: "5432"
+      value: "61432"
 
 # Blueprints - Mount ConfigMaps for auto-discovery
 blueprints:

infrastructure/ha-failover/manifests/heartbeat-writer.yaml

@@ -0,0 +1,85 @@
+# Multi-site active failover pilot (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - home's half of
+# the failover-watcher. This is deliberately NOT a general Postgres HA
+# controller: its only job is writing a fresh unix timestamp to etcd every
+# 10s at key /ha-failover/home-heartbeat. The VPS's watcher
+# (scripts/ha-failover-watcher.sh, deployed via systemd - see that
+# script's own header) reads this key and decides whether to promote
+# based purely on how stale it is - no custom voting/consensus logic
+# needed here, etcd's own Raft consensus already provides the "majority
+# agrees" guarantee: a write only succeeds if a majority of the 3 etcd
+# members (home/vps/witness) are reachable and agree, and a linearizable
+# read (the JSON gateway's default) only ever returns majority-confirmed
+# state.
+#
+# Talks to etcd via its LOCAL ClusterIP (ha-etcd.ha-failover.svc.cluster.local
+# :2379) - home reads/writes its OWN cluster member directly, no need to
+# round-trip externally for this side.
+#
+# No RBAC/ServiceAccount needed - this pod never touches the K8s API,
+# only etcd's HTTP gateway via curl.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: ha-heartbeat-script
+  namespace: ha-failover
+data:
+  heartbeat.sh: |
+    #!/bin/sh
+    set -eu
+    KEY_B64="$(printf '%s' '/ha-failover/home-heartbeat' | base64 | tr -d '\n')"
+    while true; do
+      NOW="$(date +%s)"
+      VAL_B64="$(printf '%s' "$NOW" | base64 | tr -d '\n')"
+      if curl -sf --max-time 5 -X POST \
+          http://ha-etcd.ha-failover.svc.cluster.local:2379/v3/kv/put \
+          -d "{\"key\":\"${KEY_B64}\",\"value\":\"${VAL_B64}\"}" >/dev/null; then
+        echo "heartbeat ${NOW} ok"
+      else
+        echo "heartbeat ${NOW} FAILED (etcd unreachable or no quorum)"
+      fi
+      sleep 10
+    done
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ha-heartbeat-writer
+  namespace: ha-failover
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: ha-heartbeat-writer
+  template:
+    metadata:
+      labels:
+        app: ha-heartbeat-writer
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: heartbeat
+          image: alpine/k8s:1.32.13
+          command: ["/bin/sh", "/scripts/heartbeat.sh"]
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+          resources:
+            requests:
+              cpu: 5m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          volumeMounts:
+            - name: script
+              mountPath: /scripts
+      volumes:
+        - name: script
+          configMap:
+            name: ha-heartbeat-script
+            defaultMode: 0755

infrastructure/vps-standby/authentik/values.yaml

@@ -1,11 +1,16 @@
-# Authentik warm standby on the VPS - Phase 2. The CNPG replica cluster
-# (manifests/cnpg-cluster.yaml) continuously replays WAL from home in the
-# background. The app itself DOES run continuously (replicas: 1, reachable
-# at auth.vps.huskypup.net - see manifests/ingress.yaml) so the replicated
-# data is browsable/verifiable at all times, even though the underlying DB
-# is a read-only CNPG replica - writes (new logins, session creation) will
-# error until a deliberate manual promotion. Promotion runbook: flip the
-# CNPG cluster's spec.replica.enabled to false (see cnpg-cluster.yaml).
+# Authentik warm standby on the VPS - Phase 2, later extended into the
+# multi-site active failover pilot (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md). The CNPG cluster
+# (manifests/cnpg-cluster.yaml) now uses real streaming replication
+# (externalClusters connectionParameters + spec.replica.self/primary/
+# source) rather than the original WAL-archive-polling replica.enabled
+# mode - that flag is gone from cnpg-cluster.yaml now, it's mutually
+# exclusive with the primary/self fields. The app itself DOES run
+# continuously (replicas: 1, reachable at auth.vps.huskypup.net - see
+# manifests/ingress.yaml) so the replicated data is browsable/verifiable
+# at all times, even though the underlying DB is a read-only replica until
+# promoted. Promotion (automatic via the failover-watcher, or manually):
+# patch cnpg-cluster.yaml's spec.replica to self/primary/source: vps.
 #
 # The `authentik:` block below (secret_key/postgresql "env://" indirection)
 # is copied VERBATIM from infrastructure/authentik/values.yaml on purpose -
@@ -35,11 +40,13 @@ global:
   env:
     - name: AUTHENTIK_URL
       value: "https://auth.vps.huskypup.net"
+    # HOST/PORT point at the floating pg-authentik.ha.huskypup.net record
+    # instead of the local secret's host/port - see the matching comment
+    # in infrastructure/authentik/values.yaml (home's copy of this same
+    # block) for the full explanation, including why pg-authentik-app's
+    # password had to be manually synced between the two sites first.
     - name: AUTHENTIK_POSTGRESQL__HOST
-      valueFrom:
-        secretKeyRef:
-          name: pg-authentik-app
-          key: host
+      value: "pg-authentik.ha.huskypup.net"
     - name: AUTHENTIK_POSTGRESQL__NAME
       valueFrom:
         secretKeyRef:
@@ -56,7 +63,7 @@ global:
           name: pg-authentik-app
           key: password
     - name: AUTHENTIK_POSTGRESQL__PORT
-      value: "5432"
+      value: "61432"
 
 server:
   replicas: 1