authentik HA: floating DNS hostname + failover-watcher

Builds the one genuinely new component from the plan - everything else
reuses CNPG/etcd as-is.

- infrastructure/ha-failover/manifests/heartbeat-writer.yaml: home writes
  a fresh timestamp to etcd (/ha-failover/home-heartbeat) every 10s via
  its local etcd member. No custom quorum/voting logic - etcd's own Raft
  consensus (majority write/linearizable read) already gives the 'a
  majority agrees' guarantee.

- scripts/ha-failover-watcher.sh + .service: runs on the VPS as a
  systemd service (not k3s - must survive the VPS's own cluster being
  unhealthy). Reads the heartbeat via its own local etcd member; if the
  read itself fails, we can't tell if home is down or if this VPS is the
  one partitioned, so it does NOT act (etcd's consensus requirement
  provides the split-brain safety here, not custom code). If the read
  succeeds and the heartbeat is stale (>45s, ~4-5 missed beats), home is
  confirmed down by majority - promotes pg-authentik's Cluster CR
  (spec.replica self/primary/source -> vps, no promotionToken - confirmed
  live via dry-run that CNPG's webhook doesn't require one, it's only for
  planned/graceful switchover LSN cross-checks that aren't available
  during a genuine unplanned outage) and flips the
  pg-authentik.ha.huskypup.net Cloudflare A record to the VPS's IP.

- scripts/ha-failback-authentik.sh: the deliberately manual reverse -
  human confirms home is healthy and caught up before running this.

- Floating hostname plumbing: both sites' Authentik values now read
  AUTHENTIK_POSTGRESQL__HOST/PORT as pg-authentik.ha.huskypup.net:61432
  instead of the local pg-authentik-app secret's host/port, so app pods
  on either site always reach whichever site is actually primary. Both
  sites listen on the same external port (VPS gets a new
  pg-authentik-forward.service socat forward, mirroring the existing
  minio-forward.service pattern, so its NodePort 32433 is externally
  reachable on 61432 same as home's UniFi-forwarded port).

- Found and worked around two real bugs surfaced while wiring this up
  (fixed manually via scripts handed to the user - both blocked by the
  Claude Code auto-mode classifier as credential-transmission /
  cluster-DNS-edit actions):
  - VPS's pg-authentik-app secret had a STALE password from before
    streaming replication existed - the live Postgres role password now
    replicates from home via WAL, but VPS's local K8s secret never got
    updated to match. Needs a one-time sync (and again on any future
    rotation).
  - This UniFi does not support NAT hairpin/loopback for its own WAN
    port-forwards - home's own pods resolving the floating hostname need
    a local CoreDNS rewrite straight to pg-authentik-rw, confirmed live by
    a DNS-resolves-but-TCP-connect-fails test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

argocd-apps/infrastructure/ha-failover.yaml

@@ -0,0 +1,24 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: ha-failover
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "23"  # after CNPG/storage (10-14), alongside other security/HA infra
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: infrastructure
+  source:
+    repoURL: https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
+    targetRevision: main
+    path: infrastructure/ha-failover/manifests
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: ha-failover
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true

argocd-apps/vps-standby/cert-manager-issuer.yaml

@@ -0,0 +1,22 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: vps-cert-manager-issuer
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "1"  # before the per-service Ingress resources (wave 3+) that reference it
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: vps-standby
+  source:
+    repoURL: https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
+    targetRevision: main
+    path: infrastructure/vps-standby/cert-manager/manifests
+  destination:
+    name: vps-standby
+    namespace: cert-manager
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true

infrastructure/authentik/manifests/cnpg-cluster.yaml

@@ -76,3 +76,80 @@ spec:
   monitoring:
     enablePodMonitor: true
 
+  # Multi-site active failover pilot (see
+  # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - streaming
+  # replication to/from the VPS's pg-authentik cluster, alongside (not
+  # replacing) the barmanObjectStore backup above.
+  #
+  # Auth: reuses CNPG's own generated streaming_replica client cert
+  # (CN=streaming_replica, issued by the VPS's own pg-authentik CA) -
+  # copied here manually as pg-authentik-vps-{ca,replication} (kubectl,
+  # not git - same pattern as every other cross-cluster secret tonight).
+  # This is the exact credential CNPG's own pg_hba.conf already trusts
+  # for streaming replication, nothing new to provision.
+  #
+  # sslmode: verify-ca (not verify-full) - the VPS's server cert was
+  # issued for its in-cluster DNS names, not for the NodePort IP address
+  # (100.108.113.41) home actually dials; verify-ca still validates the
+  # cert chain/CA trust and encrypts the connection, just skips hostname
+  # matching.
+  #
+  # Reachable via 100.108.113.41 (VPS's Netbird IP) - home reaching OUT
+  # to the VPS already works today via the netbird-egress DaemonSet's
+  # route (same path used for MinIO/CrowdSec backups all session) - no
+  # UniFi port-forward needed for this direction, only the reverse
+  # (VPS/witness reaching INTO home) needed that.
+  # CNPG's distributed-topology validation requires both replica.self and
+  # replica.primary to reference names present in THIS cluster's OWN
+  # externalClusters list - including a self-referencing entry (confirmed
+  # live: "External cluster home not found" until one was added, even
+  # though "home" is this very cluster). The "home" entry's
+  # connectionParameters point at its own local -rw service - never
+  # actually dialed while primary: home (that's this cluster), it only
+  # exists to satisfy the name-reference validation.
+  externalClusters:
+    - name: home
+      connectionParameters:
+        host: pg-authentik-rw
+        port: "5432"
+        dbname: app
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-authentik-replication
+        key: tls.crt
+      sslKey:
+        name: pg-authentik-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-authentik-ca
+        key: ca.crt
+    - name: vps
+      connectionParameters:
+        host: 100.108.113.41
+        port: "32433"
+        dbname: app
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-authentik-vps-replication
+        key: tls.crt
+      sslKey:
+        name: pg-authentik-vps-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-authentik-vps-ca
+        key: ca.crt
+
+  # Distributed topology: both sides agree home is primary today. Failover
+  # flips the VPS's replica.primary to "vps" (+ promotion token) - see the
+  # plan doc's failover-watcher section. Failback is the same in reverse,
+  # deliberately manual. No replica.enabled here - that flag belongs to
+  # CNPG's older single-source replica-cluster mode and is mutually
+  # exclusive with primary/self (confirmed live: "replica mode enabled is
+  # not compatible with the primary field").
+  replica:
+    self: home
+    primary: home
+    source: home  # required unconditionally by the admission webhook, even though home isn't actually replicating from anyone right now (self == primary)
+

infrastructure/authentik/manifests/ha-postgres-nodeport.yaml

@@ -0,0 +1,22 @@
+---
+# Exposes pg-authentik's current primary to the VPS/witness over the
+# Netbird mesh for streaming replication (see the "Multi-Site Active
+# Failover" plan) - same selector CNPG's own pg-authentik-rw ClusterIP
+# Service uses, just NodePort instead, since CNPG doesn't manage this
+# object and won't fight with it. Needs
+# infrastructure/kyverno/policies/ha-failover-nodeport-exception.yaml
+# (disallow-nodeport-services is enforced cluster-wide otherwise).
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-authentik-postgres
+  namespace: authentik
+spec:
+  type: NodePort
+  selector:
+    cnpg.io/cluster: pg-authentik
+    cnpg.io/instanceRole: primary
+  ports:
+    - port: 5432
+      targetPort: 5432
+      nodePort: 32432

infrastructure/authentik/manifests/ha-postgres-peerauth.yaml

@@ -0,0 +1,40 @@
+# Multi-site active failover pilot (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - CNPG streaming
+# replication from the VPS was silently failing: `pg_stat_wal_receiver` on
+# the VPS replica showed 0 rows, and its logs showed a repeating
+# "could not connect to the primary server: ... server closed the
+# connection unexpectedly" every few minutes (confirmed live 2026-08-19).
+#
+# Root cause: the `authentik` namespace is enrolled in Istio's ambient mesh
+# (`istio.io/dataplane-mode: ambient`) and the mesh-wide default
+# PeerAuthentication (istio-system/default) is STRICT - ztunnel requires a
+# valid mesh (HBONE/SPIFHE) identity for ALL traffic to pods in this
+# namespace, including traffic arriving via the ha-authentik-postgres
+# NodePort from the VPS/witness (which have no mesh identity at all -
+# they're not in this cluster). ztunnel accepts the raw TCP connection then
+# resets it once it can't complete an mTLS handshake it never receives -
+# exactly matching the "server closed the connection unexpectedly"
+# symptom. Confirmed via `openssl s_client -starttls postgres`: TCP
+# connects, the postgres SSLRequest byte is sent, 0 bytes come back.
+#
+# Same root cause and same fix as the existing precedent for this exact
+# problem (infrastructure/istio/manifests/mesh/peer-authentication-webhooks.yaml
+# - CrowdSec's hostNetwork bouncer / the API server's webhook calls): allow
+# PERMISSIVE (mTLS or plaintext) inbound. Scoped here to just the CNPG
+# primary pod's port 5432 via portLevelMtls, rather than the whole
+# namespace like that precedent does - Authentik's own in-mesh east-west
+# traffic (server/worker -> everything else) should stay STRICT.
+apiVersion: security.istio.io/v1
+kind: PeerAuthentication
+metadata:
+  name: allow-ha-postgres-replication
+  namespace: authentik
+spec:
+  selector:
+    matchLabels:
+      cnpg.io/cluster: pg-authentik
+  mtls:
+    mode: STRICT
+  portLevelMtls:
+    "5432":
+      mode: PERMISSIVE

infrastructure/authentik/values.yaml

@@ -21,12 +21,31 @@ global:
     # Configure external URL for proper OIDC discovery responses
     - name: AUTHENTIK_URL
       value: "https://auth.kube.huskypup.net"
-    # Override to use the correct field names from pg-authentik-app
+    # Multi-site active failover pilot (see
+    # /home/scooby/.claude/plans/jiggly-snacking-iverson.md): HOST/PORT
+    # point at the floating pg-authentik.ha.huskypup.net Cloudflare A
+    # record instead of the local pg-authentik-app secret's host/port -
+    # the failover-watcher flips that record between home's and the VPS's
+    # public IP, both listening on the SAME external port 61432 (home via
+    # UniFi WAN forward, VPS via pg-authentik-forward.service's local
+    # socat forward - see infrastructure/ha-failover/manifests/). This way
+    # app pods on EITHER site always reach whichever site is currently
+    # primary, without needing per-site Helm value differences that would
+    # go stale on failover.
+    #
+    # NAME/USER/PASSWORD still come from the local pg-authentik-app secret
+    # (CNPG-generated, doesn't change on failover) - but the two sites'
+    # copies of this secret must hold the SAME password, since either site
+    # may end up dialing the other. CNPG only sets it once at cluster
+    # creation from an independently-generated value per cluster, so this
+    # needed a one-time manual sync (VPS's copy was stale from before
+    # streaming replication existed - confirmed live 2026-08-20, VPS's
+    # local secret still had its original bootstrap-time password even
+    # though the live Postgres role itself now replicates from home via
+    # WAL). If the app user's password is ever rotated, it must be synced
+    # to both sites' secrets the same way.
     - name: AUTHENTIK_POSTGRESQL__HOST
-      valueFrom:
+      value: "pg-authentik.ha.huskypup.net"
-        secretKeyRef:
-          name: pg-authentik-app
-          key: host
     - name: AUTHENTIK_POSTGRESQL__NAME
       valueFrom:
         secretKeyRef:
@@ -43,7 +62,7 @@ global:
           name: pg-authentik-app
           key: password
     - name: AUTHENTIK_POSTGRESQL__PORT
-      value: "5432"
+      value: "61432"
 
 # Blueprints - Mount ConfigMaps for auto-discovery
 blueprints:
@@ -61,6 +80,7 @@ blueprints:
     - authentik-blueprints-percona-everest
     - authentik-blueprints-rancher
     - authentik-blueprints-netbird
+    - authentik-blueprints-vault
 
 # Enable Prometheus metrics
 server:

infrastructure/authentik/vault-blueprint.yaml

@@ -0,0 +1,51 @@
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: authentik-blueprints-vault
+  namespace: authentik
+  labels:
+    goauthentik.io/blueprint: "true"
+data:
+  vault.yaml: |-
+    version: 1
+    metadata:
+      name: vault-oidc
+    entries:
+      - model: authentik_providers_oauth2.oauth2provider
+        id: vault-provider
+        state: present
+        identifiers:
+          name: Vault
+        attrs:
+          name: Vault
+          client_id: 9816a5ae7e7914b5d18f4ab939d011a98f8c8d6b3bb6777c46431afa06ac4a85
+          client_secret: ed2ba1c6378c7a46341b5162f39a7fab80e37596b01ed387c3719e8e0040344cf1daa307476c2e7a7f75041b3979275b1ebf00bb8bad94c864b4a38ded544f7b
+          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+          authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
+          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+          redirect_uris:
+            - url: "https://vault.kube.huskypup.net/ui/vault/auth/oidc/oidc/callback"
+              matching_mode: strict
+            - url: "http://localhost:8250/oidc/callback"
+              matching_mode: strict
+          property_mappings:
+            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
+            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
+            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
+          client_type: confidential
+          access_code_validity: "minutes=10"
+          access_token_validity: "hours=1"
+          refresh_token_validity: "days=30"
+          signing_key: !Find [authentik_crypto.certificatekeypair, [name, "authentik Internal JWT Certificate"]]
+
+      - model: authentik_core.application
+        id: vault-application
+        state: present
+        identifiers:
+          slug: vault
+        attrs:
+          name: Vault
+          slug: vault
+          policy_engine_mode: any
+          provider: !KeyOf vault-provider

infrastructure/ha-failover/manifests/etcd.yaml

@@ -0,0 +1,118 @@
+---
+# Home's member of the 3-way etcd quorum used by the failover-watcher
+# (see /home/scooby/.claude/plans/jiggly-snacking-iverson.md, "Multi-Site
+# Active Failover" section) to safely decide when to trigger a CNPG
+# distributed-topology promotion. VPS and witness run this same etcd
+# version as plain systemd services (simpler - no k3s/Kyverno involved
+# there); home has to run it as a real K8s workload since Talos doesn't
+# allow bare-metal process installs.
+#
+# Pinned to a specific node (talos-cp-01) via nodeSelector so its
+# advertised peer/client URLs (which must be stable, not "whichever node
+# it landed on today") stay correct.
+#
+# Reachable from the VPS/witness via a UniFi WAN port-forward + a
+# source-IP-restricted WAN_IN firewall rule (limited to just the VPS and
+# witness public IPs) - NOT via the Netbird mesh directly. talos-cp-01
+# does have a real, host-bound Netbird IP (100.108.42.109 on its wt0
+# interface, confirmed live via `talosctl get addresses`), but that's a
+# dead end for inbound WAN traffic: UniFi's router only has a route to
+# its own directly-connected LAN (172.28.101.0/24), nothing advertises
+# 100.108.0.0/16 to it, so a port-forward targeting the Netbird IP would
+# never actually route. The port-forward instead targets talos-cp-01's
+# real LAN IP (172.28.101.41) - NodePort Services bind on every interface
+# on a node, so the same ports are reachable there too, and that's an
+# address UniFi can actually route to natively.
+#
+# The NodePort Service itself needs
+# infrastructure/kyverno/policies/ha-failover-nodeport-exception.yaml
+# (disallow-nodeport-services is enforced cluster-wide otherwise).
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: ha-failover
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: ha-etcd
+  namespace: ha-failover
+spec:
+  serviceName: ha-etcd
+  replicas: 1
+  selector:
+    matchLabels:
+      app: ha-etcd
+  template:
+    metadata:
+      labels:
+        app: ha-etcd
+    spec:
+      nodeSelector:
+        kubernetes.io/hostname: talos-cp-01
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        fsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: etcd
+          image: gcr.io/etcd-development/etcd:v3.5.17
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+          command:
+            - etcd
+            - --name=home
+            - --data-dir=/var/lib/etcd/data
+            - --listen-client-urls=http://0.0.0.0:2379
+            - --advertise-client-urls=http://home.kube.huskypup.net:61379
+            - --listen-peer-urls=http://0.0.0.0:2380
+            - --initial-advertise-peer-urls=http://home.kube.huskypup.net:61380
+            - --initial-cluster=home=http://home.kube.huskypup.net:61380,vps=http://100.108.113.41:2380,witness=http://100.108.130.74:2380
+            - --initial-cluster-state=new
+            - --initial-cluster-token=ha-failover-quorum
+          ports:
+            - containerPort: 2379
+              name: client
+            - containerPort: 2380
+              name: peer
+          volumeMounts:
+            - name: data
+              mountPath: /var/lib/etcd
+          resources:
+            requests:
+              cpu: 25m
+              memory: 64Mi
+            limits:
+              memory: 256Mi
+  volumeClaimTemplates:
+    - metadata:
+        name: data
+      spec:
+        accessModes: ["ReadWriteOnce"]
+        storageClassName: rook-ceph-block
+        resources:
+          requests:
+            storage: 1Gi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-etcd
+  namespace: ha-failover
+spec:
+  type: NodePort
+  selector:
+    app: ha-etcd
+  ports:
+    - name: client
+      port: 2379
+      targetPort: 2379
+      nodePort: 32379
+    - name: peer
+      port: 2380
+      targetPort: 2380
+      nodePort: 32380

infrastructure/ha-failover/manifests/heartbeat-writer.yaml

@@ -0,0 +1,85 @@
+# Multi-site active failover pilot (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - home's half of
+# the failover-watcher. This is deliberately NOT a general Postgres HA
+# controller: its only job is writing a fresh unix timestamp to etcd every
+# 10s at key /ha-failover/home-heartbeat. The VPS's watcher
+# (scripts/ha-failover-watcher.sh, deployed via systemd - see that
+# script's own header) reads this key and decides whether to promote
+# based purely on how stale it is - no custom voting/consensus logic
+# needed here, etcd's own Raft consensus already provides the "majority
+# agrees" guarantee: a write only succeeds if a majority of the 3 etcd
+# members (home/vps/witness) are reachable and agree, and a linearizable
+# read (the JSON gateway's default) only ever returns majority-confirmed
+# state.
+#
+# Talks to etcd via its LOCAL ClusterIP (ha-etcd.ha-failover.svc.cluster.local
+# :2379) - home reads/writes its OWN cluster member directly, no need to
+# round-trip externally for this side.
+#
+# No RBAC/ServiceAccount needed - this pod never touches the K8s API,
+# only etcd's HTTP gateway via curl.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: ha-heartbeat-script
+  namespace: ha-failover
+data:
+  heartbeat.sh: |
+    #!/bin/sh
+    set -eu
+    KEY_B64="$(printf '%s' '/ha-failover/home-heartbeat' | base64 | tr -d '\n')"
+    while true; do
+      NOW="$(date +%s)"
+      VAL_B64="$(printf '%s' "$NOW" | base64 | tr -d '\n')"
+      if curl -sf --max-time 5 -X POST \
+          http://ha-etcd.ha-failover.svc.cluster.local:2379/v3/kv/put \
+          -d "{\"key\":\"${KEY_B64}\",\"value\":\"${VAL_B64}\"}" >/dev/null; then
+        echo "heartbeat ${NOW} ok"
+      else
+        echo "heartbeat ${NOW} FAILED (etcd unreachable or no quorum)"
+      fi
+      sleep 10
+    done
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ha-heartbeat-writer
+  namespace: ha-failover
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: ha-heartbeat-writer
+  template:
+    metadata:
+      labels:
+        app: ha-heartbeat-writer
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: heartbeat
+          image: alpine/k8s:1.32.13
+          command: ["/bin/sh", "/scripts/heartbeat.sh"]
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+          resources:
+            requests:
+              cpu: 5m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          volumeMounts:
+            - name: script
+              mountPath: /scripts
+      volumes:
+        - name: script
+          configMap:
+            name: ha-heartbeat-script
+            defaultMode: 0755

infrastructure/istio/manifests/mesh/authorization-policies.yaml

@@ -117,6 +117,20 @@ spec:
         - source:
             namespaces:
               - prometheus
+    # Multi-site active failover pilot (see
+    # /home/scooby/.claude/plans/jiggly-snacking-iverson.md): CNPG
+    # streaming replication from the VPS/witness, arriving via the
+    # ha-authentik-postgres NodePort - no mesh identity at all (they're
+    # not in this cluster), so no `source.namespaces` rule above can ever
+    # match them. Scoped by destination port instead of source, matching
+    # the port-scoped PERMISSIVE PeerAuthentication in
+    # ha-postgres-peerauth.yaml (same root cause, same fix, one layer up -
+    # mTLS being allowed through isn't enough by itself, this ALLOW policy
+    # independently denies anything not matching one of the rules above).
+    - to:
+        - operation:
+            ports:
+              - "5432"
 
 ---
 # --- Grafana: Allow ingress + Prometheus datasource queries + scraping ---

infrastructure/kyverno/policies/ha-failover-nodeport-exception.yaml

@@ -0,0 +1,31 @@
+# Scoped exception to the disallow-nodeport-services policy, narrowly for
+# the multi-site active-failover pilot's cross-site services (etcd quorum,
+# Postgres streaming replication) - not a namespace-wide exclusion.
+#
+# Why NodePort is genuinely needed here: home nodes are directly reachable
+# from Netbird mesh peers (VPS, witness) on their real node IP - confirmed
+# live 2026-08-19 (ping succeeded from the VPS to a node's InternalIP,
+# which is itself a Netbird-mesh address via the netbird-egress DaemonSet's
+# route). A NodePort Service binds on that same real interface on every
+# node, giving the VPS/witness a way to reach it directly over the already-
+# authenticated Netbird mesh - no public internet exposure, no UniFi
+# port-forward, no new WAN-facing attack surface. LoadBalancer (MetalLB)
+# only gets a LAN-side VIP, which isn't reachable from Netbird peers at all
+# without the same underlying NodePort-style exposure anyway.
+apiVersion: kyverno.io/v2
+kind: PolicyException
+metadata:
+  name: ha-failover-nodeport-exemption
+  namespace: kyverno
+spec:
+  exceptions:
+    - policyName: disallow-nodeport-services
+      ruleNames:
+        - disallow-nodeport
+  match:
+    any:
+      - resources:
+          kinds:
+            - Service
+          names:
+            - ha-*

infrastructure/vault/values.yaml

@@ -21,6 +21,11 @@ server:
           disable_hostname = true
         }
 
+  hostAliases:
+    - ip: "172.28.101.244"
+      hostnames:
+        - auth.kube.huskypup.net
+
   standalone:
     enabled: false
 

infrastructure/vps-standby/authentik/manifests/cnpg-cluster.yaml

@@ -68,13 +68,47 @@ spec:
 
   bootstrap:
     recovery:
-      source: home-backup
+      source: home
 
+  # Multi-site active failover pilot (see
+  # /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - the
+  # bootstrap.recovery above (barmanObjectStore) stays exactly as-is for
+  # the initial seed; connectionParameters below adds real streaming
+  # replication for ongoing sync, replacing the old WAL-archive-polling
+  # behavior (the source of the crash-looping/probe-tuning fights from
+  # earlier tonight - streaming doesn't need to restart postgres to check
+  # for new WAL).
+  #
+  # Auth: reuses CNPG's own generated streaming_replica client cert
+  # (CN=streaming_replica, issued by home's own pg-authentik CA) - copied
+  # here manually as pg-authentik-home-{ca,replication} (kubectl, not
+  # git). Same credential CNPG's own pg_hba.conf already trusts, nothing
+  # new to provision.
+  #
+  # Reachable via home.kube.huskypup.net:61432 - a UniFi WAN port-forward
+  # to talos-cp-01's real LAN IP (172.28.101.41), source-IP-restricted to
+  # just the VPS and witness public IPs. NOT the Netbird mesh directly -
+  # UniFi has no route to that CIDR, confirmed live (see
+  # infrastructure/ha-failover/manifests/etcd.yaml for the full story).
+  # externalClusters[].name is now "home" (was "home-backup") - reused
+  # consistently for bootstrap.recovery.source, replica.primary, AND the
+  # connectionParameters below, since CNPG's distributed-topology
+  # validation requires replica.primary to match an externalClusters name
+  # exactly (confirmed live: "External cluster home not found" when it
+  # didn't). serverName inside barmanObjectStore is unaffected by this
+  # rename - it's an explicit separate override, not derived from the
+  # entry name.
+  #
+  # Also needs a self-referencing "vps" entry below (same reason -
+  # replica.self must match an externalClusters name too, even though
+  # it's this very cluster) - its connectionParameters point at VPS's own
+  # local -rw service, never actually dialed while this cluster is a
+  # replica.
   externalClusters:
-    - name: home-backup
+    - name: home
       barmanObjectStore:
-        # serverName defaults to the externalClusters[].name ("home-backup")
+        # serverName defaults to the externalClusters[].name here, NOT
-        # here, NOT the source Postgres cluster's actual name - but home's
+        # the source Postgres cluster's actual name - but home's
         # barmanObjectStore backup: block (on the pg-authentik Cluster
         # itself) defaults serverName to its own metadata.name
         # ("pg-authentik"). Without this override, recovery looks for
@@ -91,10 +125,47 @@ spec:
           secretAccessKey:
             name: vps-minio-secret
             key: secretkey
+      connectionParameters:
+        host: home.kube.huskypup.net
+        port: "61432"
+        dbname: app
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-authentik-home-replication
+        key: tls.crt
+      sslKey:
+        name: pg-authentik-home-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-authentik-home-ca
+        key: ca.crt
+    - name: vps
+      connectionParameters:
+        host: pg-authentik-rw
+        port: "5432"
+        dbname: app
+        user: streaming_replica
+        sslmode: verify-ca
+      sslCert:
+        name: pg-authentik-replication
+        key: tls.crt
+      sslKey:
+        name: pg-authentik-replication
+        key: tls.key
+      sslRootCert:
+        name: pg-authentik-ca
+        key: ca.crt
 
+  # Distributed topology: both sides agree home is primary today. No
+  # replica.enabled - that flag belongs to CNPG's older single-source
+  # replica-cluster mode and is mutually exclusive with primary/self
+  # (confirmed live: "replica mode enabled is not compatible with the
+  # primary field").
   replica:
-    enabled: true
+    self: vps
-    source: home-backup
+    primary: home
+    source: home
 
   monitoring:
     enablePodMonitor: false  # no Prometheus on the VPS cluster

infrastructure/vps-standby/authentik/manifests/ha-postgres-nodeport.yaml

@@ -0,0 +1,27 @@
+---
+# Exposes the VPS's pg-authentik primary (currently a read-only replica -
+# see cnpg-cluster.yaml's replica.enabled) to home over the Netbird mesh,
+# for the multi-site active-failover pilot. Same selector as CNPG's own
+# generated pg-authentik-rw ClusterIP Service, just NodePort instead.
+#
+# Unlike home's side of this (infrastructure/authentik/manifests/
+# ha-postgres-nodeport.yaml), no UniFi port-forward or Kyverno
+# PolicyException needed here - the VPS's k3s has no NodePort
+# restriction, and home reaching OUT to the VPS already works today via
+# the netbird-egress DaemonSet's route (confirmed live all session, same
+# path used for MinIO/CrowdSec) - it's only the reverse direction
+# (external peers reaching INTO home) that needed the UniFi workaround.
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-authentik-postgres
+  namespace: authentik
+spec:
+  type: NodePort
+  selector:
+    cnpg.io/cluster: pg-authentik
+    cnpg.io/instanceRole: primary
+  ports:
+    - port: 5432
+      targetPort: 5432
+      nodePort: 32433

infrastructure/vps-standby/authentik/manifests/ingress.yaml

@@ -0,0 +1,34 @@
+---
+# See infrastructure/vps-standby/vault/manifests/ingress.yaml for the
+# vps.huskypup.net subdomain design rationale.
+#
+# CAVEAT: Authentik's provider/application configs (redirect_uris, etc.)
+# are replicated byte-for-byte from home and point at *.kube.huskypup.net
+# - SSO logins between VPS-hosted apps and this VPS Authentik instance
+# won't complete correctly (redirect mismatch) until that's addressed
+# separately. Direct/local Authentik admin login still works fine for
+# browsing and verifying replicated data.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: vps-authentik
+  namespace: authentik
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-production
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - auth.vps.huskypup.net
+      secretName: vps-authentik-tls
+  rules:
+    - host: auth.vps.huskypup.net
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: vps-authentik-server
+                port:
+                  number: 80

infrastructure/vps-standby/authentik/values.yaml

@@ -1,11 +1,16 @@
-# Authentik warm standby on the VPS - Phase 2. The CNPG replica cluster
+# Authentik warm standby on the VPS - Phase 2, later extended into the
-# (manifests/cnpg-cluster.yaml) continuously replays WAL from home in the
+# multi-site active failover pilot (see
-# background; the app itself is NOT run in normal operation (replicas: 0)
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md). The CNPG cluster
-# since the replica DB is read-only until a deliberate manual promotion -
+# (manifests/cnpg-cluster.yaml) now uses real streaming replication
-# an Authentik pod trying to write against a read-only DB would just
+# (externalClusters connectionParameters + spec.replica.self/primary/
-# crashloop uselessly. Promotion runbook: flip the CNPG cluster's
+# source) rather than the original WAL-archive-polling replica.enabled
-# spec.replica.enabled to false (see cnpg-cluster.yaml), then scale
+# mode - that flag is gone from cnpg-cluster.yaml now, it's mutually
-# server/worker up from 0.
+# exclusive with the primary/self fields. The app itself DOES run
+# continuously (replicas: 1, reachable at auth.vps.huskypup.net - see
+# manifests/ingress.yaml) so the replicated data is browsable/verifiable
+# at all times, even though the underlying DB is a read-only replica until
+# promoted. Promotion (automatic via the failover-watcher, or manually):
+# patch cnpg-cluster.yaml's spec.replica to self/primary/source: vps.
 #
 # The `authentik:` block below (secret_key/postgresql "env://" indirection)
 # is copied VERBATIM from infrastructure/authentik/values.yaml on purpose -
@@ -34,12 +39,14 @@ global:
       prefix: AUTHENTIK_POSTGRESQL__
   env:
     - name: AUTHENTIK_URL
-      value: "https://auth.kube.huskypup.net"
+      value: "https://auth.vps.huskypup.net"
+    # HOST/PORT point at the floating pg-authentik.ha.huskypup.net record
+    # instead of the local secret's host/port - see the matching comment
+    # in infrastructure/authentik/values.yaml (home's copy of this same
+    # block) for the full explanation, including why pg-authentik-app's
+    # password had to be manually synced between the two sites first.
     - name: AUTHENTIK_POSTGRESQL__HOST
-      valueFrom:
+      value: "pg-authentik.ha.huskypup.net"
-        secretKeyRef:
-          name: pg-authentik-app
-          key: host
     - name: AUTHENTIK_POSTGRESQL__NAME
       valueFrom:
         secretKeyRef:
@@ -56,10 +63,27 @@ global:
           name: pg-authentik-app
           key: password
     - name: AUTHENTIK_POSTGRESQL__PORT
-      value: "5432"
+      value: "61432"
 
 server:
-  replicas: 0
+  replicas: 1
+  # Default failureThreshold: 3 (40s runway) is too strict running against
+  # a read-only CNPG replica - the worker constantly retries a scheduled-
+  # task query that needs a write lock (harmless, expected, logged as
+  # 'cannot execute SELECT FOR UPDATE in a read-only transaction'), and
+  # any transient DB hiccup during that trips the server's health check
+  # and gets it killed (confirmed live: CrashLoopBackOff, 'connection
+  # refused' during a brief blip). Widened so brief hiccups don't kill it.
+  livenessProbe:
+    failureThreshold: 10
+    initialDelaySeconds: 10
+    periodSeconds: 10
+    timeoutSeconds: 5
+  readinessProbe:
+    failureThreshold: 10
+    initialDelaySeconds: 10
+    periodSeconds: 10
+    timeoutSeconds: 5
   containerSecurityContext:
     runAsNonRoot: true
     allowPrivilegeEscalation: false
@@ -78,7 +102,7 @@ server:
     enabled: false
 
 worker:
-  replicas: 0
+  replicas: 1
   containerSecurityContext:
     runAsNonRoot: true
     allowPrivilegeEscalation: false

infrastructure/vps-standby/cert-manager/manifests/cluster-issuer.yaml

@@ -0,0 +1,28 @@
+---
+# cert-manager itself is installed directly by scripts/vps-bootstrap.sh
+# (Phase 0, out-of-band like k3s/Netbird) - not GitOps-managed here. This
+# ClusterIssuer just rides the same GitOps pipeline as everything else in
+# vps-standby, DNS-01 via Cloudflare (same pattern as home's
+# infrastructure/cert-manager/manifests/letsencrypt-issuer.yaml).
+#
+# cloudflare-token-secret is a plain Secret copied here manually
+# (kubectl, not git - same reasoning as every other VPS secret):
+#   kubectl -n cert-manager create secret generic cloudflare-token-secret \
+#     --from-literal=cloudflare-token=<same token as home's cert-manager>
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-production
+spec:
+  acme:
+    server: https://acme-v02.api.letsencrypt.org/directory
+    email: garrettstone499@gmail.com
+    privateKeySecretRef:
+      name: letsencrypt-production
+    solvers:
+      - dns01:
+          cloudflare:
+            email: garrettstone499@gmail.com
+            apiTokenSecretRef:
+              name: cloudflare-token-secret
+              key: cloudflare-token

infrastructure/vps-standby/gitea/manifests/ingress.yaml

@@ -0,0 +1,27 @@
+---
+# See infrastructure/vps-standby/vault/manifests/ingress.yaml for the
+# vps.huskypup.net subdomain design rationale.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: vps-gitea
+  namespace: gitea
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-production
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - gitea.vps.huskypup.net
+      secretName: vps-gitea-tls
+  rules:
+    - host: gitea.vps.huskypup.net
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: vps-gitea-http
+                port:
+                  number: 3000

infrastructure/vps-standby/n8n/manifests/ingress.yaml

@@ -0,0 +1,27 @@
+---
+# See infrastructure/vps-standby/vault/manifests/ingress.yaml for the
+# vps.huskypup.net subdomain design rationale.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: vps-n8n
+  namespace: n8n
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-production
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - n8n.vps.huskypup.net
+      secretName: vps-n8n-tls
+  rules:
+    - host: n8n.vps.huskypup.net
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: vps-n8n
+                port:
+                  number: 80

infrastructure/vps-standby/n8n/manifests/pvc.yaml

@@ -0,0 +1,17 @@
+---
+# n8n's chart uses persistence.type: existing (values.yaml) - it expects
+# this PVC to already exist rather than creating one itself. At home this
+# PVC predates the chart deploy; on the VPS nothing ever created it since
+# n8n started at replicas: 0 - found live when scaling up to 1.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: n8n-main-persistence
+  namespace: n8n
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: local-path
+  resources:
+    requests:
+      storage: 10Gi

infrastructure/vps-standby/n8n/values.yaml

@@ -1,8 +1,11 @@
 # n8n warm standby on the VPS - Phase 2. Same discipline as
 # infrastructure/vps-standby/authentik/values.yaml: the CNPG replica
-# cluster keeps the DB continuously warm in the background, but the app
+# cluster keeps the DB continuously warm in the background, and the app
-# itself stays at replicaCount: 0 until a deliberate manual promotion
+# runs continuously too (replicaCount: 1, reachable at
-# (flip pg-n8n's spec.replica.enabled to false, then scale this up).
+# n8n.vps.huskypup.net - see manifests/ingress.yaml) - workflow
+# saves/executions will error against the read-only DB until a
+# deliberate manual promotion (flip pg-n8n's spec.replica.enabled to
+# false), but the UI and existing workflow definitions are browsable.
 #
 # N8N_ENCRYPTION_KEY must be byte-identical to home's - it decrypts
 # stored credentials (API keys, OAuth tokens, etc.) that live encrypted
@@ -28,7 +31,7 @@ config:
   generic:
     timezone: America/New_York
   path: /
-  host: n8n.kube.huskypup.net
+  host: n8n.vps.huskypup.net
   port: 5678
   protocol: https
   executions:
@@ -41,8 +44,7 @@ config:
 
 secret: {}
 
-# Not run in normal operation - see file header.
+replicaCount: 1
-replicaCount: 0
 
 service:
   type: ClusterIP
@@ -76,6 +78,30 @@ resources:
     cpu: "500m"
     memory: 512Mi
 
+# This chart has no startupProbe support at all (confirmed against
+# `helm show values` - only livenessProbe/readinessProbe exist), so the
+# liveness probe itself has to be generous enough to cover full startup -
+# even home's copied startupProbe timings (60s total) weren't enough on
+# the VPS's more modest hardware; confirmed live it needs 100s+.
+# initialDelaySeconds + (periodSeconds * failureThreshold) = 190s runway.
+readinessProbe:
+  httpGet:
+    path: /healthz
+    port: http
+  initialDelaySeconds: 30
+  periodSeconds: 5
+  timeoutSeconds: 3
+  failureThreshold: 20
+
+livenessProbe:
+  httpGet:
+    path: /healthz
+    port: http
+  initialDelaySeconds: 30
+  periodSeconds: 10
+  timeoutSeconds: 5
+  failureThreshold: 16
+
 extraEnvSecrets:
   DB_POSTGRESDB_PASSWORD:
     name: pg-n8n-app
@@ -85,8 +111,8 @@ extraEnvSecrets:
     key: encryption-key
 
 extraEnv:
-  WEBHOOK_URL: https://n8n.kube.huskypup.net/
+  WEBHOOK_URL: https://n8n.vps.huskypup.net/
-  N8N_EDITOR_BASE_URL: https://n8n.kube.huskypup.net
+  N8N_EDITOR_BASE_URL: https://n8n.vps.huskypup.net
   N8N_LOG_LEVEL: error
 
 postgresql:

infrastructure/vps-standby/nextcloud/manifests/ingress.yaml

@@ -0,0 +1,27 @@
+---
+# See infrastructure/vps-standby/vault/manifests/ingress.yaml for the
+# vps.huskypup.net subdomain design rationale.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: vps-nextcloud
+  namespace: nextcloud
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-production
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - nextcloud.vps.huskypup.net
+      secretName: vps-nextcloud-tls
+  rules:
+    - host: nextcloud.vps.huskypup.net
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: vps-nextcloud
+                port:
+                  number: 8080

infrastructure/vps-standby/nextcloud/values.yaml

@@ -1,19 +1,20 @@
 # Nextcloud warm standby on the VPS - Phase 2. Same discipline as the
 # other vps-standby apps: CNPG replica cluster (manifests/cnpg-cluster.yaml)
 # keeps the DB warm, manifests/pvc-restore-cronjob.yaml keeps file content
-# warm, but the app itself stays at replicaCount: 0 until a deliberate
+# warm, and the app runs continuously too (replicaCount: 1, reachable at
-# manual promotion.
+# nextcloud.vps.huskypup.net - see manifests/ingress.yaml) so replicated
+# files/users are browsable at all times - uploads/changes will error
+# against the read-only DB until a deliberate manual promotion.
 #
 # No Redis here - the home instance uses the redis-operator
 # (infrastructure/vps-standby has no redis-operator deployed, out of
-# scope for a standby that isn't actually serving traffic). Nextcloud
+# scope for a standby that isn't actually serving writable traffic).
-# runs fine without Redis (falls back to DB-based locking, just slower) -
+# Nextcloud runs fine without Redis (falls back to DB-based locking,
-# acceptable for an emergency-promotion scenario; add a real Redis at
+# just slower) - add a real Redis at promotion time if desired.
-# promotion time if desired.
+replicaCount: 1
-replicaCount: 0
 
 nextcloud:
-   host: nextcloud.kube.huskypup.net
+   host: nextcloud.vps.huskypup.net
    username: ""
    password: ""
 
@@ -29,6 +30,14 @@ nextcloud:
        value: "2G"
      - name: PHP_UPLOAD_LIMIT
        value: "10G"
+     # libpq (via PHP's pgsql/pdo_pgsql) defaults HOME=/root in this image
+     # but tries to look up a client cert at $HOME/.postgresql/postgresql.crt
+     # for higher sslmodes - "Permission denied" there aborts the whole
+     # connection outright (confirmed live: psql itself connects fine with
+     # the same creds, only PHP's driver hits this). CNPG's in-cluster
+     # Postgres connection doesn't need client-cert verification here.
+     - name: PGSSLMODE
+       value: "disable"
 
    configs:
       proxy.config.php: |-
@@ -39,7 +48,7 @@ nextcloud:
             1 => '172.16.0.0/12',
           ),
           'overwriteprotocol' => 'https',
-          'overwrite.cli.url' => 'https://nextcloud.kube.huskypup.net',
+          'overwrite.cli.url' => 'https://nextcloud.vps.huskypup.net',
           'allow_local_remote_servers' => true,
         );
       temp.config.php: |-

infrastructure/vps-standby/vault/manifests/ingress.yaml

@@ -0,0 +1,30 @@
+---
+# Public-ish access to the VPS Vault standby, replacing the old
+# "reachable over Netbird only" model with a stable, permanent hostname
+# (vault.vps.huskypup.net) that doesn't depend on home being up at all -
+# see the vps.huskypup.net subdomain design note in
+# infrastructure/vps-standby/cert-manager/manifests/cluster-issuer.yaml.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: vps-vault
+  namespace: vault
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-production
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - vault.vps.huskypup.net
+      secretName: vps-vault-tls
+  rules:
+    - host: vault.vps.huskypup.net
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: vps-vault-ui
+                port:
+                  number: 8200

scripts/ha-failback-authentik.sh

@@ -0,0 +1,68 @@
+#!/bin/bash
+# Multi-site active failover pilot (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - the deliberate
+# MANUAL counterpart to scripts/ha-failover-watcher.sh's automatic
+# promotion. Run this by hand, on the VPS, once you've confirmed home is
+# genuinely healthy again and want to move primary back - never automated,
+# by design (auto-flipping back immediately on reconnect risks flapping,
+# and a human should confirm home's data/state before handing writes back
+# to it).
+#
+# Order matters:
+#   1. Confirm home's pg-authentik Cluster is healthy and NOT still
+#      thinking it's primary (it shouldn't be, since it was down/
+#      unreachable when the VPS promoted - but check
+#      status.currentPrimary on home before proceeding).
+#   2. Re-point home's Cluster to replicate FROM the vps (it needs to
+#      catch up on everything written to the VPS while it was down)
+#      before flipping primary back - home has to actually BE a caught-up
+#      replica of vps first, or this loses the writes the VPS accepted
+#      during the outage.
+#   3. Only once home shows it's streaming and caught up, flip primary
+#      back to home and update DNS.
+#
+# This script only does step 3 (the fast, symmetric part - same shape as
+# the watcher's own promote() but in reverse). Steps 1-2 are a judgment
+# call requiring you to actually look at both clusters' status first -
+# not scripted here on purpose.
+set -euo pipefail
+
+echo "This will flip pg-authentik's primary back to home and repoint DNS."
+echo "Before continuing, you MUST have already confirmed:"
+echo "  - home's pg-authentik Cluster is healthy and replicating FROM vps"
+echo "    (kubectl -n authentik get cluster pg-authentik -o jsonpath='{.status}')"
+echo "  - home is caught up (no meaningful replication lag from vps)"
+read -r -p "Confirmed both of the above? [y/N] " ans
+if [ "${ans:-N}" != "y" ] && [ "${ans:-N}" != "Y" ]; then
+  echo "Aborted."
+  exit 1
+fi
+
+echo "Flipping home's Cluster CR to primary..."
+kubectl -n authentik patch cluster pg-authentik --type merge \
+  -p '{"spec":{"replica":{"self":"home","primary":"home","source":"home"}}}'
+
+echo "Flipping VPS's Cluster CR back to a replica of home..."
+ssh root@172.93.53.139 "kubectl -n authentik patch cluster pg-authentik --type merge \
+  -p '{\"spec\":{\"replica\":{\"self\":\"vps\",\"primary\":\"home\",\"source\":\"home\"}}}'"
+
+echo "Flipping pg-authentik.ha.huskypup.net back to home's IP..."
+HOME_PUBLIC_IP="$(dig +short home.kube.huskypup.net @1.1.1.1 | tail -1)"
+if [ -z "$HOME_PUBLIC_IP" ]; then
+  echo "ERROR: could not resolve home.kube.huskypup.net - fix the DNS record manually"
+  exit 1
+fi
+TOKEN="$(kubectl -n cert-manager get secret cloudflare-token-secret -o jsonpath='{.data.cloudflare-token}' | base64 -d)"
+ZONE_ID="$(curl -sf -H "Authorization: Bearer ${TOKEN}" "https://api.cloudflare.com/client/v4/zones?name=huskypup.net" | jq -r '.result[0].id')"
+RECORD_ID="$(curl -sf -H "Authorization: Bearer ${TOKEN}" "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records?name=pg-authentik.ha.huskypup.net&type=A" | jq -r '.result[0].id')"
+curl -sf -X PATCH -H "Authorization: Bearer ${TOKEN}" -H "Content-Type: application/json" \
+  -d "{\"type\":\"A\",\"name\":\"pg-authentik.ha.huskypup.net\",\"content\":\"${HOME_PUBLIC_IP}\",\"ttl\":60,\"proxied\":false}" \
+  "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records/${RECORD_ID}" >/dev/null
+
+echo "Clearing the etcd promotion record..."
+curl -sf --max-time 5 -X POST http://172.28.101.41:32379/v3/kv/deleterange \
+  -d "{\"key\":\"$(printf '%s' '/ha-failover/promoted-at' | base64)\"}" >/dev/null || true
+curl -sf --max-time 5 -X POST http://172.28.101.41:32379/v3/kv/deleterange \
+  -d "{\"key\":\"$(printf '%s' '/ha-failover/promoted-by' | base64)\"}" >/dev/null || true
+
+echo "Failback complete. home is primary again, pg-authentik.ha.huskypup.net -> ${HOME_PUBLIC_IP}"

scripts/ha-failover-watcher.service

@@ -0,0 +1,25 @@
+# Multi-site active failover pilot (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md). Install on the
+# VPS (172.93.53.139) alongside ha-failover-watcher.sh:
+#   scp scripts/ha-failover-watcher.sh root@172.93.53.139:/usr/local/bin/
+#   ssh root@172.93.53.139 chmod +x /usr/local/bin/ha-failover-watcher.sh
+#   mkdir -p /etc/ha-failover on the VPS, put the Cloudflare DNS-edit
+#     token (same one cert-manager's cloudflare-token-secret uses) in
+#     /etc/ha-failover/cloudflare-token, chmod 600, chown root
+#   scp this file to /etc/systemd/system/ha-failover-watcher.service
+#   systemctl daemon-reload && systemctl enable --now ha-failover-watcher
+#
+# Requires: curl, jq, kubectl, base64 - all already present on the VPS
+# from earlier in this session.
+[Unit]
+Description=HA failover watcher - promotes pg-authentik to primary if home's etcd heartbeat goes stale (majority-confirmed via etcd quorum)
+After=network.target k3s.service
+
+[Service]
+ExecStart=/usr/local/bin/ha-failover-watcher.sh
+Restart=always
+RestartSec=5
+User=root
+
+[Install]
+WantedBy=multi-user.target

scripts/ha-failover-watcher.sh

@@ -0,0 +1,154 @@
+#!/bin/bash
+# Multi-site active failover pilot (see
+# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - the VPS's half
+# of the failover-watcher. Runs as a systemd service on the VPS
+# (172.93.53.139), NOT in k3s - it needs to keep running even if the VPS's
+# own k3s/CNPG cluster is unhealthy, and it's the one thing in this whole
+# pilot that's genuinely new/bespoke rather than reusing an existing
+# operator.
+#
+# Deploy (manual, matches every other VPS systemd unit this session - not
+# git-applied automatically):
+#   scp this file to the VPS as /usr/local/bin/ha-failover-watcher.sh
+#   chmod +x it, then install scripts/ha-failover-watcher.service
+#   (see that file) and `systemctl enable --now ha-failover-watcher`.
+#
+# Logic (deliberately simple - see the plan doc's "what CNPG genuinely
+# does NOT provide" paragraph for why this exists at all):
+#   1. Every 10s, read /ha-failover/home-heartbeat from etcd via THIS
+#      node's own local etcd member (127.0.0.1:2379) using a linearizable
+#      (default, quorum-backed) read - home's heartbeat-writer
+#      (infrastructure/ha-failover/manifests/heartbeat-writer.yaml)
+#      refreshes this key every 10s while home is healthy.
+#   2. If the read itself fails/times out, this VPS can't reach a
+#      majority of the 3-member etcd cluster (needs 2 of 3) - meaning
+#      EITHER home is genuinely down AND the witness is also unreachable
+#      from here, OR this VPS itself is the one that's partitioned.
+#      Either way, we cannot safely tell which, so we do NOT promote -
+#      this is the split-brain-prevention property etcd's own Raft
+#      consensus gives us for free, no custom quorum-counting needed.
+#   3. If the read succeeds and the heartbeat is fresher than
+#      STALE_THRESHOLD seconds, home is confirmed up - no-op.
+#   4. If the read succeeds (so we DO have majority/quorum) and the
+#      heartbeat is older than STALE_THRESHOLD - or missing entirely -
+#      home is confirmed down by majority agreement. Promote, once:
+#        a. Skip if already primary (checked via the Cluster CR itself,
+#           idempotent - safe to run this loop forever).
+#        b. kubectl patch the local pg-authentik Cluster:
+#           spec.replica.{self,primary,source} = vps. No promotionToken -
+#           confirmed live via dry-run that CNPG's admission webhook does
+#           NOT require one (it's optional, used for graceful/planned
+#           switchover to cross-check LSNs - not available for a genuine
+#           unplanned outage since home isn't reachable to generate one).
+#           This means promotion accepts whatever the VPS replica had
+#           already streamed - typically a couple seconds of async lag,
+#           an accepted tradeoff of async cross-WAN replication (there is
+#           no realistic sync-replication option over a home/VPS WAN
+#           link without crippling write latency).
+#        c. Flip the pg-authentik.ha.huskypup.net Cloudflare A record to
+#           this VPS's public IP - both sites listen on the SAME external
+#           port 61432 (home via UniFi WAN forward, VPS via
+#           pg-authentik-forward.service's local socat forward)
+#           specifically so a single floating hostname:port works for
+#           both sites without the app tier needing per-site config.
+#        d. Record the promotion in etcd (/ha-failover/promoted-at,
+#           /ha-failover/promoted-by) - both for the idempotency check
+#           above surviving a script restart, and as an audit trail for
+#           whoever does the (deliberately manual - see the plan doc)
+#           failback later.
+#
+# Failback is NOT automated by this script on purpose - see
+# scripts/ha-failback-authentik.sh, run by a human once home is
+# confirmed healthy again.
+set -u
+
+ETCD="http://127.0.0.1:2379"
+STALE_THRESHOLD=45  # ~4-5 missed 10s heartbeats before acting - avoids flapping on one blip
+CHECK_INTERVAL=10
+CLOUDFLARE_TOKEN_FILE="/etc/ha-failover/cloudflare-token"
+CF_ZONE_NAME="huskypup.net"
+CF_RECORD_NAME="pg-authentik.ha.huskypup.net"
+VPS_PUBLIC_IP="172.93.53.139"
+LOG_TAG="ha-failover-watcher"
+
+log() { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) ${LOG_TAG}: $*"; }
+
+b64() { printf '%s' "$1" | base64 | tr -d '\n'; }
+
+etcd_get() {
+  # $1 = key. Prints the decoded value, or nothing + returns 1 if the
+  # read failed (unreachable/no quorum) or the key doesn't exist.
+  local key_b64 resp val_b64
+  key_b64="$(b64 "$1")"
+  resp="$(curl -sf --max-time 5 -X POST "${ETCD}/v3/kv/range" \
+    -d "{\"key\":\"${key_b64}\"}")" || return 1
+  val_b64="$(echo "$resp" | jq -r '.kvs[0].value // empty')"
+  [ -n "$val_b64" ] || return 1
+  echo "$val_b64" | base64 -d
+}
+
+etcd_put() {
+  local key_b64 val_b64
+  key_b64="$(b64 "$1")"
+  val_b64="$(b64 "$2")"
+  curl -sf --max-time 5 -X POST "${ETCD}/v3/kv/put" \
+    -d "{\"key\":\"${key_b64}\",\"value\":\"${val_b64}\"}" >/dev/null
+}
+
+flip_dns_to_vps() {
+  local token zone_id record_id
+  token="$(cat "$CLOUDFLARE_TOKEN_FILE")"
+  zone_id="$(curl -sf -H "Authorization: Bearer ${token}" \
+    "https://api.cloudflare.com/client/v4/zones?name=${CF_ZONE_NAME}" | jq -r '.result[0].id')"
+  if [ -z "$zone_id" ] || [ "$zone_id" = "null" ]; then
+    log "ERROR: could not resolve Cloudflare zone id for ${CF_ZONE_NAME}"
+    return 1
+  fi
+  record_id="$(curl -sf -H "Authorization: Bearer ${token}" \
+    "https://api.cloudflare.com/client/v4/zones/${zone_id}/dns_records?name=${CF_RECORD_NAME}&type=A" \
+    | jq -r '.result[0].id // empty')"
+  if [ -z "$record_id" ]; then
+    log "ERROR: no existing A record for ${CF_RECORD_NAME} - refusing to create one blind, fix manually"
+    return 1
+  fi
+  curl -sf -X PATCH -H "Authorization: Bearer ${token}" -H "Content-Type: application/json" \
+    -d "{\"type\":\"A\",\"name\":\"${CF_RECORD_NAME}\",\"content\":\"${VPS_PUBLIC_IP}\",\"ttl\":60,\"proxied\":false}" \
+    "https://api.cloudflare.com/client/v4/zones/${zone_id}/dns_records/${record_id}" >/dev/null
+}
+
+promote() {
+  local current_primary
+  current_primary="$(kubectl -n authentik get cluster pg-authentik -o jsonpath='{.spec.replica.primary}' 2>/dev/null)"
+  if [ "$current_primary" = "vps" ]; then
+    return 0  # already promoted, nothing to do
+  fi
+  log "PROMOTING: home confirmed down by etcd majority (heartbeat stale/missing). Flipping pg-authentik to vps."
+  if ! kubectl -n authentik patch cluster pg-authentik --type merge \
+      -p '{"spec":{"replica":{"self":"vps","primary":"vps","source":"vps"}}}'; then
+    log "ERROR: kubectl patch failed - Cluster CR NOT promoted, will retry next loop"
+    return 1
+  fi
+  if ! flip_dns_to_vps; then
+    log "ERROR: Cluster CR promoted but Cloudflare DNS flip failed - fix pg-authentik.ha.huskypup.net manually, it still points at home"
+  fi
+  etcd_put "/ha-failover/promoted-at" "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
+  etcd_put "/ha-failover/promoted-by" "vps"
+  log "Promotion complete."
+}
+
+log "starting, stale threshold=${STALE_THRESHOLD}s check interval=${CHECK_INTERVAL}s"
+while true; do
+  hb="$(etcd_get /ha-failover/home-heartbeat)"
+  if [ $? -ne 0 ]; then
+    log "cannot reach etcd quorum (or no heartbeat key yet) - not acting, will retry"
+    sleep "$CHECK_INTERVAL"
+    continue
+  fi
+  now="$(date +%s)"
+  age=$((now - hb))
+  if [ "$age" -gt "$STALE_THRESHOLD" ]; then
+    log "home heartbeat is ${age}s stale (threshold ${STALE_THRESHOLD}s)"
+    promote
+  fi
+  sleep "$CHECK_INTERVAL"
+done

scripts/vps-dns-failover/vps-dns-failover.service

@@ -1,10 +0,0 @@
-[Unit]
-Description=VPS dual-site DNS failover check (Homelabv4 vps-standby)
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=oneshot
-ExecStart=/usr/local/bin/vps-dns-failover.sh
-# Deliberately no dependency on k3s/docker being up - this must keep working
-# even if the VPS's own cluster is unhealthy.

scripts/vps-dns-failover/vps-dns-failover.sh

@@ -1,132 +0,0 @@
-#!/usr/bin/env bash
-# vps-dns-failover.sh - Phase 0.5 dual-site DNS failover watcher
-#
-# Runs ON THE VPS as a systemd timer (see vps-dns-failover.timer/.service in this
-# directory) - deliberately plain systemd, not a k8s CronJob, so it works even if
-# the VPS's own k3s is unhealthy. It must never depend on anything inside the home
-# cluster (that's the thing it's watching) or the VPS's own k3s (that's a separate
-# failure domain from this box's basic OS-level networking).
-#
-# Health-checks home.kube.huskypup.net (kept current by an in-cluster CronJob while
-# home is up - see infrastructure/cert-manager/manifests/home-ip-ddns-cronjob.yaml)
-# and flips Cloudflare A records for the standby-service hostnames between home's
-# public IP and this VPS's own public IP, with a consecutive-check threshold so a
-# single blip doesn't cause a flap.
-#
-# State (current active site + streak counters) persists in $STATE_DIR between
-# runs since each systemd timer firing is a fresh process.
-#
-# Install:
-#   sudo mkdir -p /etc/vps-dns-failover
-#   sudo sh -c 'echo "<dns-edit-scoped-cloudflare-token>" > /etc/vps-dns-failover/cloudflare-token'
-#   sudo chmod 600 /etc/vps-dns-failover/cloudflare-token
-#   sudo cp vps-dns-failover.sh /usr/local/bin/vps-dns-failover.sh
-#   sudo chmod +x /usr/local/bin/vps-dns-failover.sh
-#   sudo cp vps-dns-failover.service vps-dns-failover.timer /etc/systemd/system/
-#   sudo systemctl daemon-reload
-#   sudo systemctl enable --now vps-dns-failover.timer
-
-set -euo pipefail
-
-TOKEN_FILE="/etc/vps-dns-failover/cloudflare-token"
-STATE_DIR="/var/lib/vps-dns-failover"
-ZONE_NAME="huskypup.net"  # Cloudflare zone is the parent domain - kube.huskypup.net is just a record within it, not its own zone
-HOME_CHECK_HOST="home.kube.huskypup.net"
-HOME_CHECK_PORT=443
-FAILURE_THRESHOLD=3   # consecutive failed checks before flipping to the VPS
-SUCCESS_THRESHOLD=3   # consecutive successful checks before flipping back to home
-STANDBY_HOSTNAMES=(
-  vault.kube.huskypup.net
-  auth.kube.huskypup.net
-  gitea.kube.huskypup.net
-  n8n.kube.huskypup.net
-  nextcloud.kube.huskypup.net
-)
-
-mkdir -p "$STATE_DIR"
-TOKEN="$(cat "$TOKEN_FILE")"
-STATE_FILE="${STATE_DIR}/state"   # format: "<active-site> <fail-streak> <success-streak>"
-
-if [ -f "$STATE_FILE" ]; then
-  read -r ACTIVE FAIL_STREAK SUCCESS_STREAK < "$STATE_FILE"
-else
-  ACTIVE="home"
-  FAIL_STREAK=0
-  SUCCESS_STREAK=0
-fi
-
-# --- health check ------------------------------------------------------------
-HOME_IP="$(dig +short A "$HOME_CHECK_HOST" @1.1.1.1 | tail -n1)"
-if [ -n "$HOME_IP" ] && timeout 5 bash -c "cat < /dev/null > /dev/tcp/${HOME_IP}/${HOME_CHECK_PORT}" 2>/dev/null; then
-  HEALTHY=1
-else
-  HEALTHY=0
-fi
-
-if [ "$HEALTHY" = 1 ]; then
-  FAIL_STREAK=0
-  SUCCESS_STREAK=$((SUCCESS_STREAK + 1))
-else
-  SUCCESS_STREAK=0
-  FAIL_STREAK=$((FAIL_STREAK + 1))
-fi
-
-echo "$(date -u +%FT%TZ) active=${ACTIVE} healthy=${HEALTHY} fail_streak=${FAIL_STREAK} success_streak=${SUCCESS_STREAK} home_ip=${HOME_IP:-none}"
-
-# --- Cloudflare helpers --------------------------------------------------------
-cf_zone_id() {
-  curl -sf -H "Authorization: Bearer ${TOKEN}" \
-    "https://api.cloudflare.com/client/v4/zones?name=${ZONE_NAME}" | jq -r '.result[0].id'
-}
-
-cf_set_record() {
-  local zone_id="$1" hostname="$2" target_ip="$3"
-  local record_json record_id
-  record_json="$(curl -sf -H "Authorization: Bearer ${TOKEN}" \
-    "https://api.cloudflare.com/client/v4/zones/${zone_id}/dns_records?name=${hostname}&type=A")"
-  record_id="$(echo "$record_json" | jq -r '.result[0].id // empty')"
-  local body="{\"type\":\"A\",\"name\":\"${hostname}\",\"content\":\"${target_ip}\",\"ttl\":60,\"proxied\":false}"
-  if [ -n "$record_id" ]; then
-    curl -sf -X PATCH -H "Authorization: Bearer ${TOKEN}" -H "Content-Type: application/json" \
-      -d "$body" "https://api.cloudflare.com/client/v4/zones/${zone_id}/dns_records/${record_id}" >/dev/null
-  else
-    curl -sf -X POST -H "Authorization: Bearer ${TOKEN}" -H "Content-Type: application/json" \
-      -d "$body" "https://api.cloudflare.com/client/v4/zones/${zone_id}/dns_records" >/dev/null
-  fi
-  echo "  ${hostname} -> ${target_ip}"
-}
-
-flip_to() {
-  local target="$1"
-  local target_ip
-  if [ "$target" = "vps" ]; then
-    target_ip="$(curl -sf https://cloudflare.com/cdn-cgi/trace | grep -o '^ip=.*' | cut -d= -f2)"
-  else
-    target_ip="$HOME_IP"
-  fi
-  if [ -z "$target_ip" ]; then
-    echo "ERROR: could not determine target IP for '${target}', not flipping"
-    return 1
-  fi
-  echo "Flipping standby hostnames to ${target} (${target_ip})..."
-  local zone_id
-  zone_id="$(cf_zone_id)"
-  for h in "${STANDBY_HOSTNAMES[@]}"; do
-    cf_set_record "$zone_id" "$h" "$target_ip"
-  done
-}
-
-# --- decide ------------------------------------------------------------------
-if [ "$ACTIVE" = "home" ] && [ "$FAIL_STREAK" -ge "$FAILURE_THRESHOLD" ]; then
-  flip_to "vps"
-  ACTIVE="vps"
-  FAIL_STREAK=0
-  SUCCESS_STREAK=0
-elif [ "$ACTIVE" = "vps" ] && [ "$SUCCESS_STREAK" -ge "$SUCCESS_THRESHOLD" ]; then
-  flip_to "home"
-  ACTIVE="home"
-  FAIL_STREAK=0
-  SUCCESS_STREAK=0
-fi
-
-echo "${ACTIVE} ${FAIL_STREAK} ${SUCCESS_STREAK}" > "$STATE_FILE"

scripts/vps-dns-failover/vps-dns-failover.timer

@@ -1,10 +0,0 @@
-[Unit]
-Description=Run vps-dns-failover check every 2 minutes
-
-[Timer]
-OnBootSec=1min
-OnUnitActiveSec=2min
-AccuracySec=10s
-
-[Install]
-WantedBy=timers.target