Multi-site active failover pilot: home etcd member + Postgres NodePort exposure

Part of the Authentik HA pilot (see plan doc). Home's etcd quorum member (StatefulSet, pinned to talos-cp-01 for a stable advertised address) plus a NodePort exposing pg-authentik's current primary - both reachable from the VPS/witness over the already-authenticated Netbird mesh (confirmed live: home nodes are directly reachable from Netbird peers on their real node IP, via the netbird-egress DaemonSet's route). Deliberately NOT going through UniFi/public-internet exposure - this stays entirely inside the private mesh, a materially safer path than the WAN port- forward originally considered. Needs a scoped Kyverno PolicyException (ha-failover-nodeport-exception.yaml) since disallow-nodeport-services is enforced cluster-wide - narrowly scoped to Services named ha-*, matching the existing netbird-egress- exemption.yaml precedent for exceptions.
Widen VPS Authentik server liveness/readiness probes
2026-08-20 23:16:49 +00:00 · 2026-08-18 21:12:45 -05:00 · 2026-08-18 19:01:01 -05:00 · 2026-08-18 18:51:50 -05:00 · 2026-08-18 18:40:52 -05:00 · 2026-08-18 18:24:30 -05:00
21 changed files with 530 additions and 178 deletions
@@ -0,0 +1,24 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: ha-failover
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "23"  # after CNPG/storage (10-14), alongside other security/HA infra
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: infrastructure
+  source:
+    repoURL: https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
+    targetRevision: main
+    path: infrastructure/ha-failover/manifests
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: ha-failover
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
@@ -0,0 +1,22 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: vps-cert-manager-issuer
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "1"  # before the per-service Ingress resources (wave 3+) that reference it
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: vps-standby
+  source:
+    repoURL: https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
+    targetRevision: main
+    path: infrastructure/vps-standby/cert-manager/manifests
+  destination:
+    name: vps-standby
+    namespace: cert-manager
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
@@ -0,0 +1,22 @@
+---
+# Exposes pg-authentik's current primary to the VPS/witness over the
+# Netbird mesh for streaming replication (see the "Multi-Site Active
+# Failover" plan) - same selector CNPG's own pg-authentik-rw ClusterIP
+# Service uses, just NodePort instead, since CNPG doesn't manage this
+# object and won't fight with it. Needs
+# infrastructure/kyverno/policies/ha-failover-nodeport-exception.yaml
+# (disallow-nodeport-services is enforced cluster-wide otherwise).
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-authentik-postgres
+  namespace: authentik
+spec:
+  type: NodePort
+  selector:
+    cnpg.io/cluster: pg-authentik
+    cnpg.io/instanceRole: primary
+  ports:
+    - port: 5432
+      targetPort: 5432
+      nodePort: 32432
@@ -61,6 +61,7 @@ blueprints:
    - authentik-blueprints-percona-everest
    - authentik-blueprints-rancher
    - authentik-blueprints-netbird
+    - authentik-blueprints-vault

 # Enable Prometheus metrics
 server:
@@ -0,0 +1,51 @@
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: authentik-blueprints-vault
+  namespace: authentik
+  labels:
+    goauthentik.io/blueprint: "true"
+data:
+  vault.yaml: |-
+    version: 1
+    metadata:
+      name: vault-oidc
+    entries:
+      - model: authentik_providers_oauth2.oauth2provider
+        id: vault-provider
+        state: present
+        identifiers:
+          name: Vault
+        attrs:
+          name: Vault
+          client_id: 9816a5ae7e7914b5d18f4ab939d011a98f8c8d6b3bb6777c46431afa06ac4a85
+          client_secret: ed2ba1c6378c7a46341b5162f39a7fab80e37596b01ed387c3719e8e0040344cf1daa307476c2e7a7f75041b3979275b1ebf00bb8bad94c864b4a38ded544f7b
+          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+          authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
+          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+          redirect_uris:
+            - url: "https://vault.kube.huskypup.net/ui/vault/auth/oidc/oidc/callback"
+              matching_mode: strict
+            - url: "http://localhost:8250/oidc/callback"
+              matching_mode: strict
+          property_mappings:
+            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, openid]]
+            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, email]]
+            - !Find [authentik_providers_oauth2.scopemapping, [scope_name, profile]]
+          client_type: confidential
+          access_code_validity: "minutes=10"
+          access_token_validity: "hours=1"
+          refresh_token_validity: "days=30"
+          signing_key: !Find [authentik_crypto.certificatekeypair, [name, "authentik Internal JWT Certificate"]]
+
+      - model: authentik_core.application
+        id: vault-application
+        state: present
+        identifiers:
+          slug: vault
+        attrs:
+          name: Vault
+          slug: vault
+          policy_engine_mode: any
+          provider: !KeyOf vault-provider
@@ -0,0 +1,106 @@
+---
+# Home's member of the 3-way etcd quorum used by the failover-watcher
+# (see /home/scooby/.claude/plans/jiggly-snacking-iverson.md, "Multi-Site
+# Active Failover" section) to safely decide when to trigger a CNPG
+# distributed-topology promotion. VPS and witness run this same etcd
+# version as plain systemd services (simpler - no k3s/Kyverno involved
+# there); home has to run it as a real K8s workload since Talos doesn't
+# allow bare-metal process installs.
+#
+# Pinned to a specific node (talos-cp-01) via nodeSelector so its
+# advertised peer/client URLs (which must be stable, not "whichever node
+# it landed on today") stay correct. Reachable from the VPS/witness via
+# that node's real IP - confirmed live 2026-08-19 that home nodes are
+# directly reachable from Netbird mesh peers on their node IP (the
+# netbird-egress DaemonSet's route) - and exposed via a NodePort Service,
+# which needs infrastructure/kyverno/policies/ha-failover-nodeport-exception.yaml
+# (disallow-nodeport-services is enforced cluster-wide otherwise).
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: ha-failover
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: ha-etcd
+  namespace: ha-failover
+spec:
+  serviceName: ha-etcd
+  replicas: 1
+  selector:
+    matchLabels:
+      app: ha-etcd
+  template:
+    metadata:
+      labels:
+        app: ha-etcd
+    spec:
+      nodeSelector:
+        kubernetes.io/hostname: talos-cp-01
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        fsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: etcd
+          image: gcr.io/etcd-development/etcd:v3.5.17
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+          command:
+            - etcd
+            - --name=home
+            - --data-dir=/var/lib/etcd/data
+            - --listen-client-urls=http://0.0.0.0:2379
+            - --advertise-client-urls=http://100.108.42.109:32379
+            - --listen-peer-urls=http://0.0.0.0:2380
+            - --initial-advertise-peer-urls=http://100.108.42.109:32380
+            - --initial-cluster=home=http://100.108.42.109:32380,vps=http://100.108.113.41:2380,witness=http://100.108.130.74:2380
+            - --initial-cluster-state=new
+            - --initial-cluster-token=ha-failover-quorum
+          ports:
+            - containerPort: 2379
+              name: client
+            - containerPort: 2380
+              name: peer
+          volumeMounts:
+            - name: data
+              mountPath: /var/lib/etcd
+          resources:
+            requests:
+              cpu: 25m
+              memory: 64Mi
+            limits:
+              memory: 256Mi
+  volumeClaimTemplates:
+    - metadata:
+        name: data
+      spec:
+        accessModes: ["ReadWriteOnce"]
+        storageClassName: rook-ceph-block
+        resources:
+          requests:
+            storage: 1Gi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: ha-etcd
+  namespace: ha-failover
+spec:
+  type: NodePort
+  selector:
+    app: ha-etcd
+  ports:
+    - name: client
+      port: 2379
+      targetPort: 2379
+      nodePort: 32379
+    - name: peer
+      port: 2380
+      targetPort: 2380
+      nodePort: 32380
@@ -0,0 +1,31 @@
+# Scoped exception to the disallow-nodeport-services policy, narrowly for
+# the multi-site active-failover pilot's cross-site services (etcd quorum,
+# Postgres streaming replication) - not a namespace-wide exclusion.
+#
+# Why NodePort is genuinely needed here: home nodes are directly reachable
+# from Netbird mesh peers (VPS, witness) on their real node IP - confirmed
+# live 2026-08-19 (ping succeeded from the VPS to a node's InternalIP,
+# which is itself a Netbird-mesh address via the netbird-egress DaemonSet's
+# route). A NodePort Service binds on that same real interface on every
+# node, giving the VPS/witness a way to reach it directly over the already-
+# authenticated Netbird mesh - no public internet exposure, no UniFi
+# port-forward, no new WAN-facing attack surface. LoadBalancer (MetalLB)
+# only gets a LAN-side VIP, which isn't reachable from Netbird peers at all
+# without the same underlying NodePort-style exposure anyway.
+apiVersion: kyverno.io/v2
+kind: PolicyException
+metadata:
+  name: ha-failover-nodeport-exemption
+  namespace: kyverno
+spec:
+  exceptions:
+    - policyName: disallow-nodeport-services
+      ruleNames:
+        - disallow-nodeport
+  match:
+    any:
+      - resources:
+          kinds:
+            - Service
+          names:
+            - ha-*
@@ -21,6 +21,11 @@ server:
          disable_hostname = true
        }

+  hostAliases:
+    - ip: "172.28.101.244"
+      hostnames:
+        - auth.kube.huskypup.net
+
  standalone:
    enabled: false

@@ -0,0 +1,34 @@
+---
+# See infrastructure/vps-standby/vault/manifests/ingress.yaml for the
+# vps.huskypup.net subdomain design rationale.
+#
+# CAVEAT: Authentik's provider/application configs (redirect_uris, etc.)
+# are replicated byte-for-byte from home and point at *.kube.huskypup.net
+# - SSO logins between VPS-hosted apps and this VPS Authentik instance
+# won't complete correctly (redirect mismatch) until that's addressed
+# separately. Direct/local Authentik admin login still works fine for
+# browsing and verifying replicated data.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: vps-authentik
+  namespace: authentik
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-production
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - auth.vps.huskypup.net
+      secretName: vps-authentik-tls
+  rules:
+    - host: auth.vps.huskypup.net
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: vps-authentik-server
+                port:
+                  number: 80
@@ -1,11 +1,11 @@
 # Authentik warm standby on the VPS - Phase 2. The CNPG replica cluster
 # (manifests/cnpg-cluster.yaml) continuously replays WAL from home in the
-# background; the app itself is NOT run in normal operation (replicas: 0)
-# since the replica DB is read-only until a deliberate manual promotion -
-# an Authentik pod trying to write against a read-only DB would just
-# crashloop uselessly. Promotion runbook: flip the CNPG cluster's
-# spec.replica.enabled to false (see cnpg-cluster.yaml), then scale
-# server/worker up from 0.
+# background. The app itself DOES run continuously (replicas: 1, reachable
+# at auth.vps.huskypup.net - see manifests/ingress.yaml) so the replicated
+# data is browsable/verifiable at all times, even though the underlying DB
+# is a read-only CNPG replica - writes (new logins, session creation) will
+# error until a deliberate manual promotion. Promotion runbook: flip the
+# CNPG cluster's spec.replica.enabled to false (see cnpg-cluster.yaml).
 #
 # The `authentik:` block below (secret_key/postgresql "env://" indirection)
 # is copied VERBATIM from infrastructure/authentik/values.yaml on purpose -
@@ -34,7 +34,7 @@ global:
      prefix: AUTHENTIK_POSTGRESQL__
  env:
    - name: AUTHENTIK_URL
-      value: "https://auth.kube.huskypup.net"
+      value: "https://auth.vps.huskypup.net"
    - name: AUTHENTIK_POSTGRESQL__HOST
      valueFrom:
        secretKeyRef:
@@ -59,7 +59,24 @@ global:
      value: "5432"

 server:
-  replicas: 0
+  replicas: 1
+  # Default failureThreshold: 3 (40s runway) is too strict running against
+  # a read-only CNPG replica - the worker constantly retries a scheduled-
+  # task query that needs a write lock (harmless, expected, logged as
+  # 'cannot execute SELECT FOR UPDATE in a read-only transaction'), and
+  # any transient DB hiccup during that trips the server's health check
+  # and gets it killed (confirmed live: CrashLoopBackOff, 'connection
+  # refused' during a brief blip). Widened so brief hiccups don't kill it.
+  livenessProbe:
+    failureThreshold: 10
+    initialDelaySeconds: 10
+    periodSeconds: 10
+    timeoutSeconds: 5
+  readinessProbe:
+    failureThreshold: 10
+    initialDelaySeconds: 10
+    periodSeconds: 10
+    timeoutSeconds: 5
  containerSecurityContext:
    runAsNonRoot: true
    allowPrivilegeEscalation: false
@@ -78,7 +95,7 @@ server:
    enabled: false

 worker:
-  replicas: 0
+  replicas: 1
  containerSecurityContext:
    runAsNonRoot: true
    allowPrivilegeEscalation: false
@@ -0,0 +1,28 @@
+---
+# cert-manager itself is installed directly by scripts/vps-bootstrap.sh
+# (Phase 0, out-of-band like k3s/Netbird) - not GitOps-managed here. This
+# ClusterIssuer just rides the same GitOps pipeline as everything else in
+# vps-standby, DNS-01 via Cloudflare (same pattern as home's
+# infrastructure/cert-manager/manifests/letsencrypt-issuer.yaml).
+#
+# cloudflare-token-secret is a plain Secret copied here manually
+# (kubectl, not git - same reasoning as every other VPS secret):
+#   kubectl -n cert-manager create secret generic cloudflare-token-secret \
+#     --from-literal=cloudflare-token=<same token as home's cert-manager>
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-production
+spec:
+  acme:
+    server: https://acme-v02.api.letsencrypt.org/directory
+    email: garrettstone499@gmail.com
+    privateKeySecretRef:
+      name: letsencrypt-production
+    solvers:
+      - dns01:
+          cloudflare:
+            email: garrettstone499@gmail.com
+            apiTokenSecretRef:
+              name: cloudflare-token-secret
+              key: cloudflare-token
@@ -0,0 +1,27 @@
+---
+# See infrastructure/vps-standby/vault/manifests/ingress.yaml for the
+# vps.huskypup.net subdomain design rationale.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: vps-gitea
+  namespace: gitea
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-production
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - gitea.vps.huskypup.net
+      secretName: vps-gitea-tls
+  rules:
+    - host: gitea.vps.huskypup.net
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: vps-gitea-http
+                port:
+                  number: 3000
@@ -0,0 +1,27 @@
+---
+# See infrastructure/vps-standby/vault/manifests/ingress.yaml for the
+# vps.huskypup.net subdomain design rationale.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: vps-n8n
+  namespace: n8n
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-production
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - n8n.vps.huskypup.net
+      secretName: vps-n8n-tls
+  rules:
+    - host: n8n.vps.huskypup.net
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: vps-n8n
+                port:
+                  number: 80
@@ -0,0 +1,17 @@
+---
+# n8n's chart uses persistence.type: existing (values.yaml) - it expects
+# this PVC to already exist rather than creating one itself. At home this
+# PVC predates the chart deploy; on the VPS nothing ever created it since
+# n8n started at replicas: 0 - found live when scaling up to 1.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: n8n-main-persistence
+  namespace: n8n
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: local-path
+  resources:
+    requests:
+      storage: 10Gi
@@ -1,8 +1,11 @@
 # n8n warm standby on the VPS - Phase 2. Same discipline as
 # infrastructure/vps-standby/authentik/values.yaml: the CNPG replica
-# cluster keeps the DB continuously warm in the background, but the app
-# itself stays at replicaCount: 0 until a deliberate manual promotion
-# (flip pg-n8n's spec.replica.enabled to false, then scale this up).
+# cluster keeps the DB continuously warm in the background, and the app
+# runs continuously too (replicaCount: 1, reachable at
+# n8n.vps.huskypup.net - see manifests/ingress.yaml) - workflow
+# saves/executions will error against the read-only DB until a
+# deliberate manual promotion (flip pg-n8n's spec.replica.enabled to
+# false), but the UI and existing workflow definitions are browsable.
 #
 # N8N_ENCRYPTION_KEY must be byte-identical to home's - it decrypts
 # stored credentials (API keys, OAuth tokens, etc.) that live encrypted
@@ -28,7 +31,7 @@ config:
  generic:
    timezone: America/New_York
  path: /
-  host: n8n.kube.huskypup.net
+  host: n8n.vps.huskypup.net
  port: 5678
  protocol: https
  executions:
@@ -41,8 +44,7 @@ config:

 secret: {}

-# Not run in normal operation - see file header.
-replicaCount: 0
+replicaCount: 1

 service:
  type: ClusterIP
@@ -76,6 +78,30 @@ resources:
    cpu: "500m"
    memory: 512Mi

+# This chart has no startupProbe support at all (confirmed against
+# `helm show values` - only livenessProbe/readinessProbe exist), so the
+# liveness probe itself has to be generous enough to cover full startup -
+# even home's copied startupProbe timings (60s total) weren't enough on
+# the VPS's more modest hardware; confirmed live it needs 100s+.
+# initialDelaySeconds + (periodSeconds * failureThreshold) = 190s runway.
+readinessProbe:
+  httpGet:
+    path: /healthz
+    port: http
+  initialDelaySeconds: 30
+  periodSeconds: 5
+  timeoutSeconds: 3
+  failureThreshold: 20
+
+livenessProbe:
+  httpGet:
+    path: /healthz
+    port: http
+  initialDelaySeconds: 30
+  periodSeconds: 10
+  timeoutSeconds: 5
+  failureThreshold: 16
+
 extraEnvSecrets:
  DB_POSTGRESDB_PASSWORD:
    name: pg-n8n-app
@@ -85,8 +111,8 @@ extraEnvSecrets:
    key: encryption-key

 extraEnv:
-  WEBHOOK_URL: https://n8n.kube.huskypup.net/
-  N8N_EDITOR_BASE_URL: https://n8n.kube.huskypup.net
+  WEBHOOK_URL: https://n8n.vps.huskypup.net/
+  N8N_EDITOR_BASE_URL: https://n8n.vps.huskypup.net
  N8N_LOG_LEVEL: error

 postgresql:
@@ -0,0 +1,27 @@
+---
+# See infrastructure/vps-standby/vault/manifests/ingress.yaml for the
+# vps.huskypup.net subdomain design rationale.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: vps-nextcloud
+  namespace: nextcloud
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-production
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - nextcloud.vps.huskypup.net
+      secretName: vps-nextcloud-tls
+  rules:
+    - host: nextcloud.vps.huskypup.net
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: vps-nextcloud
+                port:
+                  number: 8080
@@ -1,19 +1,20 @@
 # Nextcloud warm standby on the VPS - Phase 2. Same discipline as the
 # other vps-standby apps: CNPG replica cluster (manifests/cnpg-cluster.yaml)
 # keeps the DB warm, manifests/pvc-restore-cronjob.yaml keeps file content
-# warm, but the app itself stays at replicaCount: 0 until a deliberate
-# manual promotion.
+# warm, and the app runs continuously too (replicaCount: 1, reachable at
+# nextcloud.vps.huskypup.net - see manifests/ingress.yaml) so replicated
+# files/users are browsable at all times - uploads/changes will error
+# against the read-only DB until a deliberate manual promotion.
 #
 # No Redis here - the home instance uses the redis-operator
 # (infrastructure/vps-standby has no redis-operator deployed, out of
-# scope for a standby that isn't actually serving traffic). Nextcloud
-# runs fine without Redis (falls back to DB-based locking, just slower) -
-# acceptable for an emergency-promotion scenario; add a real Redis at
-# promotion time if desired.
-replicaCount: 0
+# scope for a standby that isn't actually serving writable traffic).
+# Nextcloud runs fine without Redis (falls back to DB-based locking,
+# just slower) - add a real Redis at promotion time if desired.
+replicaCount: 1

 nextcloud:
-   host: nextcloud.kube.huskypup.net
+   host: nextcloud.vps.huskypup.net
   username: ""
   password: ""

@@ -29,6 +30,14 @@ nextcloud:
       value: "2G"
     - name: PHP_UPLOAD_LIMIT
       value: "10G"
+     # libpq (via PHP's pgsql/pdo_pgsql) defaults HOME=/root in this image
+     # but tries to look up a client cert at $HOME/.postgresql/postgresql.crt
+     # for higher sslmodes - "Permission denied" there aborts the whole
+     # connection outright (confirmed live: psql itself connects fine with
+     # the same creds, only PHP's driver hits this). CNPG's in-cluster
+     # Postgres connection doesn't need client-cert verification here.
+     - name: PGSSLMODE
+       value: "disable"

   configs:
      proxy.config.php: |-
@@ -39,7 +48,7 @@ nextcloud:
            1 => '172.16.0.0/12',
          ),
          'overwriteprotocol' => 'https',
-          'overwrite.cli.url' => 'https://nextcloud.kube.huskypup.net',
+          'overwrite.cli.url' => 'https://nextcloud.vps.huskypup.net',
          'allow_local_remote_servers' => true,
        );
      temp.config.php: |-
@@ -0,0 +1,30 @@
+---
+# Public-ish access to the VPS Vault standby, replacing the old
+# "reachable over Netbird only" model with a stable, permanent hostname
+# (vault.vps.huskypup.net) that doesn't depend on home being up at all -
+# see the vps.huskypup.net subdomain design note in
+# infrastructure/vps-standby/cert-manager/manifests/cluster-issuer.yaml.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: vps-vault
+  namespace: vault
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-production
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - vault.vps.huskypup.net
+      secretName: vps-vault-tls
+  rules:
+    - host: vault.vps.huskypup.net
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: vps-vault-ui
+                port:
+                  number: 8200
@@ -1,10 +0,0 @@
-[Unit]
-Description=VPS dual-site DNS failover check (Homelabv4 vps-standby)
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=oneshot
-ExecStart=/usr/local/bin/vps-dns-failover.sh
-# Deliberately no dependency on k3s/docker being up - this must keep working
-# even if the VPS's own cluster is unhealthy.
@@ -1,132 +0,0 @@
-#!/usr/bin/env bash
-# vps-dns-failover.sh - Phase 0.5 dual-site DNS failover watcher
-#
-# Runs ON THE VPS as a systemd timer (see vps-dns-failover.timer/.service in this
-# directory) - deliberately plain systemd, not a k8s CronJob, so it works even if
-# the VPS's own k3s is unhealthy. It must never depend on anything inside the home
-# cluster (that's the thing it's watching) or the VPS's own k3s (that's a separate
-# failure domain from this box's basic OS-level networking).
-#
-# Health-checks home.kube.huskypup.net (kept current by an in-cluster CronJob while
-# home is up - see infrastructure/cert-manager/manifests/home-ip-ddns-cronjob.yaml)
-# and flips Cloudflare A records for the standby-service hostnames between home's
-# public IP and this VPS's own public IP, with a consecutive-check threshold so a
-# single blip doesn't cause a flap.
-#
-# State (current active site + streak counters) persists in $STATE_DIR between
-# runs since each systemd timer firing is a fresh process.
-#
-# Install:
-#   sudo mkdir -p /etc/vps-dns-failover
-#   sudo sh -c 'echo "<dns-edit-scoped-cloudflare-token>" > /etc/vps-dns-failover/cloudflare-token'
-#   sudo chmod 600 /etc/vps-dns-failover/cloudflare-token
-#   sudo cp vps-dns-failover.sh /usr/local/bin/vps-dns-failover.sh
-#   sudo chmod +x /usr/local/bin/vps-dns-failover.sh
-#   sudo cp vps-dns-failover.service vps-dns-failover.timer /etc/systemd/system/
-#   sudo systemctl daemon-reload
-#   sudo systemctl enable --now vps-dns-failover.timer
-
-set -euo pipefail
-
-TOKEN_FILE="/etc/vps-dns-failover/cloudflare-token"
-STATE_DIR="/var/lib/vps-dns-failover"
-ZONE_NAME="huskypup.net"  # Cloudflare zone is the parent domain - kube.huskypup.net is just a record within it, not its own zone
-HOME_CHECK_HOST="home.kube.huskypup.net"
-HOME_CHECK_PORT=443
-FAILURE_THRESHOLD=3   # consecutive failed checks before flipping to the VPS
-SUCCESS_THRESHOLD=3   # consecutive successful checks before flipping back to home
-STANDBY_HOSTNAMES=(
-  vault.kube.huskypup.net
-  auth.kube.huskypup.net
-  gitea.kube.huskypup.net
-  n8n.kube.huskypup.net
-  nextcloud.kube.huskypup.net
-)
-
-mkdir -p "$STATE_DIR"
-TOKEN="$(cat "$TOKEN_FILE")"
-STATE_FILE="${STATE_DIR}/state"   # format: "<active-site> <fail-streak> <success-streak>"
-
-if [ -f "$STATE_FILE" ]; then
-  read -r ACTIVE FAIL_STREAK SUCCESS_STREAK < "$STATE_FILE"
-else
-  ACTIVE="home"
-  FAIL_STREAK=0
-  SUCCESS_STREAK=0
-fi
-
-# --- health check ------------------------------------------------------------
-HOME_IP="$(dig +short A "$HOME_CHECK_HOST" @1.1.1.1 | tail -n1)"
-if [ -n "$HOME_IP" ] && timeout 5 bash -c "cat < /dev/null > /dev/tcp/${HOME_IP}/${HOME_CHECK_PORT}" 2>/dev/null; then
-  HEALTHY=1
-else
-  HEALTHY=0
-fi
-
-if [ "$HEALTHY" = 1 ]; then
-  FAIL_STREAK=0
-  SUCCESS_STREAK=$((SUCCESS_STREAK + 1))
-else
-  SUCCESS_STREAK=0
-  FAIL_STREAK=$((FAIL_STREAK + 1))
-fi
-
-echo "$(date -u +%FT%TZ) active=${ACTIVE} healthy=${HEALTHY} fail_streak=${FAIL_STREAK} success_streak=${SUCCESS_STREAK} home_ip=${HOME_IP:-none}"
-
-# --- Cloudflare helpers --------------------------------------------------------
-cf_zone_id() {
-  curl -sf -H "Authorization: Bearer ${TOKEN}" \
-    "https://api.cloudflare.com/client/v4/zones?name=${ZONE_NAME}" | jq -r '.result[0].id'
-}
-
-cf_set_record() {
-  local zone_id="$1" hostname="$2" target_ip="$3"
-  local record_json record_id
-  record_json="$(curl -sf -H "Authorization: Bearer ${TOKEN}" \
-    "https://api.cloudflare.com/client/v4/zones/${zone_id}/dns_records?name=${hostname}&type=A")"
-  record_id="$(echo "$record_json" | jq -r '.result[0].id // empty')"
-  local body="{\"type\":\"A\",\"name\":\"${hostname}\",\"content\":\"${target_ip}\",\"ttl\":60,\"proxied\":false}"
-  if [ -n "$record_id" ]; then
-    curl -sf -X PATCH -H "Authorization: Bearer ${TOKEN}" -H "Content-Type: application/json" \
-      -d "$body" "https://api.cloudflare.com/client/v4/zones/${zone_id}/dns_records/${record_id}" >/dev/null
-  else
-    curl -sf -X POST -H "Authorization: Bearer ${TOKEN}" -H "Content-Type: application/json" \
-      -d "$body" "https://api.cloudflare.com/client/v4/zones/${zone_id}/dns_records" >/dev/null
-  fi
-  echo "  ${hostname} -> ${target_ip}"
-}
-
-flip_to() {
-  local target="$1"
-  local target_ip
-  if [ "$target" = "vps" ]; then
-    target_ip="$(curl -sf https://cloudflare.com/cdn-cgi/trace | grep -o '^ip=.*' | cut -d= -f2)"
-  else
-    target_ip="$HOME_IP"
-  fi
-  if [ -z "$target_ip" ]; then
-    echo "ERROR: could not determine target IP for '${target}', not flipping"
-    return 1
-  fi
-  echo "Flipping standby hostnames to ${target} (${target_ip})..."
-  local zone_id
-  zone_id="$(cf_zone_id)"
-  for h in "${STANDBY_HOSTNAMES[@]}"; do
-    cf_set_record "$zone_id" "$h" "$target_ip"
-  done
-}
-
-# --- decide ------------------------------------------------------------------
-if [ "$ACTIVE" = "home" ] && [ "$FAIL_STREAK" -ge "$FAILURE_THRESHOLD" ]; then
-  flip_to "vps"
-  ACTIVE="vps"
-  FAIL_STREAK=0
-  SUCCESS_STREAK=0
-elif [ "$ACTIVE" = "vps" ] && [ "$SUCCESS_STREAK" -ge "$SUCCESS_THRESHOLD" ]; then
-  flip_to "home"
-  ACTIVE="home"
-  FAIL_STREAK=0
-  SUCCESS_STREAK=0
-fi
-
-echo "${ACTIVE} ${FAIL_STREAK} ${SUCCESS_STREAK}" > "$STATE_FILE"
@@ -1,10 +0,0 @@
-[Unit]
-Description=Run vps-dns-failover check every 2 minutes
-
-[Timer]
-OnBootSec=1min
-OnUnitActiveSec=2min
-AccuracySec=10s
-
-[Install]
-WantedBy=timers.target
Author	SHA1	Message	Date
Scooby Husky	5f52f2b36b	Multi-site active failover pilot: home etcd member + Postgres NodePort exposure Part of the Authentik HA pilot (see plan doc). Home's etcd quorum member (StatefulSet, pinned to talos-cp-01 for a stable advertised address) plus a NodePort exposing pg-authentik's current primary - both reachable from the VPS/witness over the already-authenticated Netbird mesh (confirmed live: home nodes are directly reachable from Netbird peers on their real node IP, via the netbird-egress DaemonSet's route). Deliberately NOT going through UniFi/public-internet exposure - this stays entirely inside the private mesh, a materially safer path than the WAN port- forward originally considered. Needs a scoped Kyverno PolicyException (ha-failover-nodeport-exception.yaml) since disallow-nodeport-services is enforced cluster-wide - narrowly scoped to Services named ha-*, matching the existing netbird-egress- exemption.yaml precedent for exceptions.	2026-08-18 21:12:45 -05:00
Scooby Husky	9b45cfd542	Widen VPS Authentik server liveness/readiness probes Confirmed live: CrashLoopBackOff (Bad Gateway from the ingress) caused by the default 40s-runway liveness probe tripping on a transient DB connection blip. The worker constantly retries a scheduled-task query needing a write lock against the read-only CNPG replica (expected, harmless - 'cannot execute SELECT FOR UPDATE in a read-only transaction') which appears to add enough connection pressure that a brief hiccup trips the server's stricter default probe.	2026-08-18 19:01:01 -05:00
Scooby Husky	8e33cf0524	Fix n8n liveness probe: chart has no startupProbe, needs a longer runway The n8n chart (riatlas/chart__n8n) doesn't support startupProbe at all - my earlier fix set a field the chart ignores, so the liveness probe was still killing the container ~40s into startup (exitCode 143, confirmed live via repeated crashloops even after the first 'fix'). n8n takes longer than that to bind :5678 on the VPS's more modest hardware. Widened readiness/liveness directly instead: ~190s total runway before a liveness kill.	2026-08-18 18:51:50 -05:00
Scooby Husky	08b8209455	Fix VPS Nextcloud DB connection (PGSSLMODE) and n8n probe timing Nextcloud: PHP's postgres driver tries to look up a client cert at $HOME/.postgresql/postgresql.crt (HOME=/root in this image) for verify-ca/verify-full sslmodes - 'Permission denied' there aborts the connection entirely with a confusing 'password authentication failed' secondary error. Confirmed live: psql with the identical credentials connects fine (uses sslmode=prefer, no cert lookup). PGSSLMODE=disable sidesteps it - this is an intra-cluster connection, not worth TLS here. n8n: chart's default liveness/readiness probe timing is tighter than n8n needs to actually finish starting - kubelet was killing the container (exitCode 143/SIGTERM) before it ever bound :5678, in an endless crashloop. Copied home's more generous probe timing.	2026-08-18 18:40:52 -05:00
Scooby Husky	444cb127bd	Add missing n8n-main-persistence PVC on VPS n8n's chart uses persistence.type: existing, which expects this PVC to already exist - nothing ever created it since n8n started at replicas: 0 and is only now being scaled up to 1 (confirmed live: pod stuck Pending, 'persistentvolumeclaim n8n-main-persistence not found').	2026-08-18 18:24:30 -05:00
Scooby Husky	5bc1be2f00	Replace DNS-flip failover watcher with static vps.huskypup.net subdomains The DNS-flip watcher (scripts/vps-dns-failover/) was designed but never actually installed on the VPS despite being tracked as done - real gap, found when asked whether the standby services are actually reachable. New design: instead of dynamically flipping *.kube.huskypup.net between home and VPS IPs, give the VPS site its own permanent, always-resolving subdomain - vault/gitea/auth/n8n/nextcloud.vps.huskypup.net, each with real Ingress+TLS on the VPS's own Traefik+cert-manager (both already installed by Phase 0 bootstrap, just never wired up). No token-scoping decision needed since there's no dynamic flipping - reuses the same cert-manager token pattern as home. Also scales Authentik/n8n/Nextcloud from 0 to 1 replica on the VPS so the replicated data is actually browsable at all times, not just present-but-unreachable. Their CNPG clusters are still read-only replicas (spec.replica.enabled: true) - writes will error until a deliberate manual promotion, but reads/browsing work now. Vault and Gitea were already running continuously.	2026-08-18 18:23:08 -05:00
Scooby Husky	f98c997293	Vault OIDC login via Authentik: hostAlias + blueprint config Vault's OIDC auth method needs to resolve auth.kube.huskypup.net (the issuer URL) from inside its own pod - no in-cluster DNS entry exists for that public hostname, so add a hostAlias pointing at the live istio-ingressgateway LoadBalancer IP (verified current: 172.28.101.244). authentik-blueprints-vault (vault-blueprint.yaml) was already applied to the cluster and referenced by the authentik Deployment for some time - committing it now so git matches the live, working state instead of leaving it as an untracked local file.	2026-08-18 18:07:36 -05:00