Builds the one genuinely new component from the plan - everything else
reuses CNPG/etcd as-is.
- infrastructure/ha-failover/manifests/heartbeat-writer.yaml: home writes
a fresh timestamp to etcd (/ha-failover/home-heartbeat) every 10s via
its local etcd member. No custom quorum/voting logic - etcd's own Raft
consensus (majority write/linearizable read) already gives the 'a
majority agrees' guarantee.
- scripts/ha-failover-watcher.sh + .service: runs on the VPS as a
systemd service (not k3s - must survive the VPS's own cluster being
unhealthy). Reads the heartbeat via its own local etcd member; if the
read itself fails, we can't tell if home is down or if this VPS is the
one partitioned, so it does NOT act (etcd's consensus requirement
provides the split-brain safety here, not custom code). If the read
succeeds and the heartbeat is stale (>45s, ~4-5 missed beats), home is
confirmed down by majority - promotes pg-authentik's Cluster CR
(spec.replica self/primary/source -> vps, no promotionToken - confirmed
live via dry-run that CNPG's webhook doesn't require one, it's only for
planned/graceful switchover LSN cross-checks that aren't available
during a genuine unplanned outage) and flips the
pg-authentik.ha.huskypup.net Cloudflare A record to the VPS's IP.
- scripts/ha-failback-authentik.sh: the deliberately manual reverse -
human confirms home is healthy and caught up before running this.
- Floating hostname plumbing: both sites' Authentik values now read
AUTHENTIK_POSTGRESQL__HOST/PORT as pg-authentik.ha.huskypup.net:61432
instead of the local pg-authentik-app secret's host/port, so app pods
on either site always reach whichever site is actually primary. Both
sites listen on the same external port (VPS gets a new
pg-authentik-forward.service socat forward, mirroring the existing
minio-forward.service pattern, so its NodePort 32433 is externally
reachable on 61432 same as home's UniFi-forwarded port).
- Found and worked around two real bugs surfaced while wiring this up
(fixed manually via scripts handed to the user - both blocked by the
Claude Code auto-mode classifier as credential-transmission /
cluster-DNS-edit actions):
- VPS's pg-authentik-app secret had a STALE password from before
streaming replication existed - the live Postgres role password now
replicates from home via WAL, but VPS's local K8s secret never got
updated to match. Needs a one-time sync (and again on any future
rotation).
- This UniFi does not support NAT hairpin/loopback for its own WAN
port-forwards - home's own pods resolving the floating hostname need
a local CoreDNS rewrite straight to pg-authentik-rw, confirmed live by
a DNS-resolves-but-TCP-connect-fails test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed live against the vcluster.cnpg.io admission webhook: (1)
replica.enabled is mutually exclusive with primary/self - they're two
different mechanisms (old single-source replica-cluster mode vs the
newer distributed-topology model), not composable. (2) Both replica.self
and replica.primary must reference names present in THAT cluster's own
externalClusters list - including a self-referencing entry, even though
it describes the cluster itself. Renamed VPS's 'home-backup' external
cluster to 'home' throughout (bootstrap.recovery.source too) so one name
consistently satisfies barmanObjectStore, connectionParameters, and the
primary reference; added matching self-referencing entries on both
sides.
Adds connectionParameters (real streaming, not just barmanObjectStore
WAL-archive polling) to both sides' existing externalClusters entry, plus
replica.self/primary fields declaring the distributed topology (both
currently agree home is primary). Auth reuses CNPG's own auto-generated
streaming_replica client certs, cross-copied between clusters manually
(kubectl, not git - same pattern as every other cross-cluster secret
tonight) since they're the exact credential each side's pg_hba.conf
already trusts.
Also adds the VPS-side NodePort exposing its own pg-authentik primary
back to home (home->VPS already works via the existing netbird-egress
route, no UniFi port-forward needed for that direction - only the
reverse, VPS/witness reaching into home, needed the WAN workaround).
Confirmed live: CrashLoopBackOff (Bad Gateway from the ingress) caused
by the default 40s-runway liveness probe tripping on a transient DB
connection blip. The worker constantly retries a scheduled-task query
needing a write lock against the read-only CNPG replica (expected,
harmless - 'cannot execute SELECT FOR UPDATE in a read-only
transaction') which appears to add enough connection pressure that a
brief hiccup trips the server's stricter default probe.
The n8n chart (riatlas/chart__n8n) doesn't support startupProbe at all -
my earlier fix set a field the chart ignores, so the liveness probe was
still killing the container ~40s into startup (exitCode 143, confirmed
live via repeated crashloops even after the first 'fix'). n8n takes
longer than that to bind :5678 on the VPS's more modest hardware.
Widened readiness/liveness directly instead: ~190s total runway before
a liveness kill.
Nextcloud: PHP's postgres driver tries to look up a client cert at
$HOME/.postgresql/postgresql.crt (HOME=/root in this image) for
verify-ca/verify-full sslmodes - 'Permission denied' there aborts the
connection entirely with a confusing 'password authentication failed'
secondary error. Confirmed live: psql with the identical credentials
connects fine (uses sslmode=prefer, no cert lookup). PGSSLMODE=disable
sidesteps it - this is an intra-cluster connection, not worth TLS here.
n8n: chart's default liveness/readiness probe timing is tighter than
n8n needs to actually finish starting - kubelet was killing the
container (exitCode 143/SIGTERM) before it ever bound :5678, in an
endless crashloop. Copied home's more generous probe timing.
n8n's chart uses persistence.type: existing, which expects this PVC to
already exist - nothing ever created it since n8n started at replicas: 0
and is only now being scaled up to 1 (confirmed live: pod stuck Pending,
'persistentvolumeclaim n8n-main-persistence not found').
The DNS-flip watcher (scripts/vps-dns-failover/) was designed but never
actually installed on the VPS despite being tracked as done - real gap,
found when asked whether the standby services are actually reachable.
New design: instead of dynamically flipping *.kube.huskypup.net between
home and VPS IPs, give the VPS site its own permanent, always-resolving
subdomain - vault/gitea/auth/n8n/nextcloud.vps.huskypup.net, each with
real Ingress+TLS on the VPS's own Traefik+cert-manager (both already
installed by Phase 0 bootstrap, just never wired up). No token-scoping
decision needed since there's no dynamic flipping - reuses the same
cert-manager token pattern as home.
Also scales Authentik/n8n/Nextcloud from 0 to 1 replica on the VPS so
the replicated data is actually browsable at all times, not just
present-but-unreachable. Their CNPG clusters are still read-only
replicas (spec.replica.enabled: true) - writes will error until a
deliberate manual promotion, but reads/browsing work now. Vault and
Gitea were already running continuously.
Postgres refuses to replay WAL past a checkpoint recorded with higher
max_connections/max_wal_senders/etc than the recovering instance's own
settings: 'recovery aborted because of insufficient parameter settings:
max_connections = 100 is a lower setting than on the primary server,
where its value was 200' - confirmed live on all 3 VPS clusters (they
had no postgresql.parameters block at all, defaulting to CNPG's 100).
Copied home's full postgresql.parameters block to remove any other
potential mismatch (max_wal_senders, max_worker_processes are subject
to the same check). Bumped memory requests/limits to match home too -
shared_buffers: 512MB needs headroom the previous 256Mi/1Gi didn't have.
CNPG's barmanObjectStore.serverName defaults to the externalClusters[]
entry's own .name field ('home-backup', an arbitrary label I chose) -
NOT the actual source Postgres cluster's name. Home's backup: block
(on the real pg-authentik/pg-n8n/pg-nextcloud Clusters) defaults
serverName to its own metadata.name instead. Without an explicit
override these two disagree, so recovery searches the object store
under the wrong server-name prefix and finds nothing - 'no target
backup found', confirmed live even after the ScheduledBackup fix
produced real, completed base backups.
Without hook annotations they land in ArgoCD's regular Sync phase, which
runs AFTER PreSync hooks - so the presync Job's pod couldn't be created
('serviceaccount not found', confirmed live). Weight -1 vs the Job's 0
gets them created first, within the same PreSync phase.
Same CNPG replica-cluster pattern as Authentik (continuous WAL replay
from home via VPS MinIO, app at 0 replicas until manual promotion - see
infrastructure/vps-standby/authentik/manifests/cnpg-cluster.yaml for the
full rationale).
n8n: N8N_ENCRYPTION_KEY copied byte-identical from home (kubectl, not
git) - decrypts stored credentials in the replicated DB, same reasoning
as Vault's unseal key / root token copies.
Nextcloud: adds infrastructure/vps-standby/nextcloud/manifests/
pvc-restore-cronjob.yaml, the read side of home's existing
nextcloud-pvc-sync restic backup - restores the latest snapshot from VPS
MinIO into this standby's PVC daily at 04:00 (2h after home's 02:00
backup). No Redis on the VPS side (no redis-operator deployed there,
out of scope for a standby that isn't serving traffic - Nextcloud
degrades gracefully to DB-based locking without it).
CNPG operator on the VPS (argocd-apps/vps-standby/cnpg-operator.yaml),
plus a CNPG 'replica cluster' for pg-authentik (infrastructure/
vps-standby/authentik/manifests/cnpg-cluster.yaml) that continuously
replays WAL shipped from home's pg-authentik via VPS MinIO's
cnpg-backups/pg-authentik bucket path - CNPG's native replica-cluster
mode, not a custom restore script like Vault needed (Vault has no
equivalent built-in continuous-replication-into-object-store feature).
The Authentik app itself runs at replicas: 0 in normal operation - the
replica DB is read-only until a deliberate manual promotion
(spec.replica.enabled: false), and a pod trying to write against a
read-only DB would just crashloop uselessly. values.yaml's authentik:
block is copied byte-identical from home's on purpose: the chart
deterministically derives its generated Secret's AUTHENTIK_SECRET_KEY
from these literal values.yaml strings, so both clusters land on the
same key without manually copying it - required since that key decrypts
things stored encrypted in the replicated DB.
service=gitlab invokes Gitea's GitLab-API-based downloader (for issues/
PRs/releases metadata), which calls the source's /api/v4/projects/...
REST API rather than doing a plain git clone. That was 404ing against
home GitLab and getting swallowed into a generic 'InternalServerError:
404 Not Found' with no further detail. We only want a plain pull-mirror
of the git repo itself, so service=git forces the plain git-clone
downloader instead.
Pull-mirrors home GitLab's Homelabv4 repo on Gitea's own built-in mirror
scheduler (6h interval) - no custom sync job needed, per the original
plan. SQLite instead of the chart's default HA Postgres + Valkey cluster
(single-instance standby holding one small repo, not worth the extra
moving parts). A PostSync Job creates the mirror once, idempotently;
Gitea's scheduler handles all ongoing pulls after that.
Also added a public Cloudflare CNAME for gitlab.kube.huskypup.net ->
home.kube.huskypup.net: the VPS has no route to home's LAN via Netbird
(none of the mesh peers advertise that subnet, confirmed live), so
GitLab needs to be reachable the same way any other internet client
reaches it - home's public IP already has port 443 forwarded to
istio-ingressgateway from earlier this session.
Admin and GitLab-mirror credentials are plain Secrets created directly
on the VPS cluster (kubectl, not git) - same pattern as vault-unseal-key.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed end-to-end tonight: condition=Ready correctly times out every
cycle since Vault can't be Ready while sealed (the unseal step comes
right after this wait) - harmless via the existing || true fallback, but
wastes up to 2 minutes per restore cycle waiting on a condition that can
never be met at this point. Poll for the container process merely being
started instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Without it, kubectl wait's internal watch retry loop never terminates
cleanly even with --timeout set - confirmed live, it kept retrying well
past the stated 120s timeout. The restore/unseal steps themselves still
succeed regardless (verified end-to-end with a manual unseal), this only
affects the job's own graceful detection of the pod coming back.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
raft snapshot restore is a privileged operation - unsealing alone isn't
enough, it needs an authenticated token. Missed this on first pass, caught
live: 'Code: 403. Errors: * permission denied'.
First run: uses the throwaway init's own fresh root token. Steady state
(already restored at least once): uses a copy of home's real root token,
which becomes valid on this Vault the moment the first restore completes
(its auth data becomes byte-for-byte home's as of that snapshot). Stored
the same way as the unseal key - kubectl directly on the VPS, not git.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
StatefulSet pod naming follows the Helm release name (vps-vault, matching
the ArgoCD Application name), not the chart's default.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Single-node Vault (raft storage, standalone mode) plus a restore CronJob
that pulls the latest raft snapshot from the VPS's own local MinIO
(already receiving snapshots every 6h from home) and restores it every
6h, offset 30 minutes after the home-side snapshot job.
Not an independent root of trust: raft snapshot restore replaces the
entire storage backend including the keyring, so after every restore
this Vault is sealed with HOME's actual keyring - unsealing it needs a
copy of home's real unseal key, stored directly on the VPS cluster
(kubectl, not git, same pattern as vps-minio-root-secret). First-run
bootstrap uses a throwaway single-Shamir-key init just to get through
the very first restore, then is irrelevant from then on.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
rootUser/rootPassword were left unset, relying on the chart's
auto-generation - but ArgoCD renders Helm via 'helm template', which
doesn't support the lookup() function charts use to preserve an existing
generated secret across upgrades. With selfHeal: true, every single sync
generated a BRAND NEW random root password, immediately invalidating
whatever credentials were stored in the home cluster's Vault for the
backup/CronJob consumers (vault-raft-snapshot, 3x CNPG barmanObjectStore,
nextcloud-pvc-sync) - confirmed live: all of them failing with 'Access
Key Id does not exist' right after being fixed, because the password had
already rotated out from under them again.
Fixed by pointing at a pre-existing Secret created directly on the VPS
cluster (kubectl, not git - this cluster has no Vault/ESO of its own).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Foundation for a DR/backup path using an always-on VPS as a second
ArgoCD-managed cluster, plus DB/backup standardization work that fell
out of it:
- vps-standby ArgoCD cluster destination + AppProject, MinIO backup
receiver, VPS bootstrap script (k3s, Netbird, cert-manager)
- Dual-site DNS failover watcher + home-IP DDNS CronJob, Cloudflare
token moved out of git into Vault+ExternalSecret
- Nextcloud migrated from ad-hoc MariaDB to CNPG + redis-operator
(matches n8n/Authentik/GitLab's backup-native pattern)
- Authentik's CNPG manifests moved into the actual ArgoCD-synced
manifests/ path (were present but never wired into the sync path)
- Vault raft-snapshot CronJob, CNPG barmanObjectStore backups
(Authentik/n8n/Nextcloud), Nextcloud file-PVC restic sync - all
targeting the new VPS MinIO receiver
See VPS Warm-Standby plan doc for full design rationale.
