mirror of
https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
synced 2026-08-21 05:26:49 +00:00
Scooby HuskyandClaude Sonnet 5
f7a9a03816
Builds the one genuinely new component from the plan - everything else
reuses CNPG/etcd as-is.
- infrastructure/ha-failover/manifests/heartbeat-writer.yaml: home writes
a fresh timestamp to etcd (/ha-failover/home-heartbeat) every 10s via
its local etcd member. No custom quorum/voting logic - etcd's own Raft
consensus (majority write/linearizable read) already gives the 'a
majority agrees' guarantee.
- scripts/ha-failover-watcher.sh + .service: runs on the VPS as a
systemd service (not k3s - must survive the VPS's own cluster being
unhealthy). Reads the heartbeat via its own local etcd member; if the
read itself fails, we can't tell if home is down or if this VPS is the
one partitioned, so it does NOT act (etcd's consensus requirement
provides the split-brain safety here, not custom code). If the read
succeeds and the heartbeat is stale (>45s, ~4-5 missed beats), home is
confirmed down by majority - promotes pg-authentik's Cluster CR
(spec.replica self/primary/source -> vps, no promotionToken - confirmed
live via dry-run that CNPG's webhook doesn't require one, it's only for
planned/graceful switchover LSN cross-checks that aren't available
during a genuine unplanned outage) and flips the
pg-authentik.ha.huskypup.net Cloudflare A record to the VPS's IP.
- scripts/ha-failback-authentik.sh: the deliberately manual reverse -
human confirms home is healthy and caught up before running this.
- Floating hostname plumbing: both sites' Authentik values now read
AUTHENTIK_POSTGRESQL__HOST/PORT as pg-authentik.ha.huskypup.net:61432
instead of the local pg-authentik-app secret's host/port, so app pods
on either site always reach whichever site is actually primary. Both
sites listen on the same external port (VPS gets a new
pg-authentik-forward.service socat forward, mirroring the existing
minio-forward.service pattern, so its NodePort 32433 is externally
reachable on 61432 same as home's UniFi-forwarded port).
- Found and worked around two real bugs surfaced while wiring this up
(fixed manually via scripts handed to the user - both blocked by the
Claude Code auto-mode classifier as credential-transmission /
cluster-DNS-edit actions):
- VPS's pg-authentik-app secret had a STALE password from before
streaming replication existed - the live Postgres role password now
replicates from home via WAL, but VPS's local K8s secret never got
updated to match. Needs a one-time sync (and again on any future
rotation).
- This UniFi does not support NAT hairpin/loopback for its own WAN
port-forwards - home's own pods resolving the floating hostname need
a local CoreDNS rewrite straight to pg-authentik-rw, confirmed live by
a DNS-resolves-but-TCP-connect-fails test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
69 lines
3.8 KiB
Bash
Executable File
69 lines
3.8 KiB
Bash
Executable File
#!/bin/bash
|
|
# Multi-site active failover pilot (see
|
|
# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - the deliberate
|
|
# MANUAL counterpart to scripts/ha-failover-watcher.sh's automatic
|
|
# promotion. Run this by hand, on the VPS, once you've confirmed home is
|
|
# genuinely healthy again and want to move primary back - never automated,
|
|
# by design (auto-flipping back immediately on reconnect risks flapping,
|
|
# and a human should confirm home's data/state before handing writes back
|
|
# to it).
|
|
#
|
|
# Order matters:
|
|
# 1. Confirm home's pg-authentik Cluster is healthy and NOT still
|
|
# thinking it's primary (it shouldn't be, since it was down/
|
|
# unreachable when the VPS promoted - but check
|
|
# status.currentPrimary on home before proceeding).
|
|
# 2. Re-point home's Cluster to replicate FROM the vps (it needs to
|
|
# catch up on everything written to the VPS while it was down)
|
|
# before flipping primary back - home has to actually BE a caught-up
|
|
# replica of vps first, or this loses the writes the VPS accepted
|
|
# during the outage.
|
|
# 3. Only once home shows it's streaming and caught up, flip primary
|
|
# back to home and update DNS.
|
|
#
|
|
# This script only does step 3 (the fast, symmetric part - same shape as
|
|
# the watcher's own promote() but in reverse). Steps 1-2 are a judgment
|
|
# call requiring you to actually look at both clusters' status first -
|
|
# not scripted here on purpose.
|
|
set -euo pipefail
|
|
|
|
echo "This will flip pg-authentik's primary back to home and repoint DNS."
|
|
echo "Before continuing, you MUST have already confirmed:"
|
|
echo " - home's pg-authentik Cluster is healthy and replicating FROM vps"
|
|
echo " (kubectl -n authentik get cluster pg-authentik -o jsonpath='{.status}')"
|
|
echo " - home is caught up (no meaningful replication lag from vps)"
|
|
read -r -p "Confirmed both of the above? [y/N] " ans
|
|
if [ "${ans:-N}" != "y" ] && [ "${ans:-N}" != "Y" ]; then
|
|
echo "Aborted."
|
|
exit 1
|
|
fi
|
|
|
|
echo "Flipping home's Cluster CR to primary..."
|
|
kubectl -n authentik patch cluster pg-authentik --type merge \
|
|
-p '{"spec":{"replica":{"self":"home","primary":"home","source":"home"}}}'
|
|
|
|
echo "Flipping VPS's Cluster CR back to a replica of home..."
|
|
ssh root@172.93.53.139 "kubectl -n authentik patch cluster pg-authentik --type merge \
|
|
-p '{\"spec\":{\"replica\":{\"self\":\"vps\",\"primary\":\"home\",\"source\":\"home\"}}}'"
|
|
|
|
echo "Flipping pg-authentik.ha.huskypup.net back to home's IP..."
|
|
HOME_PUBLIC_IP="$(dig +short home.kube.huskypup.net @1.1.1.1 | tail -1)"
|
|
if [ -z "$HOME_PUBLIC_IP" ]; then
|
|
echo "ERROR: could not resolve home.kube.huskypup.net - fix the DNS record manually"
|
|
exit 1
|
|
fi
|
|
TOKEN="$(kubectl -n cert-manager get secret cloudflare-token-secret -o jsonpath='{.data.cloudflare-token}' | base64 -d)"
|
|
ZONE_ID="$(curl -sf -H "Authorization: Bearer ${TOKEN}" "https://api.cloudflare.com/client/v4/zones?name=huskypup.net" | jq -r '.result[0].id')"
|
|
RECORD_ID="$(curl -sf -H "Authorization: Bearer ${TOKEN}" "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records?name=pg-authentik.ha.huskypup.net&type=A" | jq -r '.result[0].id')"
|
|
curl -sf -X PATCH -H "Authorization: Bearer ${TOKEN}" -H "Content-Type: application/json" \
|
|
-d "{\"type\":\"A\",\"name\":\"pg-authentik.ha.huskypup.net\",\"content\":\"${HOME_PUBLIC_IP}\",\"ttl\":60,\"proxied\":false}" \
|
|
"https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records/${RECORD_ID}" >/dev/null
|
|
|
|
echo "Clearing the etcd promotion record..."
|
|
curl -sf --max-time 5 -X POST http://172.28.101.41:32379/v3/kv/deleterange \
|
|
-d "{\"key\":\"$(printf '%s' '/ha-failover/promoted-at' | base64)\"}" >/dev/null || true
|
|
curl -sf --max-time 5 -X POST http://172.28.101.41:32379/v3/kv/deleterange \
|
|
-d "{\"key\":\"$(printf '%s' '/ha-failover/promoted-by' | base64)\"}" >/dev/null || true
|
|
|
|
echo "Failback complete. home is primary again, pg-authentik.ha.huskypup.net -> ${HOME_PUBLIC_IP}"
|