mirror of
https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
synced 2026-08-20 23:16:49 +00:00
Scooby HuskyandClaude Sonnet 5
f7a9a03816
Builds the one genuinely new component from the plan - everything else
reuses CNPG/etcd as-is.
- infrastructure/ha-failover/manifests/heartbeat-writer.yaml: home writes
a fresh timestamp to etcd (/ha-failover/home-heartbeat) every 10s via
its local etcd member. No custom quorum/voting logic - etcd's own Raft
consensus (majority write/linearizable read) already gives the 'a
majority agrees' guarantee.
- scripts/ha-failover-watcher.sh + .service: runs on the VPS as a
systemd service (not k3s - must survive the VPS's own cluster being
unhealthy). Reads the heartbeat via its own local etcd member; if the
read itself fails, we can't tell if home is down or if this VPS is the
one partitioned, so it does NOT act (etcd's consensus requirement
provides the split-brain safety here, not custom code). If the read
succeeds and the heartbeat is stale (>45s, ~4-5 missed beats), home is
confirmed down by majority - promotes pg-authentik's Cluster CR
(spec.replica self/primary/source -> vps, no promotionToken - confirmed
live via dry-run that CNPG's webhook doesn't require one, it's only for
planned/graceful switchover LSN cross-checks that aren't available
during a genuine unplanned outage) and flips the
pg-authentik.ha.huskypup.net Cloudflare A record to the VPS's IP.
- scripts/ha-failback-authentik.sh: the deliberately manual reverse -
human confirms home is healthy and caught up before running this.
- Floating hostname plumbing: both sites' Authentik values now read
AUTHENTIK_POSTGRESQL__HOST/PORT as pg-authentik.ha.huskypup.net:61432
instead of the local pg-authentik-app secret's host/port, so app pods
on either site always reach whichever site is actually primary. Both
sites listen on the same external port (VPS gets a new
pg-authentik-forward.service socat forward, mirroring the existing
minio-forward.service pattern, so its NodePort 32433 is externally
reachable on 61432 same as home's UniFi-forwarded port).
- Found and worked around two real bugs surfaced while wiring this up
(fixed manually via scripts handed to the user - both blocked by the
Claude Code auto-mode classifier as credential-transmission /
cluster-DNS-edit actions):
- VPS's pg-authentik-app secret had a STALE password from before
streaming replication existed - the live Postgres role password now
replicates from home via WAL, but VPS's local K8s secret never got
updated to match. Needs a one-time sync (and again on any future
rotation).
- This UniFi does not support NAT hairpin/loopback for its own WAN
port-forwards - home's own pods resolving the floating hostname need
a local CoreDNS rewrite straight to pg-authentik-rw, confirmed live by
a DNS-resolves-but-TCP-connect-fails test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
86 lines
2.7 KiB
YAML
86 lines
2.7 KiB
YAML
# Multi-site active failover pilot (see
|
|
# /home/scooby/.claude/plans/jiggly-snacking-iverson.md) - home's half of
|
|
# the failover-watcher. This is deliberately NOT a general Postgres HA
|
|
# controller: its only job is writing a fresh unix timestamp to etcd every
|
|
# 10s at key /ha-failover/home-heartbeat. The VPS's watcher
|
|
# (scripts/ha-failover-watcher.sh, deployed via systemd - see that
|
|
# script's own header) reads this key and decides whether to promote
|
|
# based purely on how stale it is - no custom voting/consensus logic
|
|
# needed here, etcd's own Raft consensus already provides the "majority
|
|
# agrees" guarantee: a write only succeeds if a majority of the 3 etcd
|
|
# members (home/vps/witness) are reachable and agree, and a linearizable
|
|
# read (the JSON gateway's default) only ever returns majority-confirmed
|
|
# state.
|
|
#
|
|
# Talks to etcd via its LOCAL ClusterIP (ha-etcd.ha-failover.svc.cluster.local
|
|
# :2379) - home reads/writes its OWN cluster member directly, no need to
|
|
# round-trip externally for this side.
|
|
#
|
|
# No RBAC/ServiceAccount needed - this pod never touches the K8s API,
|
|
# only etcd's HTTP gateway via curl.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: ha-heartbeat-script
|
|
namespace: ha-failover
|
|
data:
|
|
heartbeat.sh: |
|
|
#!/bin/sh
|
|
set -eu
|
|
KEY_B64="$(printf '%s' '/ha-failover/home-heartbeat' | base64 | tr -d '\n')"
|
|
while true; do
|
|
NOW="$(date +%s)"
|
|
VAL_B64="$(printf '%s' "$NOW" | base64 | tr -d '\n')"
|
|
if curl -sf --max-time 5 -X POST \
|
|
http://ha-etcd.ha-failover.svc.cluster.local:2379/v3/kv/put \
|
|
-d "{\"key\":\"${KEY_B64}\",\"value\":\"${VAL_B64}\"}" >/dev/null; then
|
|
echo "heartbeat ${NOW} ok"
|
|
else
|
|
echo "heartbeat ${NOW} FAILED (etcd unreachable or no quorum)"
|
|
fi
|
|
sleep 10
|
|
done
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: ha-heartbeat-writer
|
|
namespace: ha-failover
|
|
spec:
|
|
replicas: 1
|
|
selector:
|
|
matchLabels:
|
|
app: ha-heartbeat-writer
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: ha-heartbeat-writer
|
|
spec:
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 1000
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: heartbeat
|
|
image: alpine/k8s:1.32.13
|
|
command: ["/bin/sh", "/scripts/heartbeat.sh"]
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
resources:
|
|
requests:
|
|
cpu: 5m
|
|
memory: 16Mi
|
|
limits:
|
|
memory: 64Mi
|
|
volumeMounts:
|
|
- name: script
|
|
mountPath: /scripts
|
|
volumes:
|
|
- name: script
|
|
configMap:
|
|
name: ha-heartbeat-script
|
|
defaultMode: 0755
|