Add VPS warm-standby/backup site (Phase 0-1b)

Foundation for a DR/backup path using an always-on VPS as a second
ArgoCD-managed cluster, plus DB/backup standardization work that fell
out of it:

- vps-standby ArgoCD cluster destination + AppProject, MinIO backup
  receiver, VPS bootstrap script (k3s, Netbird, cert-manager)
- Dual-site DNS failover watcher + home-IP DDNS CronJob, Cloudflare
  token moved out of git into Vault+ExternalSecret
- Nextcloud migrated from ad-hoc MariaDB to CNPG + redis-operator
  (matches n8n/Authentik/GitLab's backup-native pattern)
- Authentik's CNPG manifests moved into the actual ArgoCD-synced
  manifests/ path (were present but never wired into the sync path)
- Vault raft-snapshot CronJob, CNPG barmanObjectStore backups
  (Authentik/n8n/Nextcloud), Nextcloud file-PVC restic sync - all
  targeting the new VPS MinIO receiver

See VPS Warm-Standby plan doc for full design rationale.
This commit is contained in:
Scooby Husky
2026-08-17 14:59:26 -05:00
parent 5163403e24
commit 7990f1fa47
25 changed files with 1161 additions and 139 deletions
@@ -0,0 +1,116 @@
---
# Periodic Vault raft snapshot, shipped to the VPS MinIO backup receiver.
# This is the DR path for Vault's data independent of the unseal-key custody
# story - a Ceph/cluster-loss disaster is recovered by standing up a fresh
# Vault and `vault operator raft snapshot restore`ing the latest one of these,
# not by anything to do with the unseal key itself.
#
# Requires a one-time manual step after VPS bootstrap: store the VPS MinIO
# root credentials (see infrastructure/vps-standby/minio/values.yaml) into
# this cluster's Vault so ESO can hand them to the CronJob:
# vault kv put secret/vps-minio-credentials \
# access-key=<minio-root-user> secret-key=<minio-root-password>
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: vps-minio-credentials
namespace: vault
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: vault-backend
target:
name: vps-minio-credentials
creationPolicy: Owner
data:
- secretKey: access-key
remoteRef:
key: vps-minio-credentials
property: access-key
- secretKey: secret-key
remoteRef:
key: vps-minio-credentials
property: secret-key
---
apiVersion: v1
kind: ConfigMap
metadata:
name: vault-raft-snapshot-script
namespace: vault
data:
snapshot.sh: |
#!/bin/bash
set -euo pipefail
# VPS's Netbird address - replace with the actual peer IP/hostname once
# the VPS is bootstrapped and joined to the mesh (scripts/vps-bootstrap.sh).
VPS_MINIO_ENDPOINT="${VPS_MINIO_ENDPOINT:-vps-minio.netbird.internal:30900}"
BUCKET="vault-raft-snapshots"
SNAP_NAME="vault-raft-$(date -u +%Y%m%dT%H%M%SZ).snap"
ROOT_TOKEN="$(kubectl -n vault get secret vault-init-keys -o jsonpath='{.data.VAULT_ROOT_TOKEN}' | base64 -d)"
echo "==> Taking raft snapshot from vault-0..."
kubectl -n vault exec vault-0 -- env VAULT_TOKEN="$ROOT_TOKEN" \
vault operator raft snapshot save "/tmp/${SNAP_NAME}"
echo "==> Copying snapshot out of vault-0..."
kubectl -n vault cp "vault-0:/tmp/${SNAP_NAME}" "/tmp/${SNAP_NAME}"
kubectl -n vault exec vault-0 -- rm -f "/tmp/${SNAP_NAME}"
echo "==> Installing mc (MinIO client)..."
curl -sf https://dl.min.io/client/mc/release/linux-amd64/mc -o /usr/local/bin/mc
chmod +x /usr/local/bin/mc
mc alias set vps-minio "http://${VPS_MINIO_ENDPOINT}" \
"${MINIO_ACCESS_KEY}" "${MINIO_SECRET_KEY}" >/dev/null
echo "==> Uploading ${SNAP_NAME} to vps-minio/${BUCKET}..."
mc cp "/tmp/${SNAP_NAME}" "vps-minio/${BUCKET}/${SNAP_NAME}"
rm -f "/tmp/${SNAP_NAME}"
echo "==> Pruning snapshots older than 30 days..."
mc find "vps-minio/${BUCKET}" --older-than 30d --exec "mc rm {}" || true
echo "==> Done: ${SNAP_NAME}"
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: vault-raft-snapshot
namespace: vault
spec:
schedule: "0 */6 * * *" # every 6 hours
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 2
template:
spec:
serviceAccountName: argocd-hook-sa # already has kubectl exec rights in this namespace (see vault-init-job.yaml)
restartPolicy: Never
containers:
- name: raft-snapshot
image: alpine/k8s:1.32.13
command: ["/bin/bash", "/scripts/snapshot.sh"]
env:
- name: MINIO_ACCESS_KEY
valueFrom:
secretKeyRef:
name: vps-minio-credentials
key: access-key
- name: MINIO_SECRET_KEY
valueFrom:
secretKeyRef:
name: vps-minio-credentials
key: secret-key
volumeMounts:
- name: scripts
mountPath: /scripts
volumes:
- name: scripts
configMap:
name: vault-raft-snapshot-script
defaultMode: 0755