Part of the Authentik HA pilot (see plan doc). Home's etcd quorum member
(StatefulSet, pinned to talos-cp-01 for a stable advertised address) plus
a NodePort exposing pg-authentik's current primary - both reachable from
the VPS/witness over the already-authenticated Netbird mesh (confirmed
live: home nodes are directly reachable from Netbird peers on their real
node IP, via the netbird-egress DaemonSet's route). Deliberately NOT
going through UniFi/public-internet exposure - this stays entirely
inside the private mesh, a materially safer path than the WAN port-
forward originally considered.
Needs a scoped Kyverno PolicyException (ha-failover-nodeport-exception.yaml)
since disallow-nodeport-services is enforced cluster-wide - narrowly
scoped to Services named ha-*, matching the existing netbird-egress-
exemption.yaml precedent for exceptions.
The DNS-flip watcher (scripts/vps-dns-failover/) was designed but never
actually installed on the VPS despite being tracked as done - real gap,
found when asked whether the standby services are actually reachable.
New design: instead of dynamically flipping *.kube.huskypup.net between
home and VPS IPs, give the VPS site its own permanent, always-resolving
subdomain - vault/gitea/auth/n8n/nextcloud.vps.huskypup.net, each with
real Ingress+TLS on the VPS's own Traefik+cert-manager (both already
installed by Phase 0 bootstrap, just never wired up). No token-scoping
decision needed since there's no dynamic flipping - reuses the same
cert-manager token pattern as home.
Also scales Authentik/n8n/Nextcloud from 0 to 1 replica on the VPS so
the replicated data is actually browsable at all times, not just
present-but-unreachable. Their CNPG clusters are still read-only
replicas (spec.replica.enabled: true) - writes will error until a
deliberate manual promotion, but reads/browsing work now. Vault and
Gitea were already running continuously.
Same CNPG replica-cluster pattern as Authentik (continuous WAL replay
from home via VPS MinIO, app at 0 replicas until manual promotion - see
infrastructure/vps-standby/authentik/manifests/cnpg-cluster.yaml for the
full rationale).
n8n: N8N_ENCRYPTION_KEY copied byte-identical from home (kubectl, not
git) - decrypts stored credentials in the replicated DB, same reasoning
as Vault's unseal key / root token copies.
Nextcloud: adds infrastructure/vps-standby/nextcloud/manifests/
pvc-restore-cronjob.yaml, the read side of home's existing
nextcloud-pvc-sync restic backup - restores the latest snapshot from VPS
MinIO into this standby's PVC daily at 04:00 (2h after home's 02:00
backup). No Redis on the VPS side (no redis-operator deployed there,
out of scope for a standby that isn't serving traffic - Nextcloud
degrades gracefully to DB-based locking without it).
CNPG operator on the VPS (argocd-apps/vps-standby/cnpg-operator.yaml),
plus a CNPG 'replica cluster' for pg-authentik (infrastructure/
vps-standby/authentik/manifests/cnpg-cluster.yaml) that continuously
replays WAL shipped from home's pg-authentik via VPS MinIO's
cnpg-backups/pg-authentik bucket path - CNPG's native replica-cluster
mode, not a custom restore script like Vault needed (Vault has no
equivalent built-in continuous-replication-into-object-store feature).
The Authentik app itself runs at replicas: 0 in normal operation - the
replica DB is read-only until a deliberate manual promotion
(spec.replica.enabled: false), and a pod trying to write against a
read-only DB would just crashloop uselessly. values.yaml's authentik:
block is copied byte-identical from home's on purpose: the chart
deterministically derives its generated Secret's AUTHENTIK_SECRET_KEY
from these literal values.yaml strings, so both clusters land on the
same key without manually copying it - required since that key decrypts
things stored encrypted in the replicated DB.
Pull-mirrors home GitLab's Homelabv4 repo on Gitea's own built-in mirror
scheduler (6h interval) - no custom sync job needed, per the original
plan. SQLite instead of the chart's default HA Postgres + Valkey cluster
(single-instance standby holding one small repo, not worth the extra
moving parts). A PostSync Job creates the mirror once, idempotently;
Gitea's scheduler handles all ongoing pulls after that.
Also added a public Cloudflare CNAME for gitlab.kube.huskypup.net ->
home.kube.huskypup.net: the VPS has no route to home's LAN via Netbird
(none of the mesh peers advertise that subnet, confirmed live), so
GitLab needs to be reachable the same way any other internet client
reaches it - home's public IP already has port 443 forwarded to
istio-ingressgateway from earlier this session.
Admin and GitLab-mirror credentials are plain Secrets created directly
on the VPS cluster (kubectl, not git) - same pattern as vault-unseal-key.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Single-node Vault (raft storage, standalone mode) plus a restore CronJob
that pulls the latest raft snapshot from the VPS's own local MinIO
(already receiving snapshots every 6h from home) and restores it every
6h, offset 30 minutes after the home-side snapshot job.
Not an independent root of trust: raft snapshot restore replaces the
entire storage backend including the keyring, so after every restore
this Vault is sealed with HOME's actual keyring - unsealing it needs a
copy of home's real unseal key, stored directly on the VPS cluster
(kubectl, not git, same pattern as vps-minio-root-secret). First-run
bootstrap uses a throwaway single-Shamir-key init just to get through
the very first restore, then is irrelevant from then on.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Foundation for a DR/backup path using an always-on VPS as a second
ArgoCD-managed cluster, plus DB/backup standardization work that fell
out of it:
- vps-standby ArgoCD cluster destination + AppProject, MinIO backup
receiver, VPS bootstrap script (k3s, Netbird, cert-manager)
- Dual-site DNS failover watcher + home-IP DDNS CronJob, Cloudflare
token moved out of git into Vault+ExternalSecret
- Nextcloud migrated from ad-hoc MariaDB to CNPG + redis-operator
(matches n8n/Authentik/GitLab's backup-native pattern)
- Authentik's CNPG manifests moved into the actual ArgoCD-synced
manifests/ path (were present but never wired into the sync path)
- Vault raft-snapshot CronJob, CNPG barmanObjectStore backups
(Authentik/n8n/Nextcloud), Nextcloud file-PVC restic sync - all
targeting the new VPS MinIO receiver
See VPS Warm-Standby plan doc for full design rationale.
NetBird router deployments require NET_ADMIN capability, which is
blocked by the default baseline PodSecurity standard. Add
managedNamespaceMetadata with pod-security.kubernetes.io/enforce:
privileged to argocd, frigate, gitlab, grafana, guacamole,
home-assistant, teslamate, and vault ArgoCD Applications.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Network policies in the network-policies/ subdirectory were not being
synced because ArgoCD was not recursing into subdirectories.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The scylla-manager Application was missing the third source for raw
manifests, so the VirtualService in infrastructure/scylla/manifests/
was never deployed. This caused Kiali to flag the ext-authz policy
host as having no matching service entry.
Also removed stale PERMISSIVE PeerAuthentication from authentik
namespace (manually applied, not in git, unnecessary with mesh-wide
STRICT mTLS).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- n8n: Set replicaCount=1 (RWO PVC incompatible with multiple replicas),
add resource limits to satisfy Kyverno policy
- gitlab: Add ignoreDifferences for redis-gitlab-additional service
(port names and selectors managed by Redis operator)
- netbird-operator: Add ExternalSecret for netbird-mgmt-api-key in
netbird namespace and add manifests source to Application
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Gateway API CRDs are already managed by istio; skip them
from the vendored envoy-gateway chart to prevent OutOfSync.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The checksum/lapi-secret annotation on the LAPI Deployment changes
when CrowdSec auto-generates credentials, causing permanent
OutOfSync with ArgoCD ServerSideDiff.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ArgoCD v3.3.3 has a bug where OCI chart revision resolution
doesn't include the chart name in the Docker Hub API path,
causing 401 errors. The chart is already deployed at v1.6.3.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ArgoCD requires the chart field for OCI sources. The repoURL
contains the registry/org path, and chart specifies the chart name.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move chart name into repoURL for OCI registry compatibility.
ArgoCD resolves OCI chart references differently from HTTP repos.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Works with the ignoreDifferences config in argocd-cm to prevent
auto-sync from triggering on controller-managed field changes
(ESO finalizers, HTTPRoute annotations, etc.)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Apps using ServerSideApply without ServerSideDiff causes ArgoCD to
compute diffs client-side, which detects controller-added fields
(finalizers, defaults, annotations) as changes and keeps apps
permanently OutOfSync. ServerSideDiff uses dry-run SSA for accurate
diff computation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Rook operator reconciles OSD deployments and removes the
expand-bluefs=/bin/true workaround patches needed for BlueStore UUID
mismatch. Keep operator scaled to 0 with manual sync only until the
underlying issue is resolved.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Envoy Gateway moved to OCI registry (docker.io/envoyproxy).
NetBird operator charts moved to https://netbirdio.github.io/helms
with chart name kubernetes-operator.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
