Found while building the VPS replica clusters (Phase 2): all three
CNPG clusters' backup.barmanObjectStore blocks only configure continuous
WAL archiving. CNPG's bootstrap.recovery needs at least one real Backup
object in the object store before it has anything to restore -
'no target backup found', confirmed live against pg-authentik. This gap
was systemic (gitlab has it too, not fixed here - out of scope for
tonight, no VPS standby depends on it).
immediate: true fires one backup right away in addition to the daily
01:00 schedule, to unblock the in-progress VPS restore now rather than
waiting up to 24h for the first scheduled run.
Now that the endpoint/credentials are fixed and this job can actually
reach the VPS, it surfaced two more real bugs: no fsGroup (couldn't read
the PVC data at all - nextcloud's real Deployment uses fsGroup 33/www-data,
this job never matched it) and no writable cache dir for runAsUser 65534
(restic defaults to $HOME/.cache).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause of tonight's earlier CrowdSec/CNPG-backup workarounds: no node
in the cluster had any route into the Netbird mesh CIDR (100.108.0.0/16)
for pod-originated traffic. The per-namespace netbird 'router' pods are
inbound-only infrastructure (external peers reaching K8s services); their
own architecture has no reverse path.
Fix, in two parts:
1. infrastructure/netbird/manifests/egress-daemonset.yaml - one netbird
client per node, hostNetwork so its wt0 interface lives in the node's
real network namespace, plus a sidecar that adds a host route sending
100.108.0.0/16 out via it. hostNetwork requires a scoped Kyverno
PolicyException (infrastructure/kyverno/policies/netbird-egress-exception.yaml)
to the disallow-host-namespaces STIG policy - narrowly for this one
DaemonSet by name, not a namespace-wide exclusion.
2. Discovered the route alone wasn't enough for k3s NodePort traffic
(vps-minio:30900): Netbird manages its own nftables ACLs independent of
iptables/Kyverno, and its forward chain (netbird-rt-fwd) only permits
*established* connections through a peer acting as a router - never new
ones, by design, unless a Netbird 'Network Route' policy is explicitly
configured (it isn't, for this VPS). Locally-terminated connections
(tinyproxy) go through a separate, already-permissive ACL chain, which
is why the CrowdSec proxy fix from earlier tonight worked. Replicated
that working pattern for MinIO: minio-forward.service on the VPS host
(systemd, socat) forwards 100.108.113.41:9000 -> MinIO's ClusterIP,
avoiding the NodePort path entirely.
Re-enabled everything that was disabled/suspended earlier tonight because
of this gap, pointed at the new endpoint:
- CrowdSec CAPI/console-enroll (removed DISABLE_ONLINE_API, restored the
VPS proxy env vars)
- n8n/nextcloud/authentik CNPG backup.barmanObjectStore
- vault-raft-snapshot CronJob (unsuspended)
- nextcloud PVC content sync CronJob endpoint
vps-minio.netbird.internal is retired everywhere - it was never actually
resolvable (Netbird has no DNS configured) even before today's routing
fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Discovered while debugging pg-n8n's recurring 'Instance Status Extraction
Error': the root cause is the same missing pod-egress route to the Netbird
mesh found tonight while fixing CrowdSec - continuousArchiving kept failing
to connect to vps-minio.netbird.internal, and CNPG correctly holds
Ready=False while archiving is broken (a real condition, not cosmetic).
That's what was driving these three apps' ArgoCD health flapping.
No node in the cluster has any route into 100.108.0.0/16 for
pod-originated traffic - the per-namespace netbird 'router' pods
(gitlab, vault, argocd, etc.) are inbound-only infrastructure, not egress
gateways. gitlab's own CNPG backup is unaffected (points at a local
in-cluster MinIO, not the VPS).
Commented out rather than deleted - re-enable once real pod-egress
routing exists, tracked as a separate task. No data loss: this is WAL
archiving/backup, not the live database.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
redis-nextcloud-0 was CrashLoopBackOff: "mkdir: can't create directory
'/node-conf': Permission denied" and same for appendonlydir - the pod
had no podSecurityContext, so it couldn't write to its own PVC-backed
data directories. Added fsGroup/runAsUser: 1000 matching the
opstree/redis image's default non-root user. Verified live: Redis now
starts cleanly ("Ready to accept connections").
apps/nextcloud/manifests/redis-cr.yaml used redis.redis.opstreelabs.in/v1beta2
(guessed, flagged as unverified when written) - the actually-installed CRD
only has v1beta1. ArgoCD validates all resource API groups/versions before
syncing anything, so this one bad apiVersion blocked the ENTIRE nextcloud
Application from syncing at all (every resource showed OutOfSync, not just
this one).
Also fixed against the live CRD schema: redisSecret belongs nested under
kubernetesConfig, not top-level, and redisExporter.image is required by the
schema even when enabled: false.
Verified live: kubectl apply succeeded and the Redis resource was created
before committing this.
- ZONE_NAME was "kube.huskypup.net" in both the home-ip-ddns CronJob and
the VPS failover watcher - that's a record, not a Cloudflare zone (the
actual zone is the parent "huskypup.net"). Caused home-ip-ddns to fail
every run (zone lookup returned zero results, curl -f exit 22) -
confirmed live and fixed.
- Added seccompProfile/non-root/dropped-capabilities securityContext to
the three CronJobs added this session that were missing it (flagged by
the cluster's "restricted" PodSecurity admission). Repointed the raft
snapshot job's mc binary install from /usr/local/bin to /tmp so it
still works running as non-root.
Foundation for a DR/backup path using an always-on VPS as a second
ArgoCD-managed cluster, plus DB/backup standardization work that fell
out of it:
- vps-standby ArgoCD cluster destination + AppProject, MinIO backup
receiver, VPS bootstrap script (k3s, Netbird, cert-manager)
- Dual-site DNS failover watcher + home-IP DDNS CronJob, Cloudflare
token moved out of git into Vault+ExternalSecret
- Nextcloud migrated from ad-hoc MariaDB to CNPG + redis-operator
(matches n8n/Authentik/GitLab's backup-native pattern)
- Authentik's CNPG manifests moved into the actual ArgoCD-synced
manifests/ path (were present but never wired into the sync path)
- Vault raft-snapshot CronJob, CNPG barmanObjectStore backups
(Authentik/n8n/Nextcloud), Nextcloud file-PVC restic sync - all
targeting the new VPS MinIO receiver
See VPS Warm-Standby plan doc for full design rationale.
Scooby Husky