Found while building the VPS replica clusters (Phase 2): all three
CNPG clusters' backup.barmanObjectStore blocks only configure continuous
WAL archiving. CNPG's bootstrap.recovery needs at least one real Backup
object in the object store before it has anything to restore -
'no target backup found', confirmed live against pg-authentik. This gap
was systemic (gitlab has it too, not fixed here - out of scope for
tonight, no VPS standby depends on it).
immediate: true fires one backup right away in addition to the daily
01:00 schedule, to unblock the in-progress VPS restore now rather than
waiting up to 24h for the first scheduled run.
Now that the endpoint/credentials are fixed and this job can actually
reach the VPS, it surfaced two more real bugs: no fsGroup (couldn't read
the PVC data at all - nextcloud's real Deployment uses fsGroup 33/www-data,
this job never matched it) and no writable cache dir for runAsUser 65534
(restic defaults to $HOME/.cache).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Repeated restarts from tonight's network disruption kept OOMKilling
rancher - each restart's startup reconciliation is memory-hungry, and 2Gi
wasn't enough headroom for that burst even though steady-state usage is
normally fine. Nodes have ample free memory (26-36% used).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
redis.enabled: true was unintentionally deploying the chart's BUNDLED redis
subchart (redis-21.1.3, StatefulSet nextcloud-redis-replicas) alongside
the actual redis-operator instance - that key's schema is pure subchart
passthrough (auth/image/master/replica), it has no host field. The
host/existingSecret keys previously set there were silently ignored.
Found while investigating why nextcloud-redis-replicas-0 was
crash-looping on failed liveness probes (confirmed via : the correct key for pointing at an external Redis
is the separate externalRedis: block).
This means Nextcloud's config.php may never have been correctly pointed
at the redis-operator instance despite apps/nextcloud/manifests/redis-cr.yaml
existing and being healthy - worth confirming file-locking/caching
actually engages after this deploys.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause of tonight's earlier CrowdSec/CNPG-backup workarounds: no node
in the cluster had any route into the Netbird mesh CIDR (100.108.0.0/16)
for pod-originated traffic. The per-namespace netbird 'router' pods are
inbound-only infrastructure (external peers reaching K8s services); their
own architecture has no reverse path.
Fix, in two parts:
1. infrastructure/netbird/manifests/egress-daemonset.yaml - one netbird
client per node, hostNetwork so its wt0 interface lives in the node's
real network namespace, plus a sidecar that adds a host route sending
100.108.0.0/16 out via it. hostNetwork requires a scoped Kyverno
PolicyException (infrastructure/kyverno/policies/netbird-egress-exception.yaml)
to the disallow-host-namespaces STIG policy - narrowly for this one
DaemonSet by name, not a namespace-wide exclusion.
2. Discovered the route alone wasn't enough for k3s NodePort traffic
(vps-minio:30900): Netbird manages its own nftables ACLs independent of
iptables/Kyverno, and its forward chain (netbird-rt-fwd) only permits
*established* connections through a peer acting as a router - never new
ones, by design, unless a Netbird 'Network Route' policy is explicitly
configured (it isn't, for this VPS). Locally-terminated connections
(tinyproxy) go through a separate, already-permissive ACL chain, which
is why the CrowdSec proxy fix from earlier tonight worked. Replicated
that working pattern for MinIO: minio-forward.service on the VPS host
(systemd, socat) forwards 100.108.113.41:9000 -> MinIO's ClusterIP,
avoiding the NodePort path entirely.
Re-enabled everything that was disabled/suspended earlier tonight because
of this gap, pointed at the new endpoint:
- CrowdSec CAPI/console-enroll (removed DISABLE_ONLINE_API, restored the
VPS proxy env vars)
- n8n/nextcloud/authentik CNPG backup.barmanObjectStore
- vault-raft-snapshot CronJob (unsuspended)
- nextcloud PVC content sync CronJob endpoint
vps-minio.netbird.internal is retired everywhere - it was never actually
resolvable (Netbird has no DNS configured) even before today's routing
fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Discovered while debugging pg-n8n's recurring 'Instance Status Extraction
Error': the root cause is the same missing pod-egress route to the Netbird
mesh found tonight while fixing CrowdSec - continuousArchiving kept failing
to connect to vps-minio.netbird.internal, and CNPG correctly holds
Ready=False while archiving is broken (a real condition, not cosmetic).
That's what was driving these three apps' ArgoCD health flapping.
No node in the cluster has any route into 100.108.0.0/16 for
pod-originated traffic - the per-namespace netbird 'router' pods
(gitlab, vault, argocd, etc.) are inbound-only infrastructure, not egress
gateways. gitlab's own CNPG backup is unaffected (points at a local
in-cluster MinIO, not the VPS).
Commented out rather than deleted - re-enable once real pod-egress
routing exists, tracked as a separate task. No data loss: this is WAL
archiving/backup, not the live database.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every other major namespace (gitlab, vault, argocd, nextcloud, etc.) has
its own 3-replica netbird-router deployment auto-provisioned from an
NBRoutingPeer CR; crowdsec never got one since it never previously needed
mesh connectivity. Without it, crowdsec pods had no dedicated mesh routing
path, causing intermittent connection resets/hangs reaching the VPS proxy
used to route around CrowdSec's WAF block on home's public IP.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
redis-nextcloud-0 was CrashLoopBackOff: "mkdir: can't create directory
'/node-conf': Permission denied" and same for appendonlydir - the pod
had no podSecurityContext, so it couldn't write to its own PVC-backed
data directories. Added fsGroup/runAsUser: 1000 matching the
opstree/redis image's default non-root user. Verified live: Redis now
starts cleanly ("Ready to accept connections").
apps/nextcloud/manifests/redis-cr.yaml used redis.redis.opstreelabs.in/v1beta2
(guessed, flagged as unverified when written) - the actually-installed CRD
only has v1beta1. ArgoCD validates all resource API groups/versions before
syncing anything, so this one bad apiVersion blocked the ENTIRE nextcloud
Application from syncing at all (every resource showed OutOfSync, not just
this one).
Also fixed against the live CRD schema: redisSecret belongs nested under
kubernetesConfig, not top-level, and redisExporter.image is required by the
schema even when enabled: false.
Verified live: kubectl apply succeeded and the Redis resource was created
before committing this.
Two related fixes, found while working through why authentik/n8n/gitlab
ArgoCD Applications showed Unknown/stuck health despite pods being fine:
1. ArgoCD has no built-in health check for CNPG's Cluster CRD, so it
always reported "Unknown" - and since app-level health rolls up to
the worst resource status, every app with a CNPG Cluster showed
Unknown/Progressing regardless of actual state. Added a Lua health
check (resource.customizations.health.postgresql.cnpg.io_Cluster)
reading .status.conditions[Ready] / .status.phase.
2. Once that started reporting real status instead of masking it,
pg-authentik showed a genuine problem: CNPG's operator couldn't
reach its Postgres instances' status API (port 8000) - "Cannot
extract Pod status ... context deadline exceeded" - because:
a) authentik-ingress's CiliumNetworkPolicy never allowed cnpg-system
as a source (crowdsec-ingress already had this exception,
authentik-ingress was just missing it - inconsistency, not
deliberate).
b) Even after fixing (a), still blocked - pg-authentik's pods are
ambient-mesh-enrolled, so the connection actually goes through
ztunnel's HBONE tunnel (port 15008) first, same underlying issue
as the argocd-redis fix from earlier today. Rather than keep
finding and patching this per-namespace as it recurs, added a
cluster-wide CiliumClusterwideNetworkPolicy allowing HBONE
broadly - ztunnel's own mTLS/SPIFFE identity verification is the
real security boundary for mesh traffic; Cilium blocking the
tunnel port itself was only breaking legitimate traffic, not
adding meaningful protection on top.
Verified live: CNPG operator immediately went from failing status
extraction to successfully reconciling (recreating a pod to reattach
its PVC) once both fixes were in place.
apps/nextcloud/values.yaml had oidc_login_client_id/secret as literal
empty strings, causing "client_id is missing or invalid" on login.
The matching Authentik blueprint (infrastructure/authentik/nextcloud-
blueprint.yaml) already defines fixed credentials for this OAuth2
provider - just never got copied over. Filled in to match.
Also found and left alone: an orphaned nextcloud-oidc-config ConfigMap
(from January) that already had the correct values but isn't actually
mounted by the current Deployment (which uses nextcloud-config instead)
- harmless dead resource, not the active config path.
- ZONE_NAME was "kube.huskypup.net" in both the home-ip-ddns CronJob and
the VPS failover watcher - that's a record, not a Cloudflare zone (the
actual zone is the parent "huskypup.net"). Caused home-ip-ddns to fail
every run (zone lookup returned zero results, curl -f exit 22) -
confirmed live and fixed.
- Added seccompProfile/non-root/dropped-capabilities securityContext to
the three CronJobs added this session that were missing it (flagged by
the cluster's "restricted" PodSecurity admission). Repointed the raft
snapshot job's mc binary install from /usr/local/bin to /tmp so it
still works running as non-root.
Root cause (confirmed via ztunnel logs): the argo-cd chart's default
NetworkPolicies only allow each component's application port (e.g. 6379
for redis), but Istio ambient mode routes ALL pod-to-pod traffic through
ztunnel's HBONE tunnel on port 15008 first - so the tunnel itself was
being blocked even though the "real" port was allowed. Every inter-pod
connection in the argocd namespace hung for exactly 10s then reset;
ztunnel's own log named it directly ("connection timed out, maybe a
NetworkPolicy is blocking HBONE port 15008"). This broke argocd-server's
Redis-backed session/cluster-info caching cluster-wide and was silently
preventing the root Application from picking up new child Applications.
Fix: additive NetworkPolicy allowing ingress on 15008 for all argocd
pods (NetworkPolicies union across multiple policies selecting a pod,
so this doesn't touch/replace the chart's rendered ones - safe across
Helm upgrades).
Also: dropped argocd from the namespace-enrollment job's waypoint list.
argocd's only AuthorizationPolicy (allow-argocd-access) is a plain
source-namespace/IP match with no L7 rules - its own status shows
"attached to ztunnel", not waypoint - so forcing L7 waypoint processing
onto the namespace was unnecessary overhead, not a security requirement.
ztunnel's mTLS still fully covers it. (This was a red herring for the
HBONE bug itself, not the fix, but a valid simplification found along
the way.)
Foundation for a DR/backup path using an always-on VPS as a second
ArgoCD-managed cluster, plus DB/backup standardization work that fell
out of it:
- vps-standby ArgoCD cluster destination + AppProject, MinIO backup
receiver, VPS bootstrap script (k3s, Netbird, cert-manager)
- Dual-site DNS failover watcher + home-IP DDNS CronJob, Cloudflare
token moved out of git into Vault+ExternalSecret
- Nextcloud migrated from ad-hoc MariaDB to CNPG + redis-operator
(matches n8n/Authentik/GitLab's backup-native pattern)
- Authentik's CNPG manifests moved into the actual ArgoCD-synced
manifests/ path (were present but never wired into the sync path)
- Vault raft-snapshot CronJob, CNPG barmanObjectStore backups
(Authentik/n8n/Nextcloud), Nextcloud file-PVC restic sync - all
targeting the new VPS MinIO receiver
See VPS Warm-Standby plan doc for full design rationale.
Gitaly CPU limit was 250m causing throttling on git operations (485ms
p99 to webservice). Increased to 1 core with 100m request. Also added
workerProcesses: 3 to webservice for better request concurrency
(was default 2).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Traffic goes through Istio (not Envoy Gateway), so cookie-based
consistent hashing must be on the Istio DestinationRule. Removes
the unused Envoy Gateway BackendTrafficPolicy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fixes OIDC nonce validation failures with 2 replicas by using a
BackendTrafficPolicy with cookie-based consistent hashing instead of
Service-level ClientIP affinity (which doesn't work behind Envoy proxy).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
With 2 replicas, OIDC callbacks can hit a different pod than the
one that generated the state/nonce, causing auth failures. Session
affinity ensures the same client always reaches the same pod.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change EXTENSION_PRIORITY from "*,openid" to "openid,*" so
unauthenticated users are redirected to Authentik OIDC instead
of seeing the database login form.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fixes NameError: uninitialized constant Gitlab::Redis::ALL_CLASSES
caused by 7_prometheus_metrics.rb loading before 7_redis.rb
(alphabetically 'p' < 'r'). The ConfigMap provides a 0_redis_early.rb
initializer that requires gitlab/redis before other initializers run.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Scale CNPG pg-teslamate to 1 instance: on-disk storage corruption
(corrupted sequences, system catalog indexes) means replicas cloned
via pg_basebackup inherit corruption and can't open any database.
Primary works from cached catalogs. Sequences already rebuilt live.
- Remove talos-cp-02 from Rook-Ceph nodes list: ceph-volume inventory
hangs on BIOS partition via nsenter lvs in host namespace. Talos
read-only /etc prevents LVM filter fix. OSD 1 keeps running from
its existing deployment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Home Assistant was returning 400 errors because it received
X-Forwarded-For headers from the Istio ingress gateway but wasn't
configured to trust reverse proxies. Add use_x_forwarded_for and
trusted_proxies to the init container's configuration.yaml template.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- n8n: Set replicaCount=1 (RWO PVC incompatible with multiple replicas),
add resource limits to satisfy Kyverno policy
- gitlab: Add ignoreDifferences for redis-gitlab-additional service
(port names and selectors managed by Redis operator)
- netbird-operator: Add ExternalSecret for netbird-mgmt-api-key in
netbird namespace and add manifests source to Application
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Webservice was at 94% memory (1875Mi/2Gi) with only 50m CPU request,
causing CPU throttling and slow first requests after idle periods.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
s6-overlay's log-prepare needs chmod on /dev/shm/logs/* which
requires CAP_FOWNER when all capabilities are dropped.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Revert talos-client-cert secret to prometheus namespace (prometheus
pods mount this secret, not kube-system pods)
- Add CHOWN/DAC_OVERRIDE/SETUID/SETGID capabilities to frigate
(s6-overlay needs chown for log directories)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
