redis-nextcloud-0 was CrashLoopBackOff: "mkdir: can't create directory
'/node-conf': Permission denied" and same for appendonlydir - the pod
had no podSecurityContext, so it couldn't write to its own PVC-backed
data directories. Added fsGroup/runAsUser: 1000 matching the
opstree/redis image's default non-root user. Verified live: Redis now
starts cleanly ("Ready to accept connections").
apps/nextcloud/manifests/redis-cr.yaml used redis.redis.opstreelabs.in/v1beta2
(guessed, flagged as unverified when written) - the actually-installed CRD
only has v1beta1. ArgoCD validates all resource API groups/versions before
syncing anything, so this one bad apiVersion blocked the ENTIRE nextcloud
Application from syncing at all (every resource showed OutOfSync, not just
this one).
Also fixed against the live CRD schema: redisSecret belongs nested under
kubernetesConfig, not top-level, and redisExporter.image is required by the
schema even when enabled: false.
Verified live: kubectl apply succeeded and the Redis resource was created
before committing this.
Two related fixes, found while working through why authentik/n8n/gitlab
ArgoCD Applications showed Unknown/stuck health despite pods being fine:
1. ArgoCD has no built-in health check for CNPG's Cluster CRD, so it
always reported "Unknown" - and since app-level health rolls up to
the worst resource status, every app with a CNPG Cluster showed
Unknown/Progressing regardless of actual state. Added a Lua health
check (resource.customizations.health.postgresql.cnpg.io_Cluster)
reading .status.conditions[Ready] / .status.phase.
2. Once that started reporting real status instead of masking it,
pg-authentik showed a genuine problem: CNPG's operator couldn't
reach its Postgres instances' status API (port 8000) - "Cannot
extract Pod status ... context deadline exceeded" - because:
a) authentik-ingress's CiliumNetworkPolicy never allowed cnpg-system
as a source (crowdsec-ingress already had this exception,
authentik-ingress was just missing it - inconsistency, not
deliberate).
b) Even after fixing (a), still blocked - pg-authentik's pods are
ambient-mesh-enrolled, so the connection actually goes through
ztunnel's HBONE tunnel (port 15008) first, same underlying issue
as the argocd-redis fix from earlier today. Rather than keep
finding and patching this per-namespace as it recurs, added a
cluster-wide CiliumClusterwideNetworkPolicy allowing HBONE
broadly - ztunnel's own mTLS/SPIFFE identity verification is the
real security boundary for mesh traffic; Cilium blocking the
tunnel port itself was only breaking legitimate traffic, not
adding meaningful protection on top.
Verified live: CNPG operator immediately went from failing status
extraction to successfully reconciling (recreating a pod to reattach
its PVC) once both fixes were in place.
apps/nextcloud/values.yaml had oidc_login_client_id/secret as literal
empty strings, causing "client_id is missing or invalid" on login.
The matching Authentik blueprint (infrastructure/authentik/nextcloud-
blueprint.yaml) already defines fixed credentials for this OAuth2
provider - just never got copied over. Filled in to match.
Also found and left alone: an orphaned nextcloud-oidc-config ConfigMap
(from January) that already had the correct values but isn't actually
mounted by the current Deployment (which uses nextcloud-config instead)
- harmless dead resource, not the active config path.
- ZONE_NAME was "kube.huskypup.net" in both the home-ip-ddns CronJob and
the VPS failover watcher - that's a record, not a Cloudflare zone (the
actual zone is the parent "huskypup.net"). Caused home-ip-ddns to fail
every run (zone lookup returned zero results, curl -f exit 22) -
confirmed live and fixed.
- Added seccompProfile/non-root/dropped-capabilities securityContext to
the three CronJobs added this session that were missing it (flagged by
the cluster's "restricted" PodSecurity admission). Repointed the raft
snapshot job's mc binary install from /usr/local/bin to /tmp so it
still works running as non-root.
Root cause (confirmed via ztunnel logs): the argo-cd chart's default
NetworkPolicies only allow each component's application port (e.g. 6379
for redis), but Istio ambient mode routes ALL pod-to-pod traffic through
ztunnel's HBONE tunnel on port 15008 first - so the tunnel itself was
being blocked even though the "real" port was allowed. Every inter-pod
connection in the argocd namespace hung for exactly 10s then reset;
ztunnel's own log named it directly ("connection timed out, maybe a
NetworkPolicy is blocking HBONE port 15008"). This broke argocd-server's
Redis-backed session/cluster-info caching cluster-wide and was silently
preventing the root Application from picking up new child Applications.
Fix: additive NetworkPolicy allowing ingress on 15008 for all argocd
pods (NetworkPolicies union across multiple policies selecting a pod,
so this doesn't touch/replace the chart's rendered ones - safe across
Helm upgrades).
Also: dropped argocd from the namespace-enrollment job's waypoint list.
argocd's only AuthorizationPolicy (allow-argocd-access) is a plain
source-namespace/IP match with no L7 rules - its own status shows
"attached to ztunnel", not waypoint - so forcing L7 waypoint processing
onto the namespace was unnecessary overhead, not a security requirement.
ztunnel's mTLS still fully covers it. (This was a red herring for the
HBONE bug itself, not the fix, but a valid simplification found along
the way.)
Foundation for a DR/backup path using an always-on VPS as a second
ArgoCD-managed cluster, plus DB/backup standardization work that fell
out of it:
- vps-standby ArgoCD cluster destination + AppProject, MinIO backup
receiver, VPS bootstrap script (k3s, Netbird, cert-manager)
- Dual-site DNS failover watcher + home-IP DDNS CronJob, Cloudflare
token moved out of git into Vault+ExternalSecret
- Nextcloud migrated from ad-hoc MariaDB to CNPG + redis-operator
(matches n8n/Authentik/GitLab's backup-native pattern)
- Authentik's CNPG manifests moved into the actual ArgoCD-synced
manifests/ path (were present but never wired into the sync path)
- Vault raft-snapshot CronJob, CNPG barmanObjectStore backups
(Authentik/n8n/Nextcloud), Nextcloud file-PVC restic sync - all
targeting the new VPS MinIO receiver
See VPS Warm-Standby plan doc for full design rationale.
Gitaly CPU limit was 250m causing throttling on git operations (485ms
p99 to webservice). Increased to 1 core with 100m request. Also added
workerProcesses: 3 to webservice for better request concurrency
(was default 2).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Traffic goes through Istio (not Envoy Gateway), so cookie-based
consistent hashing must be on the Istio DestinationRule. Removes
the unused Envoy Gateway BackendTrafficPolicy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fixes OIDC nonce validation failures with 2 replicas by using a
BackendTrafficPolicy with cookie-based consistent hashing instead of
Service-level ClientIP affinity (which doesn't work behind Envoy proxy).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
With 2 replicas, OIDC callbacks can hit a different pod than the
one that generated the state/nonce, causing auth failures. Session
affinity ensures the same client always reaches the same pod.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change EXTENSION_PRIORITY from "*,openid" to "openid,*" so
unauthenticated users are redirected to Authentik OIDC instead
of seeing the database login form.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fixes NameError: uninitialized constant Gitlab::Redis::ALL_CLASSES
caused by 7_prometheus_metrics.rb loading before 7_redis.rb
(alphabetically 'p' < 'r'). The ConfigMap provides a 0_redis_early.rb
initializer that requires gitlab/redis before other initializers run.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Scale CNPG pg-teslamate to 1 instance: on-disk storage corruption
(corrupted sequences, system catalog indexes) means replicas cloned
via pg_basebackup inherit corruption and can't open any database.
Primary works from cached catalogs. Sequences already rebuilt live.
- Remove talos-cp-02 from Rook-Ceph nodes list: ceph-volume inventory
hangs on BIOS partition via nsenter lvs in host namespace. Talos
read-only /etc prevents LVM filter fix. OSD 1 keeps running from
its existing deployment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Home Assistant was returning 400 errors because it received
X-Forwarded-For headers from the Istio ingress gateway but wasn't
configured to trust reverse proxies. Add use_x_forwarded_for and
trusted_proxies to the init container's configuration.yaml template.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- n8n: Set replicaCount=1 (RWO PVC incompatible with multiple replicas),
add resource limits to satisfy Kyverno policy
- gitlab: Add ignoreDifferences for redis-gitlab-additional service
(port names and selectors managed by Redis operator)
- netbird-operator: Add ExternalSecret for netbird-mgmt-api-key in
netbird namespace and add manifests source to Application
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Webservice was at 94% memory (1875Mi/2Gi) with only 50m CPU request,
causing CPU throttling and slow first requests after idle periods.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
s6-overlay's log-prepare needs chmod on /dev/shm/logs/* which
requires CAP_FOWNER when all capabilities are dropped.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Revert talos-client-cert secret to prometheus namespace (prometheus
pods mount this secret, not kube-system pods)
- Add CHOWN/DAC_OVERRIDE/SETUID/SETGID capabilities to frigate
(s6-overlay needs chown for log directories)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scooby Husky