Root cause (confirmed via ztunnel logs): the argo-cd chart's default
NetworkPolicies only allow each component's application port (e.g. 6379
for redis), but Istio ambient mode routes ALL pod-to-pod traffic through
ztunnel's HBONE tunnel on port 15008 first - so the tunnel itself was
being blocked even though the "real" port was allowed. Every inter-pod
connection in the argocd namespace hung for exactly 10s then reset;
ztunnel's own log named it directly ("connection timed out, maybe a
NetworkPolicy is blocking HBONE port 15008"). This broke argocd-server's
Redis-backed session/cluster-info caching cluster-wide and was silently
preventing the root Application from picking up new child Applications.
Fix: additive NetworkPolicy allowing ingress on 15008 for all argocd
pods (NetworkPolicies union across multiple policies selecting a pod,
so this doesn't touch/replace the chart's rendered ones - safe across
Helm upgrades).
Also: dropped argocd from the namespace-enrollment job's waypoint list.
argocd's only AuthorizationPolicy (allow-argocd-access) is a plain
source-namespace/IP match with no L7 rules - its own status shows
"attached to ztunnel", not waypoint - so forcing L7 waypoint processing
onto the namespace was unnecessary overhead, not a security requirement.
ztunnel's mTLS still fully covers it. (This was a red herring for the
HBONE bug itself, not the fix, but a valid simplification found along
the way.)
Foundation for a DR/backup path using an always-on VPS as a second
ArgoCD-managed cluster, plus DB/backup standardization work that fell
out of it:
- vps-standby ArgoCD cluster destination + AppProject, MinIO backup
receiver, VPS bootstrap script (k3s, Netbird, cert-manager)
- Dual-site DNS failover watcher + home-IP DDNS CronJob, Cloudflare
token moved out of git into Vault+ExternalSecret
- Nextcloud migrated from ad-hoc MariaDB to CNPG + redis-operator
(matches n8n/Authentik/GitLab's backup-native pattern)
- Authentik's CNPG manifests moved into the actual ArgoCD-synced
manifests/ path (were present but never wired into the sync path)
- Vault raft-snapshot CronJob, CNPG barmanObjectStore backups
(Authentik/n8n/Nextcloud), Nextcloud file-PVC restic sync - all
targeting the new VPS MinIO receiver
See VPS Warm-Standby plan doc for full design rationale.
RollingUpdate with a ReadWriteOnce PVC causes a Multi-Attach deadlock
where the new pod can't start because the old pod still holds the
volume. Recreate strategy ensures the old pod is terminated before
the new one is created.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The operator watches for netbird.io/expose (presence-based), not
netbird.io/resource which was from the old Helmfile bootstrap script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The operator was missing managementURL, causing it to default to the
NetBird SaaS cloud (api.netbird.io) instead of the self-hosted instance.
This resulted in an infinite reconcile loop and no networks/resources
being created.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Includes security fixes (CVE-2023-43040) and RBD/RADOS improvements.
Rook operator upgraded to v1.19.2 via Helm.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
LVM filter now applied via Talos machine config on all nodes to prevent
lvs hang on BIOS partitions. talos-cp-02 can safely run OSD prepare jobs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Scale CNPG pg-teslamate to 1 instance: on-disk storage corruption
(corrupted sequences, system catalog indexes) means replicas cloned
via pg_basebackup inherit corruption and can't open any database.
Primary works from cached catalogs. Sequences already rebuilt live.
- Remove talos-cp-02 from Rook-Ceph nodes list: ceph-volume inventory
hangs on BIOS partition via nsenter lvs in host namespace. Talos
read-only /etc prevents LVM filter fix. OSD 1 keeps running from
its existing deployment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
X-Forwarded-Proto approach didn't work - Ceph dashboard SAML doesn't
read it. Re-enabling native SSL so the dashboard sees https:// URLs
directly. Updated both Istio and Envoy Gateway backends for TLS with
insecureSkipVerify for the self-signed cert.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dashboard runs with ssl: false behind TLS-terminating proxy, so SAML
library sees http:// URLs and rejects responses expecting https://.
Explicitly set X-Forwarded-Proto: https on both Istio VirtualService
and Envoy Gateway HTTPRoute so the SAML ACS URL matches.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dashboard has ssl: false (plain HTTP on port 7000), but the DestinationRule
had tls.mode: SIMPLE which caused Istio to attempt TLS to the backend,
resulting in WRONG_VERSION_NUMBER errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 4 OSDs successfully recreated in clean raw mode (NVMe only, no DB devices).
Fixed talos-cp-02 NVMe device path from nvme0n1 to nvme1n1 to match actual hardware.
Removed temporary fix-rook-osd-metadata-device Kyverno policy as it's no longer needed.
Operator runs normally without manual patches.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The dashboard has SSL disabled and listens on port 7000. Restoring the
VirtualService with the correct port for the Istio gateway path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Services behind the Envoy gateway cannot reach Istio ambient mesh
backends due to STRICT mTLS. Reverting DNS to point back to the Istio
gateway until the migration path is resolved.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All namespaces with CiliumNetworkPolicies need envoy-gateway-system
access now that services are migrating from Istio to Envoy gateway.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allows external-dns to create DNS records from Gateway API HTTPRoute
resources, needed for services using the Envoy edge gateway.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The dashboard now uses the Envoy edge gateway via HTTPRoute. The old
Istio VirtualService was causing DNS to point at the wrong gateway IP.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Ceph dashboard moved from Istio to the Envoy edge gateway but the
CiliumNetworkPolicy was still only allowing istio-system, causing 503s.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The dashboard has SSL disabled and listens on port 7000, but the
Backend was configured for TLS on port 8443.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ArgoCD, NetBird operator, and Scylla Manager are upstream Helm charts
that don't set resource limits, probes, drop-all capabilities, or
disable SA token automount. These audit-mode policy violations inflated
the cumulative Prometheus failure counter to ~3800.
- argocd: exempt from require-resource-limits, require-probes
- netbird-operator: exempt from require-resource-limits,
disallow-automount-sa-token, require-probes
- scylla-manager: exempt from require-drop-all-capabilities,
disallow-automount-sa-token
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The global-authentik-authz AuthorizationPolicy uses *.kube.huskypup.net
as a host match, but Kiali flagged KIA0104 because the wildcard had no
matching entry in Istio's service registry. Adding a MESH_EXTERNAL
ServiceEntry with resolution NONE registers the hostname without
affecting routing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use the OSD-specific activate-osd volume mount instead of rook-data
parent directory. Multiple old OSD dirs exist on each node and the
script was creating the symlink in the wrong (stale LVM) directory.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use patchesJson6902 with append (/-) for create-block-db-symlink to
ensure it runs after activate, not before. The patchStrategicMerge was
inserting it before activate, causing prime-osd-dir to clear the symlink.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The netbird-operator webhook service runs in the netbird namespace and
receives admission webhook calls from kube-apiserver using node IPs
(172.28.101.0/24). Without this, ztunnel rejects the connections with
"allow policies exist, but none allowed".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove metadataDevice (sdb) from all 4 nodes in cluster-values.yaml
since the SATA SSDs are slower than the NVMe data drives. Add temporary
Kyverno mutate policy to protect existing OSD deployments when the
operator is re-enabled - replaces expand-bluefs with a no-op and adds
conditional block.db symlink creation based on BlueStore label detection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ED25519 PRIVATE KEY header is not recognized by nginx's OpenSSL.
Convert to standard PKCS#8 PRIVATE KEY header (same DER content).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kiali was reporting "Gateway API Class not found" errors for the
envoy-gateway Gateways (gateway/edge and guacamole/guacamole).
Explicitly register istio, istio-waypoint, and envoy-gateway classes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The ztunnel container exposes metrics on port named 'ztunnel-stats' (15020),
not 'http-monitoring'. This was preventing Prometheus from scraping ztunnel
L4 metrics, causing missing traffic in Kiali.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Secret was in prometheus namespace but the talos-metrics-proxy
deployment referencing it runs in kube-system namespace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- n8n: Set replicaCount=1 (RWO PVC incompatible with multiple replicas),
add resource limits to satisfy Kyverno policy
- gitlab: Add ignoreDifferences for redis-gitlab-additional service
(port names and selectors managed by Redis operator)
- netbird-operator: Add ExternalSecret for netbird-mgmt-api-key in
netbird namespace and add manifests source to Application
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bitnami/kubectl has removed all version tags from Docker Hub,
leaving only :latest which is blocked by Kyverno disallow-latest-tag
policy. Switch to alpine/k8s:1.32.13 which has kubectl, bash, jq,
and uses proper version tags.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The :1.31 tag is gone from Docker Hub and :latest is blocked by Kyverno
disallow-latest-tag policy. The 1.29 image is cached on cluster nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The bitnami/kubectl:1.31 tag doesn't exist on Docker Hub. Updated all
hook jobs to use bitnami/kubectl:latest.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PreSync/PostSync hook Jobs reference argocd-hook-sa but it only existed
in the argocd namespace. Jobs running in app namespaces (crowdsec,
authentik, netbird, vault, istio-system) need the SA in their own
namespace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Tetragon CRD defaults followChildren to false in matchBinaries,
causing ArgoCD ServerSideDiff to detect a permanent diff.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
BlueStore has UUID inconsistency at block position 0x0 that causes
expand_devices assertion in both v19.2.0 and v19.2.3. The OSD
daemon itself runs fine; only expand-bluefs init container crashes.
Will use deployment patches with do-not-reconcile annotation instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Revert talos-client-cert secret to prometheus namespace (prometheus
pods mount this secret, not kube-system pods)
- Add CHOWN/DAC_OVERRIDE/SETUID/SETGID capabilities to frigate
(s6-overlay needs chown for log directories)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Ceph v19.2.0 has BlueStore::expand_devices assertion bug that
crashes OSD expand-bluefs init containers. v19.2.3 includes fixes.
- NetBird operator webhook failurePolicy: Fail blocks all pod
creation cluster-wide when webhook service is unreachable.
Changed to Ignore.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
