Gitaly CPU limit was 250m causing throttling on git operations (485ms
p99 to webservice). Increased to 1 core with 100m request. Also added
workerProcesses: 3 to webservice for better request concurrency
(was default 2).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RollingUpdate with a ReadWriteOnce PVC causes a Multi-Attach deadlock
where the new pod can't start because the old pod still holds the
volume. Recreate strategy ensures the old pod is terminated before
the new one is created.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
NetBird router deployments require NET_ADMIN capability, which is
blocked by the default baseline PodSecurity standard. Add
managedNamespaceMetadata with pod-security.kubernetes.io/enforce:
privileged to argocd, frigate, gitlab, grafana, guacamole,
home-assistant, teslamate, and vault ArgoCD Applications.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The operator watches for netbird.io/expose (presence-based), not
netbird.io/resource which was from the old Helmfile bootstrap script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The operator was missing managementURL, causing it to default to the
NetBird SaaS cloud (api.netbird.io) instead of the self-hosted instance.
This resulted in an infinite reconcile loop and no networks/resources
being created.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Traffic goes through Istio (not Envoy Gateway), so cookie-based
consistent hashing must be on the Istio DestinationRule. Removes
the unused Envoy Gateway BackendTrafficPolicy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fixes OIDC nonce validation failures with 2 replicas by using a
BackendTrafficPolicy with cookie-based consistent hashing instead of
Service-level ClientIP affinity (which doesn't work behind Envoy proxy).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
With 2 replicas, OIDC callbacks can hit a different pod than the
one that generated the state/nonce, causing auth failures. Session
affinity ensures the same client always reaches the same pod.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change EXTENSION_PRIORITY from "*,openid" to "openid,*" so
unauthenticated users are redirected to Authentik OIDC instead
of seeing the database login form.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fixes NameError: uninitialized constant Gitlab::Redis::ALL_CLASSES
caused by 7_prometheus_metrics.rb loading before 7_redis.rb
(alphabetically 'p' < 'r'). The ConfigMap provides a 0_redis_early.rb
initializer that requires gitlab/redis before other initializers run.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Includes security fixes (CVE-2023-43040) and RBD/RADOS improvements.
Rook operator upgraded to v1.19.2 via Helm.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
LVM filter now applied via Talos machine config on all nodes to prevent
lvs hang on BIOS partitions. talos-cp-02 can safely run OSD prepare jobs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Scale CNPG pg-teslamate to 1 instance: on-disk storage corruption
(corrupted sequences, system catalog indexes) means replicas cloned
via pg_basebackup inherit corruption and can't open any database.
Primary works from cached catalogs. Sequences already rebuilt live.
- Remove talos-cp-02 from Rook-Ceph nodes list: ceph-volume inventory
hangs on BIOS partition via nsenter lvs in host namespace. Talos
read-only /etc prevents LVM filter fix. OSD 1 keeps running from
its existing deployment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
X-Forwarded-Proto approach didn't work - Ceph dashboard SAML doesn't
read it. Re-enabling native SSL so the dashboard sees https:// URLs
directly. Updated both Istio and Envoy Gateway backends for TLS with
insecureSkipVerify for the self-signed cert.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dashboard runs with ssl: false behind TLS-terminating proxy, so SAML
library sees http:// URLs and rejects responses expecting https://.
Explicitly set X-Forwarded-Proto: https on both Istio VirtualService
and Envoy Gateway HTTPRoute so the SAML ACS URL matches.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dashboard has ssl: false (plain HTTP on port 7000), but the DestinationRule
had tls.mode: SIMPLE which caused Istio to attempt TLS to the backend,
resulting in WRONG_VERSION_NUMBER errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 4 OSDs successfully recreated in clean raw mode (NVMe only, no DB devices).
Fixed talos-cp-02 NVMe device path from nvme0n1 to nvme1n1 to match actual hardware.
Removed temporary fix-rook-osd-metadata-device Kyverno policy as it's no longer needed.
Operator runs normally without manual patches.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The dashboard has SSL disabled and listens on port 7000. Restoring the
VirtualService with the correct port for the Istio gateway path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Services behind the Envoy gateway cannot reach Istio ambient mesh
backends due to STRICT mTLS. Reverting DNS to point back to the Istio
gateway until the migration path is resolved.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All namespaces with CiliumNetworkPolicies need envoy-gateway-system
access now that services are migrating from Istio to Envoy gateway.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allows external-dns to create DNS records from Gateway API HTTPRoute
resources, needed for services using the Envoy edge gateway.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The dashboard now uses the Envoy edge gateway via HTTPRoute. The old
Istio VirtualService was causing DNS to point at the wrong gateway IP.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Network policies in the network-policies/ subdirectory were not being
synced because ArgoCD was not recursing into subdirectories.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Ceph dashboard moved from Istio to the Envoy edge gateway but the
CiliumNetworkPolicy was still only allowing istio-system, causing 503s.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The dashboard has SSL disabled and listens on port 7000, but the
Backend was configured for TLS on port 8443.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ArgoCD, NetBird operator, and Scylla Manager are upstream Helm charts
that don't set resource limits, probes, drop-all capabilities, or
disable SA token automount. These audit-mode policy violations inflated
the cumulative Prometheus failure counter to ~3800.
- argocd: exempt from require-resource-limits, require-probes
- netbird-operator: exempt from require-resource-limits,
disallow-automount-sa-token, require-probes
- scylla-manager: exempt from require-drop-all-capabilities,
disallow-automount-sa-token
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The global-authentik-authz AuthorizationPolicy uses *.kube.huskypup.net
as a host match, but Kiali flagged KIA0104 because the wildcard had no
matching entry in Istio's service registry. Adding a MESH_EXTERNAL
ServiceEntry with resolution NONE registers the hostname without
affecting routing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The scylla-manager Application was missing the third source for raw
manifests, so the VirtualService in infrastructure/scylla/manifests/
was never deployed. This caused Kiali to flag the ext-authz policy
host as having no matching service entry.
Also removed stale PERMISSIVE PeerAuthentication from authentik
namespace (manually applied, not in git, unnecessary with mesh-wide
STRICT mTLS).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use the OSD-specific activate-osd volume mount instead of rook-data
parent directory. Multiple old OSD dirs exist on each node and the
script was creating the symlink in the wrong (stale LVM) directory.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use patchesJson6902 with append (/-) for create-block-db-symlink to
ensure it runs after activate, not before. The patchStrategicMerge was
inserting it before activate, causing prime-osd-dir to clear the symlink.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The netbird-operator webhook service runs in the netbird namespace and
receives admission webhook calls from kube-apiserver using node IPs
(172.28.101.0/24). Without this, ztunnel rejects the connections with
"allow policies exist, but none allowed".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove metadataDevice (sdb) from all 4 nodes in cluster-values.yaml
since the SATA SSDs are slower than the NVMe data drives. Add temporary
Kyverno mutate policy to protect existing OSD deployments when the
operator is re-enabled - replaces expand-bluefs with a no-op and adds
conditional block.db symlink creation based on BlueStore label detection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Home Assistant was returning 400 errors because it received
X-Forwarded-For headers from the Istio ingress gateway but wasn't
configured to trust reverse proxies. Add use_x_forwarded_for and
trusted_proxies to the init container's configuration.yaml template.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ED25519 PRIVATE KEY header is not recognized by nginx's OpenSSL.
Convert to standard PKCS#8 PRIVATE KEY header (same DER content).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kiali was reporting "Gateway API Class not found" errors for the
envoy-gateway Gateways (gateway/edge and guacamole/guacamole).
Explicitly register istio, istio-waypoint, and envoy-gateway classes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The ztunnel container exposes metrics on port named 'ztunnel-stats' (15020),
not 'http-monitoring'. This was preventing Prometheus from scraping ztunnel
L4 metrics, causing missing traffic in Kiali.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Secret was in prometheus namespace but the talos-metrics-proxy
deployment referencing it runs in kube-system namespace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- n8n: Set replicaCount=1 (RWO PVC incompatible with multiple replicas),
add resource limits to satisfy Kyverno policy
- gitlab: Add ignoreDifferences for redis-gitlab-additional service
(port names and selectors managed by Redis operator)
- netbird-operator: Add ExternalSecret for netbird-mgmt-api-key in
netbird namespace and add manifests source to Application
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Gateway API CRDs are already managed by istio; skip them
from the vendored envoy-gateway chart to prevent OutOfSync.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Webservice was at 94% memory (1875Mi/2Gi) with only 50m CPU request,
causing CPU throttling and slow first requests after idle periods.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The checksum/lapi-secret annotation on the LAPI Deployment changes
when CrowdSec auto-generates credentials, causing permanent
OutOfSync with ArgoCD ServerSideDiff.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ArgoCD v3.3.3 has a bug where OCI chart revision resolution
doesn't include the chart name in the Docker Hub API path,
causing 401 errors. The chart is already deployed at v1.6.3.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ArgoCD requires the chart field for OCI sources. The repoURL
contains the registry/org path, and chart specifies the chart name.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move chart name into repoURL for OCI registry compatibility.
ArgoCD resolves OCI chart references differently from HTTP repos.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bitnami/kubectl has removed all version tags from Docker Hub,
leaving only :latest which is blocked by Kyverno disallow-latest-tag
policy. Switch to alpine/k8s:1.32.13 which has kubectl, bash, jq,
and uses proper version tags.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
