Found while building the VPS replica clusters (Phase 2): all three
CNPG clusters' backup.barmanObjectStore blocks only configure continuous
WAL archiving. CNPG's bootstrap.recovery needs at least one real Backup
object in the object store before it has anything to restore -
'no target backup found', confirmed live against pg-authentik. This gap
was systemic (gitlab has it too, not fixed here - out of scope for
tonight, no VPS standby depends on it).
immediate: true fires one backup right away in addition to the daily
01:00 schedule, to unblock the in-progress VPS restore now rather than
waiting up to 24h for the first scheduled run.
Without hook annotations they land in ArgoCD's regular Sync phase, which
runs AFTER PreSync hooks - so the presync Job's pod couldn't be created
('serviceaccount not found', confirmed live). Weight -1 vs the Job's 0
gets them created first, within the same PreSync phase.
Same CNPG replica-cluster pattern as Authentik (continuous WAL replay
from home via VPS MinIO, app at 0 replicas until manual promotion - see
infrastructure/vps-standby/authentik/manifests/cnpg-cluster.yaml for the
full rationale).
n8n: N8N_ENCRYPTION_KEY copied byte-identical from home (kubectl, not
git) - decrypts stored credentials in the replicated DB, same reasoning
as Vault's unseal key / root token copies.
Nextcloud: adds infrastructure/vps-standby/nextcloud/manifests/
pvc-restore-cronjob.yaml, the read side of home's existing
nextcloud-pvc-sync restic backup - restores the latest snapshot from VPS
MinIO into this standby's PVC daily at 04:00 (2h after home's 02:00
backup). No Redis on the VPS side (no redis-operator deployed there,
out of scope for a standby that isn't serving traffic - Nextcloud
degrades gracefully to DB-based locking without it).
CNPG operator on the VPS (argocd-apps/vps-standby/cnpg-operator.yaml),
plus a CNPG 'replica cluster' for pg-authentik (infrastructure/
vps-standby/authentik/manifests/cnpg-cluster.yaml) that continuously
replays WAL shipped from home's pg-authentik via VPS MinIO's
cnpg-backups/pg-authentik bucket path - CNPG's native replica-cluster
mode, not a custom restore script like Vault needed (Vault has no
equivalent built-in continuous-replication-into-object-store feature).
The Authentik app itself runs at replicas: 0 in normal operation - the
replica DB is read-only until a deliberate manual promotion
(spec.replica.enabled: false), and a pod trying to write against a
read-only DB would just crashloop uselessly. values.yaml's authentik:
block is copied byte-identical from home's on purpose: the chart
deterministically derives its generated Secret's AUTHENTIK_SECRET_KEY
from these literal values.yaml strings, so both clusters land on the
same key without manually copying it - required since that key decrypts
things stored encrypted in the replicated DB.
service=gitlab invokes Gitea's GitLab-API-based downloader (for issues/
PRs/releases metadata), which calls the source's /api/v4/projects/...
REST API rather than doing a plain git clone. That was 404ing against
home GitLab and getting swallowed into a generic 'InternalServerError:
404 Not Found' with no further detail. We only want a plain pull-mirror
of the git repo itself, so service=git forces the plain git-clone
downloader instead.
Pull-mirrors home GitLab's Homelabv4 repo on Gitea's own built-in mirror
scheduler (6h interval) - no custom sync job needed, per the original
plan. SQLite instead of the chart's default HA Postgres + Valkey cluster
(single-instance standby holding one small repo, not worth the extra
moving parts). A PostSync Job creates the mirror once, idempotently;
Gitea's scheduler handles all ongoing pulls after that.
Also added a public Cloudflare CNAME for gitlab.kube.huskypup.net ->
home.kube.huskypup.net: the VPS has no route to home's LAN via Netbird
(none of the mesh peers advertise that subnet, confirmed live), so
GitLab needs to be reachable the same way any other internet client
reaches it - home's public IP already has port 443 forwarded to
istio-ingressgateway from earlier this session.
Admin and GitLab-mirror credentials are plain Secrets created directly
on the VPS cluster (kubectl, not git) - same pattern as vault-unseal-key.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed end-to-end tonight: condition=Ready correctly times out every
cycle since Vault can't be Ready while sealed (the unseal step comes
right after this wait) - harmless via the existing || true fallback, but
wastes up to 2 minutes per restore cycle waiting on a condition that can
never be met at this point. Poll for the container process merely being
started instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Without it, kubectl wait's internal watch retry loop never terminates
cleanly even with --timeout set - confirmed live, it kept retrying well
past the stated 120s timeout. The restore/unseal steps themselves still
succeed regardless (verified end-to-end with a manual unseal), this only
affects the job's own graceful detection of the pod coming back.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
raft snapshot restore is a privileged operation - unsealing alone isn't
enough, it needs an authenticated token. Missed this on first pass, caught
live: 'Code: 403. Errors: * permission denied'.
First run: uses the throwaway init's own fresh root token. Steady state
(already restored at least once): uses a copy of home's real root token,
which becomes valid on this Vault the moment the first restore completes
(its auth data becomes byte-for-byte home's as of that snapshot). Stored
the same way as the unseal key - kubectl directly on the VPS, not git.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
StatefulSet pod naming follows the Helm release name (vps-vault, matching
the ArgoCD Application name), not the chart's default.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Single-node Vault (raft storage, standalone mode) plus a restore CronJob
that pulls the latest raft snapshot from the VPS's own local MinIO
(already receiving snapshots every 6h from home) and restores it every
6h, offset 30 minutes after the home-side snapshot job.
Not an independent root of trust: raft snapshot restore replaces the
entire storage backend including the keyring, so after every restore
this Vault is sealed with HOME's actual keyring - unsealing it needs a
copy of home's real unseal key, stored directly on the VPS cluster
(kubectl, not git, same pattern as vps-minio-root-secret). First-run
bootstrap uses a throwaway single-Shamir-key init just to get through
the very first restore, then is irrelevant from then on.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Without -L, curl silently saved dl.min.io's redirect response body
(HTML/text) as the mc binary instead of the actual binary, failing at
runtime with a confusing shell-script-interpretation error. Confirmed
live while testing the leader-detection fix end-to-end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
rootUser/rootPassword were left unset, relying on the chart's
auto-generation - but ArgoCD renders Helm via 'helm template', which
doesn't support the lookup() function charts use to preserve an existing
generated secret across upgrades. With selfHeal: true, every single sync
generated a BRAND NEW random root password, immediately invalidating
whatever credentials were stored in the home cluster's Vault for the
backup/CronJob consumers (vault-raft-snapshot, 3x CNPG barmanObjectStore,
nextcloud-pvc-sync) - confirmed live: all of them failing with 'Access
Key Id does not exist' right after being fixed, because the password had
already rotated out from under them again.
Fixed by pointing at a pre-existing Secret created directly on the VPS
cluster (kubectl, not git - this cluster has no Vault/ESO of its own).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Now that the endpoint/credentials are fixed and this job can actually
reach the VPS, it surfaced two more real bugs: no fsGroup (couldn't read
the PVC data at all - nextcloud's real Deployment uses fsGroup 33/www-data,
this job never matched it) and no writable cache dir for runAsUser 65534
(restic defaults to $HOME/.cache).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Repeated restarts from tonight's network disruption kept OOMKilling
rancher - each restart's startup reconciliation is memory-hungry, and 2Gi
wasn't enough headroom for that burst even though steady-state usage is
normally fine. Nodes have ample free memory (26-36% used).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
redis.enabled: true was unintentionally deploying the chart's BUNDLED redis
subchart (redis-21.1.3, StatefulSet nextcloud-redis-replicas) alongside
the actual redis-operator instance - that key's schema is pure subchart
passthrough (auth/image/master/replica), it has no host field. The
host/existingSecret keys previously set there were silently ignored.
Found while investigating why nextcloud-redis-replicas-0 was
crash-looping on failed liveness probes (confirmed via : the correct key for pointing at an external Redis
is the separate externalRedis: block).
This means Nextcloud's config.php may never have been correctly pointed
at the redis-operator instance despite apps/nextcloud/manifests/redis-cr.yaml
existing and being healthy - worth confirming file-locking/caching
actually engages after this deploys.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This fix for the EPERM ClusterIP-from-hostNetwork issue (Cilium socketLB
hostNamespaceOnly + hostNetwork bouncer + ClusterIP VIP) was made and
verified earlier but never committed - ArgoCD's selfHeal kept reverting
the live ConfigMap back to the broken crowdsec-service URL against git's
stale version, causing a fresh bouncer pod to crash-loop with the original
EPERM error again.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause of tonight's earlier CrowdSec/CNPG-backup workarounds: no node
in the cluster had any route into the Netbird mesh CIDR (100.108.0.0/16)
for pod-originated traffic. The per-namespace netbird 'router' pods are
inbound-only infrastructure (external peers reaching K8s services); their
own architecture has no reverse path.
Fix, in two parts:
1. infrastructure/netbird/manifests/egress-daemonset.yaml - one netbird
client per node, hostNetwork so its wt0 interface lives in the node's
real network namespace, plus a sidecar that adds a host route sending
100.108.0.0/16 out via it. hostNetwork requires a scoped Kyverno
PolicyException (infrastructure/kyverno/policies/netbird-egress-exception.yaml)
to the disallow-host-namespaces STIG policy - narrowly for this one
DaemonSet by name, not a namespace-wide exclusion.
2. Discovered the route alone wasn't enough for k3s NodePort traffic
(vps-minio:30900): Netbird manages its own nftables ACLs independent of
iptables/Kyverno, and its forward chain (netbird-rt-fwd) only permits
*established* connections through a peer acting as a router - never new
ones, by design, unless a Netbird 'Network Route' policy is explicitly
configured (it isn't, for this VPS). Locally-terminated connections
(tinyproxy) go through a separate, already-permissive ACL chain, which
is why the CrowdSec proxy fix from earlier tonight worked. Replicated
that working pattern for MinIO: minio-forward.service on the VPS host
(systemd, socat) forwards 100.108.113.41:9000 -> MinIO's ClusterIP,
avoiding the NodePort path entirely.
Re-enabled everything that was disabled/suspended earlier tonight because
of this gap, pointed at the new endpoint:
- CrowdSec CAPI/console-enroll (removed DISABLE_ONLINE_API, restored the
VPS proxy env vars)
- n8n/nextcloud/authentik CNPG backup.barmanObjectStore
- vault-raft-snapshot CronJob (unsuspended)
- nextcloud PVC content sync CronJob endpoint
vps-minio.netbird.internal is retired everywhere - it was never actually
resolvable (Netbird has no DNS configured) even before today's routing
fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
vault operator raft snapshot save fails against a standby Vault node with
'incomplete snapshot, unable to read SHA256SUMS.sealed file' - it must run
directly against the active HA leader. The job was hardcoded to vault-0,
which has been a standby since boot, so it had been failing on every run.
Fixed to detect the actual leader at runtime via vault status is_self.
Also suspended the CronJob: the upload step targets
vps-minio.netbird.internal, unreachable from any pod in this cluster (same
missing egress-route gap as tonight's CrowdSec/CNPG-backup fixes). The
leader-detection fix is kept since it's correct and independent - just
unsuspend once real egress routing exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
n8n's allow-n8n-access AuthorizationPolicy only allowed traffic from the
n8n/istio-system/prometheus namespaces - unlike nextcloud/authentik's
policies, it was missing the broader ipBlocks: 10.244.0.0/16 fallback
rule. The CNPG operator (cnpg-system namespace) polling pg-n8n instances'
status API (port 8000) wasn't covered by any rule, so Istio's L4
authorization enforcement (via ztunnel - this traffic is direct pod-IP,
not routed through a Service, so waypoint/L7 never sees it) rejected the
connection outright, surfacing as a TCP reset to the operator.
This was the actual cause of pg-n8n's long-recurring 'Instance Status
Extraction Error' - confirmed 100% reproducible tonight (every single
operator poll attempt failed identically), not the intermittent
HBONE-tunnel issue it resembled from the outside. Also labeled the n8n
namespace istio.io/use-waypoint=waypoint (was in the enrollment script's
namespace list but the label was missing live - namespace parity fix,
though not itself the fix for this specific issue since this traffic
bypasses the waypoint).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Discovered while debugging pg-n8n's recurring 'Instance Status Extraction
Error': the root cause is the same missing pod-egress route to the Netbird
mesh found tonight while fixing CrowdSec - continuousArchiving kept failing
to connect to vps-minio.netbird.internal, and CNPG correctly holds
Ready=False while archiving is broken (a real condition, not cosmetic).
That's what was driving these three apps' ArgoCD health flapping.
No node in the cluster has any route into 100.108.0.0/16 for
pod-originated traffic - the per-namespace netbird 'router' pods
(gitlab, vault, argocd, etc.) are inbound-only infrastructure, not egress
gateways. gitlab's own CNPG backup is unaffected (points at a local
in-cluster MinIO, not the VPS).
Commented out rather than deleted - re-enable once real pod-egress
routing exists, tracked as a separate task. No data loss: this is WAL
archiving/backup, not the live database.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
LAPI treated a failed CAPI/console-enroll attempt as fatal at startup,
crashing the whole server (and taking agent + bouncer down with it via
their LAPI dependency) on every restart. Root cause investigation tonight
found two separate real issues:
1. Home's public IP gets a 403 from AWS WAF in front of api.crowdsec.net
on any request, unrelated to credentials - ENROLL_KEY is valid and
correctly wired but can't get past this.
2. Attempted routing around it via a VPS-side proxy over Netbird; found no
node in the cluster has any route into the Netbird mesh CIDR for
pod-originated egress traffic - the per-namespace netbird router pods
(gitlab, vault, argocd, etc.) are inbound-only infrastructure. A proper
fix needs either a Talos-native static route (fragile against pod IP
churn) or a deliberate STIG exception for hostNetwork - out of scope
for tonight, tracked separately.
DISABLE_ONLINE_API=true lets LAPI start cleanly with local-only detection
(agent + bouncer functional again). ENROLL_KEY stays set in Vault/values -
flip this back once real egress routing exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every other major namespace (gitlab, vault, argocd, nextcloud, etc.) has
its own 3-replica netbird-router deployment auto-provisioned from an
NBRoutingPeer CR; crowdsec never got one since it never previously needed
mesh connectivity. Without it, crowdsec pods had no dedicated mesh routing
path, causing intermittent connection resets/hangs reaching the VPS proxy
used to route around CrowdSec's WAF block on home's public IP.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed 2026-08-17: home's public IP gets a 403 ForbiddenException from
AWS WAF in front of api.crowdsec.net, even on a bare unauthenticated test
request - the VPS's IP gets a normal 401 on the identical request. This is
an IP-level block unrelated to credentials, which is why LAPI kept hitting
403 on both anonymous auto-registration and console enroll with a fresh,
valid enrollment key.
Fix: HTTPS_PROXY/HTTP_PROXY pointed at a tinyproxy instance on the VPS
(bound to its Netbird interface only, not internet-reachable), reachable
from home over the existing Netbird mesh via netbird-cluster-router.
cscli/crowdsec's Go HTTP client honors standard proxy env vars.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
allow-netbird-operator-access (AuthorizationPolicy) and the netbird-operator
PeerAuthentication both targeted namespace 'netbird-operator', which has
never existed - the actual netbird-operator pod runs in the 'netbird'
namespace. This permanently blocked istio-mesh-config from going Synced
(kubectl apply failed: namespaces "netbird-operator" not found on every
sync attempt).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
vault-init-and-store.sh minted the auth/kubernetes/config reviewer JWT via
'kubectl create token vault-auth' with no --duration, defaulting to a 1hr
TTL, then wrote it into Vault as a static token_reviewer_jwt. ~1hr after
every cluster boot/hook rerun it silently expired, breaking TokenReview
for every kubernetes-auth login (including ESO's) with a generic 403 that
logged nothing at INFO/ERROR. This was the actual root cause of today's
broad ArgoCD Degraded wave across ~14 apps - not Vault's seal state, which
was fine the whole time.
Fix: clear token_reviewer_jwt and rely on Vault's local-JWT auto-detection
(disable_local_ca_jwt=false default), which reads the vault pod's own
kubelet-refreshed SA token from disk on every call instead of a static
copy. The vault SA already has system:auth-delegator via the existing
vault-server-binding ClusterRoleBinding.
Applied live directly against Vault to unblock immediately; confirmed
ClusterSecretStore vault-backend flipped to Ready=True and previously
broken ExternalSecrets (guacamole, teslamate, unpoller, netbird x3) all
resynced successfully.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
redis-nextcloud-0 was CrashLoopBackOff: "mkdir: can't create directory
'/node-conf': Permission denied" and same for appendonlydir - the pod
had no podSecurityContext, so it couldn't write to its own PVC-backed
data directories. Added fsGroup/runAsUser: 1000 matching the
opstree/redis image's default non-root user. Verified live: Redis now
starts cleanly ("Ready to accept connections").
apps/nextcloud/manifests/redis-cr.yaml used redis.redis.opstreelabs.in/v1beta2
(guessed, flagged as unverified when written) - the actually-installed CRD
only has v1beta1. ArgoCD validates all resource API groups/versions before
syncing anything, so this one bad apiVersion blocked the ENTIRE nextcloud
Application from syncing at all (every resource showed OutOfSync, not just
this one).
Also fixed against the live CRD schema: redisSecret belongs nested under
kubernetesConfig, not top-level, and redisExporter.image is required by the
schema even when enabled: false.
Verified live: kubectl apply succeeded and the Redis resource was created
before committing this.
Two related fixes, found while working through why authentik/n8n/gitlab
ArgoCD Applications showed Unknown/stuck health despite pods being fine:
1. ArgoCD has no built-in health check for CNPG's Cluster CRD, so it
always reported "Unknown" - and since app-level health rolls up to
the worst resource status, every app with a CNPG Cluster showed
Unknown/Progressing regardless of actual state. Added a Lua health
check (resource.customizations.health.postgresql.cnpg.io_Cluster)
reading .status.conditions[Ready] / .status.phase.
2. Once that started reporting real status instead of masking it,
pg-authentik showed a genuine problem: CNPG's operator couldn't
reach its Postgres instances' status API (port 8000) - "Cannot
extract Pod status ... context deadline exceeded" - because:
a) authentik-ingress's CiliumNetworkPolicy never allowed cnpg-system
as a source (crowdsec-ingress already had this exception,
authentik-ingress was just missing it - inconsistency, not
deliberate).
b) Even after fixing (a), still blocked - pg-authentik's pods are
ambient-mesh-enrolled, so the connection actually goes through
ztunnel's HBONE tunnel (port 15008) first, same underlying issue
as the argocd-redis fix from earlier today. Rather than keep
finding and patching this per-namespace as it recurs, added a
cluster-wide CiliumClusterwideNetworkPolicy allowing HBONE
broadly - ztunnel's own mTLS/SPIFFE identity verification is the
real security boundary for mesh traffic; Cilium blocking the
tunnel port itself was only breaking legitimate traffic, not
adding meaningful protection on top.
Verified live: CNPG operator immediately went from failing status
extraction to successfully reconciling (recreating a pod to reattach
its PVC) once both fixes were in place.
apps/nextcloud/values.yaml had oidc_login_client_id/secret as literal
empty strings, causing "client_id is missing or invalid" on login.
The matching Authentik blueprint (infrastructure/authentik/nextcloud-
blueprint.yaml) already defines fixed credentials for this OAuth2
provider - just never got copied over. Filled in to match.
Also found and left alone: an orphaned nextcloud-oidc-config ConfigMap
(from January) that already had the correct values but isn't actually
mounted by the current Deployment (which uses nextcloud-config instead)
- harmless dead resource, not the active config path.
Both pre-existing (not from today's work), found while cleaning up a
recurring stale-pod pattern (a fresh vault-init Job failing on every
sync of the vault Application):
1. Init-detection was always broken: `vault status -format=json`
pretty-prints with a space after colons ("initialized": true), but
the parser (grep -o '"initialized":[a-z]*' | cut -d: -f2) required
no space - it always matched empty, so `initialized`/`sealed` were
always blank regardless of real state. Every run concluded "not
initialized" and tried to re-init, which correctly fails once Vault
already has been ("Vault is already initialized"). Switched to jq.
Separately, the old `|| echo '{}'` fallback also discarded valid
JSON on vault status's normal non-zero exit codes (2 = sealed) -
fixed too, though the parsing bug was the actual blocker.
2. `vault_exec` wraps `kubectl exec` without `-i`, so the heredoc piped
into `vault policy write NAME -` never reached the remote command -
vault saw an empty policy body. Never previously reached in
practice since bug #1 always failed the script earlier. Added -i.
Verified end-to-end with a one-off Job run: script now correctly
detects the already-initialized state and completes successfully,
including the policy/role writes that were previously unreachable.
- ZONE_NAME was "kube.huskypup.net" in both the home-ip-ddns CronJob and
the VPS failover watcher - that's a record, not a Cloudflare zone (the
actual zone is the parent "huskypup.net"). Caused home-ip-ddns to fail
every run (zone lookup returned zero results, curl -f exit 22) -
confirmed live and fixed.
- Added seccompProfile/non-root/dropped-capabilities securityContext to
the three CronJobs added this session that were missing it (flagged by
the cluster's "restricted" PodSecurity admission). Repointed the raft
snapshot job's mc binary install from /usr/local/bin to /tmp so it
still works running as non-root.
Root cause (confirmed via ztunnel logs): the argo-cd chart's default
NetworkPolicies only allow each component's application port (e.g. 6379
for redis), but Istio ambient mode routes ALL pod-to-pod traffic through
ztunnel's HBONE tunnel on port 15008 first - so the tunnel itself was
being blocked even though the "real" port was allowed. Every inter-pod
connection in the argocd namespace hung for exactly 10s then reset;
ztunnel's own log named it directly ("connection timed out, maybe a
NetworkPolicy is blocking HBONE port 15008"). This broke argocd-server's
Redis-backed session/cluster-info caching cluster-wide and was silently
preventing the root Application from picking up new child Applications.
Fix: additive NetworkPolicy allowing ingress on 15008 for all argocd
pods (NetworkPolicies union across multiple policies selecting a pod,
so this doesn't touch/replace the chart's rendered ones - safe across
Helm upgrades).
Also: dropped argocd from the namespace-enrollment job's waypoint list.
argocd's only AuthorizationPolicy (allow-argocd-access) is a plain
source-namespace/IP match with no L7 rules - its own status shows
"attached to ztunnel", not waypoint - so forcing L7 waypoint processing
onto the namespace was unnecessary overhead, not a security requirement.
ztunnel's mTLS still fully covers it. (This was a red herring for the
HBONE bug itself, not the fix, but a valid simplification found along
the way.)
Foundation for a DR/backup path using an always-on VPS as a second
ArgoCD-managed cluster, plus DB/backup standardization work that fell
out of it:
- vps-standby ArgoCD cluster destination + AppProject, MinIO backup
receiver, VPS bootstrap script (k3s, Netbird, cert-manager)
- Dual-site DNS failover watcher + home-IP DDNS CronJob, Cloudflare
token moved out of git into Vault+ExternalSecret
- Nextcloud migrated from ad-hoc MariaDB to CNPG + redis-operator
(matches n8n/Authentik/GitLab's backup-native pattern)
- Authentik's CNPG manifests moved into the actual ArgoCD-synced
manifests/ path (were present but never wired into the sync path)
- Vault raft-snapshot CronJob, CNPG barmanObjectStore backups
(Authentik/n8n/Nextcloud), Nextcloud file-PVC restic sync - all
targeting the new VPS MinIO receiver
See VPS Warm-Standby plan doc for full design rationale.
Gitaly CPU limit was 250m causing throttling on git operations (485ms
p99 to webservice). Increased to 1 core with 100m request. Also added
workerProcesses: 3 to webservice for better request concurrency
(was default 2).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RollingUpdate with a ReadWriteOnce PVC causes a Multi-Attach deadlock
where the new pod can't start because the old pod still holds the
volume. Recreate strategy ensures the old pod is terminated before
the new one is created.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
NetBird router deployments require NET_ADMIN capability, which is
blocked by the default baseline PodSecurity standard. Add
managedNamespaceMetadata with pod-security.kubernetes.io/enforce:
privileged to argocd, frigate, gitlab, grafana, guacamole,
home-assistant, teslamate, and vault ArgoCD Applications.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The operator watches for netbird.io/expose (presence-based), not
netbird.io/resource which was from the old Helmfile bootstrap script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The operator was missing managementURL, causing it to default to the
NetBird SaaS cloud (api.netbird.io) instead of the self-hosted instance.
This resulted in an infinite reconcile loop and no networks/resources
being created.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Traffic goes through Istio (not Envoy Gateway), so cookie-based
consistent hashing must be on the Istio DestinationRule. Removes
the unused Envoy Gateway BackendTrafficPolicy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fixes OIDC nonce validation failures with 2 replicas by using a
BackendTrafficPolicy with cookie-based consistent hashing instead of
Service-level ClientIP affinity (which doesn't work behind Envoy proxy).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
With 2 replicas, OIDC callbacks can hit a different pod than the
one that generated the state/nonce, causing auth failures. Session
affinity ensures the same client always reaches the same pod.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change EXTENSION_PRIORITY from "*,openid" to "openid,*" so
unauthenticated users are redirected to Authentik OIDC instead
of seeing the database login form.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fixes NameError: uninitialized constant Gitlab::Redis::ALL_CLASSES
caused by 7_prometheus_metrics.rb loading before 7_redis.rb
(alphabetically 'p' < 'r'). The ConfigMap provides a 0_redis_early.rb
initializer that requires gitlab/redis before other initializers run.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Includes security fixes (CVE-2023-43040) and RBD/RADOS improvements.
Rook operator upgraded to v1.19.2 via Helm.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
LVM filter now applied via Talos machine config on all nodes to prevent
lvs hang on BIOS partitions. talos-cp-02 can safely run OSD prepare jobs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Scale CNPG pg-teslamate to 1 instance: on-disk storage corruption
(corrupted sequences, system catalog indexes) means replicas cloned
via pg_basebackup inherit corruption and can't open any database.
Primary works from cached catalogs. Sequences already rebuilt live.
- Remove talos-cp-02 from Rook-Ceph nodes list: ceph-volume inventory
hangs on BIOS partition via nsenter lvs in host namespace. Talos
read-only /etc prevents LVM filter fix. OSD 1 keeps running from
its existing deployment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
X-Forwarded-Proto approach didn't work - Ceph dashboard SAML doesn't
read it. Re-enabling native SSL so the dashboard sees https:// URLs
directly. Updated both Istio and Envoy Gateway backends for TLS with
insecureSkipVerify for the self-signed cert.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dashboard runs with ssl: false behind TLS-terminating proxy, so SAML
library sees http:// URLs and rejects responses expecting https://.
Explicitly set X-Forwarded-Proto: https on both Istio VirtualService
and Envoy Gateway HTTPRoute so the SAML ACS URL matches.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dashboard has ssl: false (plain HTTP on port 7000), but the DestinationRule
had tls.mode: SIMPLE which caused Istio to attempt TLS to the backend,
resulting in WRONG_VERSION_NUMBER errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 4 OSDs successfully recreated in clean raw mode (NVMe only, no DB devices).
Fixed talos-cp-02 NVMe device path from nvme0n1 to nvme1n1 to match actual hardware.
Removed temporary fix-rook-osd-metadata-device Kyverno policy as it's no longer needed.
Operator runs normally without manual patches.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The dashboard has SSL disabled and listens on port 7000. Restoring the
VirtualService with the correct port for the Istio gateway path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Services behind the Envoy gateway cannot reach Istio ambient mesh
backends due to STRICT mTLS. Reverting DNS to point back to the Istio
gateway until the migration path is resolved.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All namespaces with CiliumNetworkPolicies need envoy-gateway-system
access now that services are migrating from Istio to Envoy gateway.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allows external-dns to create DNS records from Gateway API HTTPRoute
resources, needed for services using the Envoy edge gateway.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The dashboard now uses the Envoy edge gateway via HTTPRoute. The old
Istio VirtualService was causing DNS to point at the wrong gateway IP.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Network policies in the network-policies/ subdirectory were not being
synced because ArgoCD was not recursing into subdirectories.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Ceph dashboard moved from Istio to the Envoy edge gateway but the
CiliumNetworkPolicy was still only allowing istio-system, causing 503s.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The dashboard has SSL disabled and listens on port 7000, but the
Backend was configured for TLS on port 8443.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ArgoCD, NetBird operator, and Scylla Manager are upstream Helm charts
that don't set resource limits, probes, drop-all capabilities, or
disable SA token automount. These audit-mode policy violations inflated
the cumulative Prometheus failure counter to ~3800.
- argocd: exempt from require-resource-limits, require-probes
- netbird-operator: exempt from require-resource-limits,
disallow-automount-sa-token, require-probes
- scylla-manager: exempt from require-drop-all-capabilities,
disallow-automount-sa-token
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The global-authentik-authz AuthorizationPolicy uses *.kube.huskypup.net
as a host match, but Kiali flagged KIA0104 because the wildcard had no
matching entry in Istio's service registry. Adding a MESH_EXTERNAL
ServiceEntry with resolution NONE registers the hostname without
affecting routing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The scylla-manager Application was missing the third source for raw
manifests, so the VirtualService in infrastructure/scylla/manifests/
was never deployed. This caused Kiali to flag the ext-authz policy
host as having no matching service entry.
Also removed stale PERMISSIVE PeerAuthentication from authentik
namespace (manually applied, not in git, unnecessary with mesh-wide
STRICT mTLS).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use the OSD-specific activate-osd volume mount instead of rook-data
parent directory. Multiple old OSD dirs exist on each node and the
script was creating the symlink in the wrong (stale LVM) directory.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use patchesJson6902 with append (/-) for create-block-db-symlink to
ensure it runs after activate, not before. The patchStrategicMerge was
inserting it before activate, causing prime-osd-dir to clear the symlink.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The netbird-operator webhook service runs in the netbird namespace and
receives admission webhook calls from kube-apiserver using node IPs
(172.28.101.0/24). Without this, ztunnel rejects the connections with
"allow policies exist, but none allowed".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove metadataDevice (sdb) from all 4 nodes in cluster-values.yaml
since the SATA SSDs are slower than the NVMe data drives. Add temporary
Kyverno mutate policy to protect existing OSD deployments when the
operator is re-enabled - replaces expand-bluefs with a no-op and adds
conditional block.db symlink creation based on BlueStore label detection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Home Assistant was returning 400 errors because it received
X-Forwarded-For headers from the Istio ingress gateway but wasn't
configured to trust reverse proxies. Add use_x_forwarded_for and
trusted_proxies to the init container's configuration.yaml template.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ED25519 PRIVATE KEY header is not recognized by nginx's OpenSSL.
Convert to standard PKCS#8 PRIVATE KEY header (same DER content).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kiali was reporting "Gateway API Class not found" errors for the
envoy-gateway Gateways (gateway/edge and guacamole/guacamole).
Explicitly register istio, istio-waypoint, and envoy-gateway classes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The ztunnel container exposes metrics on port named 'ztunnel-stats' (15020),
not 'http-monitoring'. This was preventing Prometheus from scraping ztunnel
L4 metrics, causing missing traffic in Kiali.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Secret was in prometheus namespace but the talos-metrics-proxy
deployment referencing it runs in kube-system namespace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- n8n: Set replicaCount=1 (RWO PVC incompatible with multiple replicas),
add resource limits to satisfy Kyverno policy
- gitlab: Add ignoreDifferences for redis-gitlab-additional service
(port names and selectors managed by Redis operator)
- netbird-operator: Add ExternalSecret for netbird-mgmt-api-key in
netbird namespace and add manifests source to Application
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Gateway API CRDs are already managed by istio; skip them
from the vendored envoy-gateway chart to prevent OutOfSync.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Webservice was at 94% memory (1875Mi/2Gi) with only 50m CPU request,
causing CPU throttling and slow first requests after idle periods.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The checksum/lapi-secret annotation on the LAPI Deployment changes
when CrowdSec auto-generates credentials, causing permanent
OutOfSync with ArgoCD ServerSideDiff.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ArgoCD v3.3.3 has a bug where OCI chart revision resolution
doesn't include the chart name in the Docker Hub API path,
causing 401 errors. The chart is already deployed at v1.6.3.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ArgoCD requires the chart field for OCI sources. The repoURL
contains the registry/org path, and chart specifies the chart name.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move chart name into repoURL for OCI registry compatibility.
ArgoCD resolves OCI chart references differently from HTTP repos.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bitnami/kubectl has removed all version tags from Docker Hub,
leaving only :latest which is blocked by Kyverno disallow-latest-tag
policy. Switch to alpine/k8s:1.32.13 which has kubectl, bash, jq,
and uses proper version tags.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The :1.31 tag is gone from Docker Hub and :latest is blocked by Kyverno
disallow-latest-tag policy. The 1.29 image is cached on cluster nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The bitnami/kubectl:1.31 tag doesn't exist on Docker Hub. Updated all
hook jobs to use bitnami/kubectl:latest.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PreSync/PostSync hook Jobs reference argocd-hook-sa but it only existed
in the argocd namespace. Jobs running in app namespaces (crowdsec,
authentik, netbird, vault, istio-system) need the SA in their own
namespace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Tetragon CRD defaults followChildren to false in matchBinaries,
causing ArgoCD ServerSideDiff to detect a permanent diff.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Works with the ignoreDifferences config in argocd-cm to prevent
auto-sync from triggering on controller-managed field changes
(ESO finalizers, HTTPRoute annotations, etc.)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Apps using ServerSideApply without ServerSideDiff causes ArgoCD to
compute diffs client-side, which detects controller-added fields
(finalizers, defaults, annotations) as changes and keeps apps
permanently OutOfSync. ServerSideDiff uses dry-run SSA for accurate
diff computation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Rook operator reconciles OSD deployments and removes the
expand-bluefs=/bin/true workaround patches needed for BlueStore UUID
mismatch. Keep operator scaled to 0 with manual sync only until the
underlying issue is resolved.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
s6-overlay's log-prepare needs chmod on /dev/shm/logs/* which
requires CAP_FOWNER when all capabilities are dropped.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
BlueStore has UUID inconsistency at block position 0x0 that causes
expand_devices assertion in both v19.2.0 and v19.2.3. The OSD
daemon itself runs fine; only expand-bluefs init container crashes.
Will use deployment patches with do-not-reconcile annotation instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Revert talos-client-cert secret to prometheus namespace (prometheus
pods mount this secret, not kube-system pods)
- Add CHOWN/DAC_OVERRIDE/SETUID/SETGID capabilities to frigate
(s6-overlay needs chown for log directories)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Ceph v19.2.0 has BlueStore::expand_devices assertion bug that
crashes OSD expand-bluefs init containers. v19.2.3 includes fixes.
- NetBird operator webhook failurePolicy: Fail blocks all pod
creation cluster-wide when webhook service is unreachable.
Changed to Ignore.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Envoy Gateway moved to OCI registry (docker.io/envoyproxy).
NetBird operator charts moved to https://netbirdio.github.io/helms
with chart name kubernetes-operator.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
