mirror of
https://gitlab.kube.huskypup.net/Scooby/Homelabv4.git
synced 2026-08-20 23:16:49 +00:00
Fix CNPG operator -> instance status communication + add CNPG health check
Two related fixes, found while working through why authentik/n8n/gitlab
ArgoCD Applications showed Unknown/stuck health despite pods being fine:
1. ArgoCD has no built-in health check for CNPG's Cluster CRD, so it
always reported "Unknown" - and since app-level health rolls up to
the worst resource status, every app with a CNPG Cluster showed
Unknown/Progressing regardless of actual state. Added a Lua health
check (resource.customizations.health.postgresql.cnpg.io_Cluster)
reading .status.conditions[Ready] / .status.phase.
2. Once that started reporting real status instead of masking it,
pg-authentik showed a genuine problem: CNPG's operator couldn't
reach its Postgres instances' status API (port 8000) - "Cannot
extract Pod status ... context deadline exceeded" - because:
a) authentik-ingress's CiliumNetworkPolicy never allowed cnpg-system
as a source (crowdsec-ingress already had this exception,
authentik-ingress was just missing it - inconsistency, not
deliberate).
b) Even after fixing (a), still blocked - pg-authentik's pods are
ambient-mesh-enrolled, so the connection actually goes through
ztunnel's HBONE tunnel (port 15008) first, same underlying issue
as the argocd-redis fix from earlier today. Rather than keep
finding and patching this per-namespace as it recurs, added a
cluster-wide CiliumClusterwideNetworkPolicy allowing HBONE
broadly - ztunnel's own mTLS/SPIFFE identity verification is the
real security boundary for mesh traffic; Cilium blocking the
tunnel port itself was only breaking legitimate traffic, not
adding meaningful protection on top.
Verified live: CNPG operator immediately went from failing status
extraction to successfully reconciling (recreating a pod to reattach
its PVC) once both fixes were in place.
This commit is contained in:
@@ -36,6 +36,44 @@ configs:
|
|||||||
jqPathExpressions:
|
jqPathExpressions:
|
||||||
- .spec.volumeClaimTemplates[].apiVersion
|
- .spec.volumeClaimTemplates[].apiVersion
|
||||||
- .spec.volumeClaimTemplates[].kind
|
- .spec.volumeClaimTemplates[].kind
|
||||||
|
# ArgoCD has no built-in health check for CNPG's Cluster CRD, so it
|
||||||
|
# defaults to "Unknown" - and since app-level health rolls up to the
|
||||||
|
# worst status of any resource, every app with a CNPG Cluster (authentik,
|
||||||
|
# n8n, gitlab, nextcloud, crowdsec) shows Unknown/Progressing overall even
|
||||||
|
# when everything is actually fine. Confirmed 2026-08-17: no amount of
|
||||||
|
# refreshing/resyncing individual apps fixed it, because it isn't a cache
|
||||||
|
# issue - there was just nothing telling ArgoCD how to read CNPG's status.
|
||||||
|
resource.customizations.health.postgresql.cnpg.io_Cluster: |
|
||||||
|
health_status = {}
|
||||||
|
if obj.status ~= nil and obj.status.conditions ~= nil then
|
||||||
|
for i, condition in ipairs(obj.status.conditions) do
|
||||||
|
if condition.type == "Ready" then
|
||||||
|
if condition.status == "True" then
|
||||||
|
health_status.status = "Healthy"
|
||||||
|
health_status.message = "Cluster is Ready"
|
||||||
|
return health_status
|
||||||
|
else
|
||||||
|
health_status.status = "Progressing"
|
||||||
|
health_status.message = condition.message or "Cluster is not Ready"
|
||||||
|
return health_status
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
if obj.status ~= nil and obj.status.phase ~= nil then
|
||||||
|
if obj.status.phase == "Cluster in healthy state" then
|
||||||
|
health_status.status = "Healthy"
|
||||||
|
elseif string.find(obj.status.phase, "Failed") then
|
||||||
|
health_status.status = "Degraded"
|
||||||
|
else
|
||||||
|
health_status.status = "Progressing"
|
||||||
|
end
|
||||||
|
health_status.message = obj.status.phase
|
||||||
|
return health_status
|
||||||
|
end
|
||||||
|
health_status.status = "Unknown"
|
||||||
|
health_status.message = "No status information available yet"
|
||||||
|
return health_status
|
||||||
oidc.config: |
|
oidc.config: |
|
||||||
name: Authentik
|
name: Authentik
|
||||||
issuer: https://auth.kube.huskypup.net/application/o/argocd/
|
issuer: https://auth.kube.huskypup.net/application/o/argocd/
|
||||||
|
|||||||
@@ -69,6 +69,14 @@ spec:
|
|||||||
- fromEndpoints:
|
- fromEndpoints:
|
||||||
- matchLabels:
|
- matchLabels:
|
||||||
k8s:io.kubernetes.pod.namespace: authentik
|
k8s:io.kubernetes.pod.namespace: authentik
|
||||||
|
# CNPG operator - status port (8000) queries on pg-authentik instances.
|
||||||
|
# crowdsec-ingress below already has this; was missing here, which broke
|
||||||
|
# CNPG's own health/status reporting for pg-authentik ("Cannot extract Pod
|
||||||
|
# status ... context deadline exceeded" in the operator logs) even though
|
||||||
|
# the database itself was working fine. Confirmed 2026-08-17.
|
||||||
|
- fromEndpoints:
|
||||||
|
- matchLabels:
|
||||||
|
k8s:io.kubernetes.pod.namespace: cnpg-system
|
||||||
# Apps doing OIDC token exchange
|
# Apps doing OIDC token exchange
|
||||||
- fromEndpoints:
|
- fromEndpoints:
|
||||||
- matchLabels:
|
- matchLabels:
|
||||||
|
|||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# Allow ztunnel's HBONE tunnel port (15008) cluster-wide.
|
||||||
|
#
|
||||||
|
# Istio ambient mode routes ALL pod-to-pod traffic through ztunnel's mTLS
|
||||||
|
# HBONE tunnel on port 15008 first, regardless of which "real" port the
|
||||||
|
# traffic is ultimately for. Per-namespace CiliumNetworkPolicies that only
|
||||||
|
# allow the application port (e.g. 6379 for Redis, 8000 for CNPG's instance
|
||||||
|
# status API) block the HBONE tunnel itself even when the intended port is
|
||||||
|
# allowed - ztunnel's own encrypted connection never gets through.
|
||||||
|
#
|
||||||
|
# Hit this same bug twice in one session (2026-08-17): argocd-redis (fixed
|
||||||
|
# with a namespace-scoped NetworkPolicy) and cnpg-system -> pg-authentik's
|
||||||
|
# status port (fixed the same way first, still broken - needed this too).
|
||||||
|
# Rather than keep discovering and patching this per-namespace, allow HBONE
|
||||||
|
# broadly: ztunnel already enforces real security here via mTLS + SPIFFE
|
||||||
|
# identity verification (and any CiliumNetworkPolicy/AuthorizationPolicy
|
||||||
|
# ingress rules on the target still apply to the traffic's real intent) -
|
||||||
|
# Cilium blocking the tunnel port itself only breaks legitimate mesh traffic,
|
||||||
|
# it doesn't add a meaningful security boundary on top of what ztunnel
|
||||||
|
# already does.
|
||||||
|
apiVersion: cilium.io/v2
|
||||||
|
kind: CiliumClusterwideNetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: allow-hbone
|
||||||
|
spec:
|
||||||
|
description: "Allows Istio ambient's ztunnel HBONE tunnel port cluster-wide - see comments"
|
||||||
|
enableDefaultDeny:
|
||||||
|
egress: false
|
||||||
|
ingress: false
|
||||||
|
endpointSelector: {}
|
||||||
|
ingress:
|
||||||
|
- toPorts:
|
||||||
|
- ports:
|
||||||
|
- port: "15008"
|
||||||
|
protocol: TCP
|
||||||
Reference in New Issue
Block a user