Per-pod NodePort Services for the 3 existing Gitaly nodes
(ha-gitaly-nodeport.yaml) plus home's Rails internal API
(ha-gitlab-internal-api-nodeport.yaml, needed for gitlab-shell hooks
whenever the VPS node is primary for a repo). PeerAuthentication
PERMISSIVE + AuthorizationPolicy port rules for both, matching the
proven Postgres pattern. CoreDNS rewrites so home's own Praefect
resolves the floating hostnames locally instead of round-tripping.
Reached over the Netbird mesh, not the UniFi WAN forward used for
Postgres - Gitaly gRPC here is plaintext (carries the gitaly_token,
gitlab_shell_secret, and raw repo contents), unlike CNPG's
TLS-negotiated stream.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per the approved plan (jiggly-snacking-iverson.md, 'GitLab True
Cross-Site Replication' section) - replacing the git-mirror CronJob
with real CNPG streaming replication for pg-gitlab and pg-praefect,
reusing the exact pattern already proven for pg-authentik.
This commit is network plumbing only, no DB replication yet:
- Repoint both CNPG clusters' barmanObjectStore backup target from
in-cluster gitlab-minio-svc to VPS MinIO (also becomes the initial
seed source for the VPS's replicas in Phase 1c).
- New ha-gitlab-postgres/ha-praefect-postgres NodePort Services
(bypass CNPG's own -rw Services) on both home and VPS.
- New PeerAuthentication with portLevelMtls PERMISSIVE on the replication
ports - ambient mesh STRICT default would otherwise reset the
VPS/witness's connection before the Postgres TLS handshake starts.
- Extended allow-gitlab-access AuthorizationPolicy with a port-scoped
rule for the same traffic (no source.namespaces rule can match
external, non-mesh peers).
- CoreDNS rewrites for pg-gitlab.ha.huskypup.net / pg-praefect.ha.huskypup.net
so home's own pods reach the floating hostname locally instead of
round-tripping through the WAN forward (no NAT hairpin support).
Still needed before Phase 1b/1c: UniFi WAN port-forwards for external
ports 61442/61443 -> talos-cp-01:32442/32443, source-restricted to the
VPS + witness public IPs (manual, same as the existing 61432 rule).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
n8n's allow-n8n-access AuthorizationPolicy only allowed traffic from the
n8n/istio-system/prometheus namespaces - unlike nextcloud/authentik's
policies, it was missing the broader ipBlocks: 10.244.0.0/16 fallback
rule. The CNPG operator (cnpg-system namespace) polling pg-n8n instances'
status API (port 8000) wasn't covered by any rule, so Istio's L4
authorization enforcement (via ztunnel - this traffic is direct pod-IP,
not routed through a Service, so waypoint/L7 never sees it) rejected the
connection outright, surfacing as a TCP reset to the operator.
This was the actual cause of pg-n8n's long-recurring 'Instance Status
Extraction Error' - confirmed 100% reproducible tonight (every single
operator poll attempt failed identically), not the intermittent
HBONE-tunnel issue it resembled from the outside. Also labeled the n8n
namespace istio.io/use-waypoint=waypoint (was in the enrollment script's
namespace list but the label was missing live - namespace parity fix,
though not itself the fix for this specific issue since this traffic
bypasses the waypoint).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause (confirmed via ztunnel logs): the argo-cd chart's default
NetworkPolicies only allow each component's application port (e.g. 6379
for redis), but Istio ambient mode routes ALL pod-to-pod traffic through
ztunnel's HBONE tunnel on port 15008 first - so the tunnel itself was
being blocked even though the "real" port was allowed. Every inter-pod
connection in the argocd namespace hung for exactly 10s then reset;
ztunnel's own log named it directly ("connection timed out, maybe a
NetworkPolicy is blocking HBONE port 15008"). This broke argocd-server's
Redis-backed session/cluster-info caching cluster-wide and was silently
preventing the root Application from picking up new child Applications.
Fix: additive NetworkPolicy allowing ingress on 15008 for all argocd
pods (NetworkPolicies union across multiple policies selecting a pod,
so this doesn't touch/replace the chart's rendered ones - safe across
Helm upgrades).
Also: dropped argocd from the namespace-enrollment job's waypoint list.
argocd's only AuthorizationPolicy (allow-argocd-access) is a plain
source-namespace/IP match with no L7 rules - its own status shows
"attached to ztunnel", not waypoint - so forcing L7 waypoint processing
onto the namespace was unnecessary overhead, not a security requirement.
ztunnel's mTLS still fully covers it. (This was a red herring for the
HBONE bug itself, not the fix, but a valid simplification found along
the way.)
The operator watches for netbird.io/expose (presence-based), not
netbird.io/resource which was from the old Helmfile bootstrap script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bitnami/kubectl has removed all version tags from Docker Hub,
leaving only :latest which is blocked by Kyverno disallow-latest-tag
policy. Switch to alpine/k8s:1.32.13 which has kubectl, bash, jq,
and uses proper version tags.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The :1.31 tag is gone from Docker Hub and :latest is blocked by Kyverno
disallow-latest-tag policy. The 1.29 image is cached on cluster nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The bitnami/kubectl:1.31 tag doesn't exist on Docker Hub. Updated all
hook jobs to use bitnami/kubectl:latest.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
