Two pre-existing gaps (not caused by this session's earlier work),
surfaced when the user started testing the VPS's standby Vault
(infrastructure/vps-standby/vault/) login via Authentik SSO:
1. VPS Vault's own auth/oidc/role/default allowed_redirect_uris only had
vault.kube.huskypup.net (home) and localhost:8250 - never had
vault.vps.huskypup.net registered, even though VPS's Vault shares the
same Authentik OAuth2 provider/client_id as home's. Fixed on both
sides: the shared blueprint (redirect_uris list) and VPS Vault's own
role config directly (separate Vault instance/backend, needed its own
write).
2. auth.kube.huskypup.net (Authentik itself) had NO public DNS record at
all - it only ever resolved via UniFi's local DNS for home-LAN
clients. Vault's OIDC backend runs server-side on the VPS and needs to
fetch Authentik's discovery document directly - it silently returned
an empty auth_url because it couldn't resolve the hostname at all
('missing auth_url' in Vault's UI, root cause only found by checking
DNS resolution from inside the VPS's own vault pod). Added a public
Cloudflare A record, same pattern as vault.kube.huskypup.net and
pg-authentik.ha.huskypup.net earlier - user confirmed before adding,
given it's Authentik itself. This likely also fixes OIDC for any other
VPS-hosted app doing SSO against home's Authentik, not just Vault.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Completes the 'make the repo proper' cleanup from earlier this session -
these were flagged but deliberately not touched in 185e9c2 given the
blast radius (live SSO for 8 apps). User confirmed: fix all 8 now.
infrastructure/authentik/{argocd,gitlab,nextcloud,grafana,n8n,guacamole,
rancher,vault}-blueprint.yaml: converted from plain ConfigMap (client_secret
hardcoded) to ExternalSecret with a templated blueprint body
(client_secret: "{{ .clientSecret }}") pulling from Vault. Chart already
supports mounting blueprints from Secrets (blueprints.secrets, alongside
blueprints.configMaps) - infrastructure/authentik/values.yaml updated to
route these 8 there instead.
For argocd/nextcloud/n8n/guacamole/rancher: Vault already had the matching
value at secret/<app>-oauth (the APP side was already Vault-backed via its
own ExternalSecret) - the blueprint was the only remaining plaintext copy.
For gitlab/grafana/vault: Vault had no copy at all yet - created
secret/{gitlab,grafana,vault}-oauth with the EXISTING live values (not
rotated - these are the actual working credentials right now, rotating
would break login until every consumer is updated in lockstep, which is
out of scope for a cleanup pass). Also fixed the OTHER plaintext copies
that existed for these three specifically:
- apps/gitlab/manifests/external-secret-oidc.yaml (new): replaces a
manually kubectl-created, never-git-tracked gitlab-oidc-secret.
- infrastructure/grafana/manifests/grafana-oauth-secret.yaml: was a
plain Secret whose own comment said 'hardcoded from blueprint'.
- infrastructure/vault/manifests/vault-init-{configmap,job}.yaml: this
one COULDN'T be converted to the same ExternalSecret-from-Vault
pattern - it's the PostSync Job that grants ESO's own Kubernetes-auth
role in Vault, so ESO can't yet authenticate to pull anything from
Vault at the point this script runs (genuinely circular). Sourced
from a new vault-oidc-bootstrap Secret instead - created once
manually (kubectl, not git, matching how Vault's own root/unseal
material is already handled), independent of the ESO pipeline.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
home's own authentik pods reach ha-authentik-postgres via the CoreDNS
rewrite on port 61432 (not just external traffic via NodePort on 5432/
61432) - discovered live that Istio ambient's port-level mTLS/L4
authorization enforcement is keyed on the port actually dialed (61432
here), not just the pod's real containerPort (5432) traffic eventually
reaches after Service translation. The existing port-5432-only rules
(from the streaming-replication fix) didn't cover this in-cluster path,
surfacing as 'server closed the connection unexpectedly' from home's own
authentik-worker pod.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause of the 'PostgreSQL connection failed, retrying... (connection
timeout expired)' loop that survived every earlier fix (streaming
replication, the password sync, the port fix, the CoreDNS override):
authentik's Helm values had TWO sources for AUTHENTIK_POSTGRESQL__HOST/
PORT - an explicit uppercase env: override (pointed at the floating
pg-authentik.ha.huskypup.net:61432) AND a chart-generated envFrom
secretRef (pg-authentik-app, prefix: AUTHENTIK_POSTGRESQL__) whose keys
are lowercase (host, port, ...), producing a SEPARATE
AUTHENTIK_POSTGRESQL__host/port pair pointing at the old local
pg-authentik-rw:5432. Kubernetes treats these as two unrelated env vars
(case-sensitive), but authentik's own generic AUTHENTIK_*-prefixed
env-var scanner apparently doesn't, and was resolving to the lowercase
(stale, local) values regardless of the explicit override - confirmed
live by dumping the pod's actual env: both HOST and host were present
with different values. Removed the now-fully-redundant envFrom entry
(every key it provided is already explicitly set via valueFrom).
Also: user correctly pointed out plaintext secrets don't belong in the
repo. Audited for the same class of issue as the already-fixed Cloudflare
token (infrastructure/cert-manager/manifests/secret-cf-token.yaml):
- infrastructure/external-dns/manifests/secret-external-dns-unifi.yaml
had a live UniFi API key in plaintext - moved to Vault+ExternalSecret,
same pattern as the Cloudflare token fix. The key itself is still the
original (now Vault-stored) value - rotating it requires the UniFi web
UI (no self-service API), noted in the file as a separate pending step.
- infrastructure/vps-eso/manifests/clustersecretstore.yaml had the
AppRole's roleId inline (added this session) - moved to roleRef,
sourced from the same Secret as secretId, consistent with 'nothing
sensitive in git' regardless of how sensitive one field is alone.
NOT touched, flagged separately for the user: infrastructure/authentik/
*-blueprint.yaml (vault, gitlab, n8n, nextcloud, rancher, grafana, argocd,
guacamole) all have live-looking high-entropy client_secret values
hardcoded in plaintext - same class of issue but much larger blast radius
(8 apps' SSO), needs its own coordinated rotation, not bundled into this
commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found via live test: the CoreDNS rewrite for pg-authentik.ha.huskypup.net
resolves the NAME correctly to a local service, but DNS rewriting can't
change the PORT a client then connects to - the app tier is configured
with AUTHENTIK_POSTGRESQL__PORT=61432 (matching the external floating
hostname), so the local target needs to actually listen on 61432 too, or
the connection times out even though DNS resolves fine. Also repoints the
rewrite target from CNPG's own pg-authentik-rw to this Service, so it
resolves to a port that actually exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
User correctly flagged this: manually copying secrets between home and
the VPS defeats the entire point of having Vault. Every VPS secret this
session (MinIO creds, Cloudflare token, CNPG certs, the pg-authentik-app
password) was a one-off kubectl create secret copy-paste, because the
VPS's k3s cluster had no Vault/ESO pipeline at all - this builds one.
- infrastructure/authentik/manifests/ha-postgres-app-pushsecret.yaml:
home pushes pg-authentik-app's password into Vault at
secret/vps/pg-authentik-app (PushSecret, not just Get - this password is
CNPG-generated, not Vault-native, so it has to originate from a push).
- infrastructure/vps-eso/manifests/clustersecretstore.yaml: ESO on the
VPS (installed via helm, out-of-band like k3s/cert-manager - see the
file's own header) authenticates to home's Vault via AppRole (not
Kubernetes auth - the VPS is a separate cluster with no federation to
home's API server). Reachable via a new public
https://vault.kube.huskypup.net record - Vault was deliberately kept
off the public internet before this, explicitly confirmed with the user
before opening it. Traffic goes through the same Istio ingress gateway
already serving other public hosts, so unlike the ha-authentik-postgres
NodePort case, no PeerAuthentication/AuthorizationPolicy change was
needed - it arrives as a normal in-mesh call from the gateway's own
identity, not raw external TCP to a pod.
Found and fixed a real Vault gotcha while wiring this up: tried to
remove the AppRole's token_bound_cidrs restriction (added first, before
discovering Vault can't see the VPS's real source IP through the
gateway - it only ever sees the gateway's own pod IP) by omitting the
field from a follow-up - that does NOT clear it, the
AppRole role endpoint preserves omitted fields rather than resetting
them to default. Had to explicitly write token_bound_cidrs=. Spent a
while chasing a misleading 403 permission denied on auth/token/
lookup-self before finding this - vault token capabilities said read
was allowed (policy was fine), the actual rejection was IP-bound token
use from an unbound context.
- infrastructure/vps-standby/authentik/manifests/
pg-authentik-app-externalsecret.yaml: pulls it back down, Merge policy
(only overwrites the password key - host/dbname/username stay local,
CNPG still needs its own local -rw hostname for internal use).
- argocd-apps/vps-standby/vps-eso.yaml: new Application for the
ClusterSecretStore.
Verified end-to-end: password now matches between home and VPS's
pg-authentik-app secrets via this pipeline (not the earlier manual patch),
confirmed by comparing both live secret values after ESO's sync.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Builds the one genuinely new component from the plan - everything else
reuses CNPG/etcd as-is.
- infrastructure/ha-failover/manifests/heartbeat-writer.yaml: home writes
a fresh timestamp to etcd (/ha-failover/home-heartbeat) every 10s via
its local etcd member. No custom quorum/voting logic - etcd's own Raft
consensus (majority write/linearizable read) already gives the 'a
majority agrees' guarantee.
- scripts/ha-failover-watcher.sh + .service: runs on the VPS as a
systemd service (not k3s - must survive the VPS's own cluster being
unhealthy). Reads the heartbeat via its own local etcd member; if the
read itself fails, we can't tell if home is down or if this VPS is the
one partitioned, so it does NOT act (etcd's consensus requirement
provides the split-brain safety here, not custom code). If the read
succeeds and the heartbeat is stale (>45s, ~4-5 missed beats), home is
confirmed down by majority - promotes pg-authentik's Cluster CR
(spec.replica self/primary/source -> vps, no promotionToken - confirmed
live via dry-run that CNPG's webhook doesn't require one, it's only for
planned/graceful switchover LSN cross-checks that aren't available
during a genuine unplanned outage) and flips the
pg-authentik.ha.huskypup.net Cloudflare A record to the VPS's IP.
- scripts/ha-failback-authentik.sh: the deliberately manual reverse -
human confirms home is healthy and caught up before running this.
- Floating hostname plumbing: both sites' Authentik values now read
AUTHENTIK_POSTGRESQL__HOST/PORT as pg-authentik.ha.huskypup.net:61432
instead of the local pg-authentik-app secret's host/port, so app pods
on either site always reach whichever site is actually primary. Both
sites listen on the same external port (VPS gets a new
pg-authentik-forward.service socat forward, mirroring the existing
minio-forward.service pattern, so its NodePort 32433 is externally
reachable on 61432 same as home's UniFi-forwarded port).
- Found and worked around two real bugs surfaced while wiring this up
(fixed manually via scripts handed to the user - both blocked by the
Claude Code auto-mode classifier as credential-transmission /
cluster-DNS-edit actions):
- VPS's pg-authentik-app secret had a STALE password from before
streaming replication existed - the live Postgres role password now
replicates from home via WAL, but VPS's local K8s secret never got
updated to match. Needs a one-time sync (and again on any future
rotation).
- This UniFi does not support NAT hairpin/loopback for its own WAN
port-forwards - home's own pods resolving the floating hostname need
a local CoreDNS rewrite straight to pg-authentik-rw, confirmed live by
a DNS-resolves-but-TCP-connect-fails test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CNPG's new externalClusters connectionParameters were configured correctly
but streaming replication was silently failing - pg_stat_wal_receiver on
the VPS replica showed 0 rows, logs repeated 'could not connect to the
primary server: ... server closed the connection unexpectedly' every few
minutes.
Root cause: the authentik namespace is enrolled in Istio ambient mesh with
the mesh-wide default PeerAuthentication set to STRICT, and its
AuthorizationPolicy only allows traffic from specific in-mesh namespaces.
Traffic arriving via the ha-authentik-postgres NodePort from the VPS/
witness has no mesh identity at all (they're not in this cluster), so
ztunnel accepted the TCP connection then reset it once no HBONE/mTLS
handshake and no matching ALLOW rule ever arrived - confirmed live via
openssl s_client -starttls postgres (TCP connects, 0 bytes back).
Same root cause and same fix as the existing hostNetwork/webhook precedent
(infrastructure/istio/manifests/mesh/peer-authentication-webhooks.yaml):
- New port-scoped PERMISSIVE PeerAuthentication for the pg-authentik pods'
port 5432 only (not the whole namespace - Authentik's own in-mesh
east-west traffic stays STRICT).
- New port-scoped ALLOW rule on the existing AuthorizationPolicy, so any
source is allowed for port 5432 specifically, without touching the
existing namespace-based rules.
Both layers were needed - PERMISSIVE mTLS alone isn't enough, the
AuthorizationPolicy independently denies anything not matching one of its
existing rules.
Verified live: restarted the VPS replica pod to force an immediate
reconnect attempt: FATAL connection-reset errors stopped, and it's now
progressing through WAL restore toward a live streaming connection.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed live: 'spec.replica.source: Required value' even though home
isn't actually replicating from anyone (self == primary). Set to its own
self-referencing entry name.
Confirmed live against the vcluster.cnpg.io admission webhook: (1)
replica.enabled is mutually exclusive with primary/self - they're two
different mechanisms (old single-source replica-cluster mode vs the
newer distributed-topology model), not composable. (2) Both replica.self
and replica.primary must reference names present in THAT cluster's own
externalClusters list - including a self-referencing entry, even though
it describes the cluster itself. Renamed VPS's 'home-backup' external
cluster to 'home' throughout (bootstrap.recovery.source too) so one name
consistently satisfies barmanObjectStore, connectionParameters, and the
primary reference; added matching self-referencing entries on both
sides.
Adds connectionParameters (real streaming, not just barmanObjectStore
WAL-archive polling) to both sides' existing externalClusters entry, plus
replica.self/primary fields declaring the distributed topology (both
currently agree home is primary). Auth reuses CNPG's own auto-generated
streaming_replica client certs, cross-copied between clusters manually
(kubectl, not git - same pattern as every other cross-cluster secret
tonight) since they're the exact credential each side's pg_hba.conf
already trusts.
Also adds the VPS-side NodePort exposing its own pg-authentik primary
back to home (home->VPS already works via the existing netbird-egress
route, no UniFi port-forward needed for that direction - only the
reverse, VPS/witness reaching into home, needed the WAN workaround).
Part of the Authentik HA pilot (see plan doc). Home's etcd quorum member
(StatefulSet, pinned to talos-cp-01 for a stable advertised address) plus
a NodePort exposing pg-authentik's current primary - both reachable from
the VPS/witness over the already-authenticated Netbird mesh (confirmed
live: home nodes are directly reachable from Netbird peers on their real
node IP, via the netbird-egress DaemonSet's route). Deliberately NOT
going through UniFi/public-internet exposure - this stays entirely
inside the private mesh, a materially safer path than the WAN port-
forward originally considered.
Needs a scoped Kyverno PolicyException (ha-failover-nodeport-exception.yaml)
since disallow-nodeport-services is enforced cluster-wide - narrowly
scoped to Services named ha-*, matching the existing netbird-egress-
exemption.yaml precedent for exceptions.
Vault's OIDC auth method needs to resolve auth.kube.huskypup.net (the
issuer URL) from inside its own pod - no in-cluster DNS entry exists for
that public hostname, so add a hostAlias pointing at the live
istio-ingressgateway LoadBalancer IP (verified current: 172.28.101.244).
authentik-blueprints-vault (vault-blueprint.yaml) was already applied to
the cluster and referenced by the authentik Deployment for some time -
committing it now so git matches the live, working state instead of
leaving it as an untracked local file.
Found while building the VPS replica clusters (Phase 2): all three
CNPG clusters' backup.barmanObjectStore blocks only configure continuous
WAL archiving. CNPG's bootstrap.recovery needs at least one real Backup
object in the object store before it has anything to restore -
'no target backup found', confirmed live against pg-authentik. This gap
was systemic (gitlab has it too, not fixed here - out of scope for
tonight, no VPS standby depends on it).
immediate: true fires one backup right away in addition to the daily
01:00 schedule, to unblock the in-progress VPS restore now rather than
waiting up to 24h for the first scheduled run.
Root cause of tonight's earlier CrowdSec/CNPG-backup workarounds: no node
in the cluster had any route into the Netbird mesh CIDR (100.108.0.0/16)
for pod-originated traffic. The per-namespace netbird 'router' pods are
inbound-only infrastructure (external peers reaching K8s services); their
own architecture has no reverse path.
Fix, in two parts:
1. infrastructure/netbird/manifests/egress-daemonset.yaml - one netbird
client per node, hostNetwork so its wt0 interface lives in the node's
real network namespace, plus a sidecar that adds a host route sending
100.108.0.0/16 out via it. hostNetwork requires a scoped Kyverno
PolicyException (infrastructure/kyverno/policies/netbird-egress-exception.yaml)
to the disallow-host-namespaces STIG policy - narrowly for this one
DaemonSet by name, not a namespace-wide exclusion.
2. Discovered the route alone wasn't enough for k3s NodePort traffic
(vps-minio:30900): Netbird manages its own nftables ACLs independent of
iptables/Kyverno, and its forward chain (netbird-rt-fwd) only permits
*established* connections through a peer acting as a router - never new
ones, by design, unless a Netbird 'Network Route' policy is explicitly
configured (it isn't, for this VPS). Locally-terminated connections
(tinyproxy) go through a separate, already-permissive ACL chain, which
is why the CrowdSec proxy fix from earlier tonight worked. Replicated
that working pattern for MinIO: minio-forward.service on the VPS host
(systemd, socat) forwards 100.108.113.41:9000 -> MinIO's ClusterIP,
avoiding the NodePort path entirely.
Re-enabled everything that was disabled/suspended earlier tonight because
of this gap, pointed at the new endpoint:
- CrowdSec CAPI/console-enroll (removed DISABLE_ONLINE_API, restored the
VPS proxy env vars)
- n8n/nextcloud/authentik CNPG backup.barmanObjectStore
- vault-raft-snapshot CronJob (unsuspended)
- nextcloud PVC content sync CronJob endpoint
vps-minio.netbird.internal is retired everywhere - it was never actually
resolvable (Netbird has no DNS configured) even before today's routing
fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Discovered while debugging pg-n8n's recurring 'Instance Status Extraction
Error': the root cause is the same missing pod-egress route to the Netbird
mesh found tonight while fixing CrowdSec - continuousArchiving kept failing
to connect to vps-minio.netbird.internal, and CNPG correctly holds
Ready=False while archiving is broken (a real condition, not cosmetic).
That's what was driving these three apps' ArgoCD health flapping.
No node in the cluster has any route into 100.108.0.0/16 for
pod-originated traffic - the per-namespace netbird 'router' pods
(gitlab, vault, argocd, etc.) are inbound-only infrastructure, not egress
gateways. gitlab's own CNPG backup is unaffected (points at a local
in-cluster MinIO, not the VPS).
Commented out rather than deleted - re-enable once real pod-egress
routing exists, tracked as a separate task. No data loss: this is WAL
archiving/backup, not the live database.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Foundation for a DR/backup path using an always-on VPS as a second
ArgoCD-managed cluster, plus DB/backup standardization work that fell
out of it:
- vps-standby ArgoCD cluster destination + AppProject, MinIO backup
receiver, VPS bootstrap script (k3s, Netbird, cert-manager)
- Dual-site DNS failover watcher + home-IP DDNS CronJob, Cloudflare
token moved out of git into Vault+ExternalSecret
- Nextcloud migrated from ad-hoc MariaDB to CNPG + redis-operator
(matches n8n/Authentik/GitLab's backup-native pattern)
- Authentik's CNPG manifests moved into the actual ArgoCD-synced
manifests/ path (were present but never wired into the sync path)
- Vault raft-snapshot CronJob, CNPG barmanObjectStore backups
(Authentik/n8n/Nextcloud), Nextcloud file-PVC restic sync - all
targeting the new VPS MinIO receiver
See VPS Warm-Standby plan doc for full design rationale.
bitnami/kubectl has removed all version tags from Docker Hub,
leaving only :latest which is blocked by Kyverno disallow-latest-tag
policy. Switch to alpine/k8s:1.32.13 which has kubectl, bash, jq,
and uses proper version tags.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The :1.31 tag is gone from Docker Hub and :latest is blocked by Kyverno
disallow-latest-tag policy. The 1.29 image is cached on cluster nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The bitnami/kubectl:1.31 tag doesn't exist on Docker Hub. Updated all
hook jobs to use bitnami/kubectl:latest.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PreSync/PostSync hook Jobs reference argocd-hook-sa but it only existed
in the argocd namespace. Jobs running in app namespaces (crowdsec,
authentik, netbird, vault, istio-system) need the SA in their own
namespace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
