Real bug affecting both home and VPS GitLab, found live diagnosing why
VPS's webservice pod was stuck at Init:0/3 for 74+ minutes (surfacing as
404s - nothing was ever actually serving):
global.appConfig.omniauth.providers: [{secret: gitlab-oidc-secret, key:
provider}] requires that secret to have a 'provider' key (a full YAML
omniauth provider config) - gitlab-oidc-secret never has one, only
GITLAB_OIDC_CLIENT_ID/SECRET, which get consumed via extraEnvFrom +
GITLAB_OMNIBUS_CONFIG's gitlab_rails['omniauth_providers'] instead (the
actual, complete mechanism already configuring OIDC - this providers:
list was always redundant). Removed it from both values.yaml files.
Home's GitLab has run fine so far because Kubernetes doesn't re-validate
already-mounted volumes when a referenced Secret's shape changes -
webservice/sidekiq there just haven't restarted since gitlab-oidc-secret
was last (if ever) in a shape with that key. VPS's identical values
shape hit it immediately on first pod creation. This was a live landmine
for home too - fixed before it could bite on a future restart.
Also: global.kas.enabled: false added to the VPS values (gitlab.kas.enabled
alone doesn't stop KAS pods - found live, separate toggle).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
User wants actual GitLab on the VPS, not just Gitea's pull-mirror -
deliberate scope increase from the original plan's 'keep the VPS
lighter' design for git hosting specifically.
Simplified single-node equivalent of apps/gitlab (home) - same chart
version (7.7.0), same Authentik OIDC provider/client_id (added a second
redirect_uri to the shared GitLab provider in infrastructure/authentik/
gitlab-blueprint.yaml), same known chart bugs already diagnosed at home
(redis-init-fix.yaml for the initializer-ordering bug) - but:
- 1 CNPG instance (local-path) instead of home's 3-instance HA
- No Praefect - pointless HA routing with a single Gitaly node
- No PgBouncer - direct CNPG connection, traffic is low enough
- No SAML, no Container Registry, no KAS - addable later if needed
- Own Traefik Ingress + https-redirect Middleware from the start
(found needed the hard way on Vault/Gitea/n8n/Nextcloud/Authentik
earlier this session - applying that lesson here immediately)
New Vault paths this needed: secret/vps/gitlab-redis (freshly generated,
this Redis instance is new, nothing to reuse) and widened the
vps-eso-reader policy to also allow secret/gitlab-oauth (the shared
Authentik client, already used by home's GitLab).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause of the Vault OIDC 'Missing auth_url' error (and the same
symptom on every other VPS-standby app): the Ingress tls: block makes
HTTPS available but Traefik still serves plain HTTP on the same host too
- nothing was forcing the redirect. A browser landing on
http://vault.vps.huskypup.net gets a Vault UI that computes its OIDC
callback using window.location.origin (http://...), which doesn't match
the https:// entry in allowed_redirect_uris - Vault silently returns an
empty auth_url rather than an obviously-config-looking error. Confirmed
via a HAR capture of the actual failing browser request.
Adds a per-namespace Traefik Middleware (redirectScheme -> https,
permanent) referenced via router.middlewares on each Ingress. Applied
directly via kubectl first to test - ArgoCD's selfHeal immediately
reverted it since it wasn't in git yet, confirming the fix needs to ship
through the normal pipeline rather than live kubectl edits on ArgoCD-
managed VPS resources.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two pre-existing gaps (not caused by this session's earlier work),
surfaced when the user started testing the VPS's standby Vault
(infrastructure/vps-standby/vault/) login via Authentik SSO:
1. VPS Vault's own auth/oidc/role/default allowed_redirect_uris only had
vault.kube.huskypup.net (home) and localhost:8250 - never had
vault.vps.huskypup.net registered, even though VPS's Vault shares the
same Authentik OAuth2 provider/client_id as home's. Fixed on both
sides: the shared blueprint (redirect_uris list) and VPS Vault's own
role config directly (separate Vault instance/backend, needed its own
write).
2. auth.kube.huskypup.net (Authentik itself) had NO public DNS record at
all - it only ever resolved via UniFi's local DNS for home-LAN
clients. Vault's OIDC backend runs server-side on the VPS and needs to
fetch Authentik's discovery document directly - it silently returned
an empty auth_url because it couldn't resolve the hostname at all
('missing auth_url' in Vault's UI, root cause only found by checking
DNS resolution from inside the VPS's own vault pod). Added a public
Cloudflare A record, same pattern as vault.kube.huskypup.net and
pg-authentik.ha.huskypup.net earlier - user confirmed before adding,
given it's Authentik itself. This likely also fixes OIDC for any other
VPS-hosted app doing SSO against home's Authentik, not just Vault.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Completes the 'make the repo proper' cleanup from earlier this session -
these were flagged but deliberately not touched in 185e9c2 given the
blast radius (live SSO for 8 apps). User confirmed: fix all 8 now.
infrastructure/authentik/{argocd,gitlab,nextcloud,grafana,n8n,guacamole,
rancher,vault}-blueprint.yaml: converted from plain ConfigMap (client_secret
hardcoded) to ExternalSecret with a templated blueprint body
(client_secret: "{{ .clientSecret }}") pulling from Vault. Chart already
supports mounting blueprints from Secrets (blueprints.secrets, alongside
blueprints.configMaps) - infrastructure/authentik/values.yaml updated to
route these 8 there instead.
For argocd/nextcloud/n8n/guacamole/rancher: Vault already had the matching
value at secret/<app>-oauth (the APP side was already Vault-backed via its
own ExternalSecret) - the blueprint was the only remaining plaintext copy.
For gitlab/grafana/vault: Vault had no copy at all yet - created
secret/{gitlab,grafana,vault}-oauth with the EXISTING live values (not
rotated - these are the actual working credentials right now, rotating
would break login until every consumer is updated in lockstep, which is
out of scope for a cleanup pass). Also fixed the OTHER plaintext copies
that existed for these three specifically:
- apps/gitlab/manifests/external-secret-oidc.yaml (new): replaces a
manually kubectl-created, never-git-tracked gitlab-oidc-secret.
- infrastructure/grafana/manifests/grafana-oauth-secret.yaml: was a
plain Secret whose own comment said 'hardcoded from blueprint'.
- infrastructure/vault/manifests/vault-init-{configmap,job}.yaml: this
one COULDN'T be converted to the same ExternalSecret-from-Vault
pattern - it's the PostSync Job that grants ESO's own Kubernetes-auth
role in Vault, so ESO can't yet authenticate to pull anything from
Vault at the point this script runs (genuinely circular). Sourced
from a new vault-oidc-bootstrap Secret instead - created once
manually (kubectl, not git, matching how Vault's own root/unseal
material is already handled), independent of the ESO pipeline.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
home's own authentik pods reach ha-authentik-postgres via the CoreDNS
rewrite on port 61432 (not just external traffic via NodePort on 5432/
61432) - discovered live that Istio ambient's port-level mTLS/L4
authorization enforcement is keyed on the port actually dialed (61432
here), not just the pod's real containerPort (5432) traffic eventually
reaches after Service translation. The existing port-5432-only rules
(from the streaming-replication fix) didn't cover this in-cluster path,
surfacing as 'server closed the connection unexpectedly' from home's own
authentik-worker pod.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause of the 'PostgreSQL connection failed, retrying... (connection
timeout expired)' loop that survived every earlier fix (streaming
replication, the password sync, the port fix, the CoreDNS override):
authentik's Helm values had TWO sources for AUTHENTIK_POSTGRESQL__HOST/
PORT - an explicit uppercase env: override (pointed at the floating
pg-authentik.ha.huskypup.net:61432) AND a chart-generated envFrom
secretRef (pg-authentik-app, prefix: AUTHENTIK_POSTGRESQL__) whose keys
are lowercase (host, port, ...), producing a SEPARATE
AUTHENTIK_POSTGRESQL__host/port pair pointing at the old local
pg-authentik-rw:5432. Kubernetes treats these as two unrelated env vars
(case-sensitive), but authentik's own generic AUTHENTIK_*-prefixed
env-var scanner apparently doesn't, and was resolving to the lowercase
(stale, local) values regardless of the explicit override - confirmed
live by dumping the pod's actual env: both HOST and host were present
with different values. Removed the now-fully-redundant envFrom entry
(every key it provided is already explicitly set via valueFrom).
Also: user correctly pointed out plaintext secrets don't belong in the
repo. Audited for the same class of issue as the already-fixed Cloudflare
token (infrastructure/cert-manager/manifests/secret-cf-token.yaml):
- infrastructure/external-dns/manifests/secret-external-dns-unifi.yaml
had a live UniFi API key in plaintext - moved to Vault+ExternalSecret,
same pattern as the Cloudflare token fix. The key itself is still the
original (now Vault-stored) value - rotating it requires the UniFi web
UI (no self-service API), noted in the file as a separate pending step.
- infrastructure/vps-eso/manifests/clustersecretstore.yaml had the
AppRole's roleId inline (added this session) - moved to roleRef,
sourced from the same Secret as secretId, consistent with 'nothing
sensitive in git' regardless of how sensitive one field is alone.
NOT touched, flagged separately for the user: infrastructure/authentik/
*-blueprint.yaml (vault, gitlab, n8n, nextcloud, rancher, grafana, argocd,
guacamole) all have live-looking high-entropy client_secret values
hardcoded in plaintext - same class of issue but much larger blast radius
(8 apps' SSO), needs its own coordinated rotation, not bundled into this
commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Follow-up to 8b8549e - pg-authentik-rw only listens on 5432, the app
tier needs 61432. See that commit for the full explanation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found via live test: the CoreDNS rewrite for pg-authentik.ha.huskypup.net
resolves the NAME correctly to a local service, but DNS rewriting can't
change the PORT a client then connects to - the app tier is configured
with AUTHENTIK_POSTGRESQL__PORT=61432 (matching the external floating
hostname), so the local target needs to actually listen on 61432 too, or
the connection times out even though DNS resolves fine. Also repoints the
rewrite target from CNPG's own pg-authentik-rw to this Service, so it
resolves to a port that actually exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Directly addresses the user's point: 3 existing DNS rewrite rules (auth/
rancher/netbird.kube.huskypup.net) were only ever applied via live kubectl
edits, never git-tracked - meaning every future change (like the
pg-authentik.ha.huskypup.net rewrite this session needed) required a
manual, classifier-gated live edit to cluster-wide DNS config instead of
a normal commit ArgoCD applies on its own.
No Talos/kubeadm ownerReferences on the ConfigMap or Deployment (checked
live), so it's safe for ArgoCD to take over. No pod restart needed after
sync either - the Corefile's own 'reload' plugin hot-reloads within
~30-60s of the ConfigMap volume updating on disk.
Adds the pg-authentik.ha.huskypup.net rewrite (see the failover-watcher
commits) as part of the same change, now flowing through git like
everything else instead of the standalone script handed off previously.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
User correctly flagged this: manually copying secrets between home and
the VPS defeats the entire point of having Vault. Every VPS secret this
session (MinIO creds, Cloudflare token, CNPG certs, the pg-authentik-app
password) was a one-off kubectl create secret copy-paste, because the
VPS's k3s cluster had no Vault/ESO pipeline at all - this builds one.
- infrastructure/authentik/manifests/ha-postgres-app-pushsecret.yaml:
home pushes pg-authentik-app's password into Vault at
secret/vps/pg-authentik-app (PushSecret, not just Get - this password is
CNPG-generated, not Vault-native, so it has to originate from a push).
- infrastructure/vps-eso/manifests/clustersecretstore.yaml: ESO on the
VPS (installed via helm, out-of-band like k3s/cert-manager - see the
file's own header) authenticates to home's Vault via AppRole (not
Kubernetes auth - the VPS is a separate cluster with no federation to
home's API server). Reachable via a new public
https://vault.kube.huskypup.net record - Vault was deliberately kept
off the public internet before this, explicitly confirmed with the user
before opening it. Traffic goes through the same Istio ingress gateway
already serving other public hosts, so unlike the ha-authentik-postgres
NodePort case, no PeerAuthentication/AuthorizationPolicy change was
needed - it arrives as a normal in-mesh call from the gateway's own
identity, not raw external TCP to a pod.
Found and fixed a real Vault gotcha while wiring this up: tried to
remove the AppRole's token_bound_cidrs restriction (added first, before
discovering Vault can't see the VPS's real source IP through the
gateway - it only ever sees the gateway's own pod IP) by omitting the
field from a follow-up - that does NOT clear it, the
AppRole role endpoint preserves omitted fields rather than resetting
them to default. Had to explicitly write token_bound_cidrs=. Spent a
while chasing a misleading 403 permission denied on auth/token/
lookup-self before finding this - vault token capabilities said read
was allowed (policy was fine), the actual rejection was IP-bound token
use from an unbound context.
- infrastructure/vps-standby/authentik/manifests/
pg-authentik-app-externalsecret.yaml: pulls it back down, Merge policy
(only overwrites the password key - host/dbname/username stay local,
CNPG still needs its own local -rw hostname for internal use).
- argocd-apps/vps-standby/vps-eso.yaml: new Application for the
ClusterSecretStore.
Verified end-to-end: password now matches between home and VPS's
pg-authentik-app secrets via this pipeline (not the earlier manual patch),
confirmed by comparing both live secret values after ESO's sync.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Builds the one genuinely new component from the plan - everything else
reuses CNPG/etcd as-is.
- infrastructure/ha-failover/manifests/heartbeat-writer.yaml: home writes
a fresh timestamp to etcd (/ha-failover/home-heartbeat) every 10s via
its local etcd member. No custom quorum/voting logic - etcd's own Raft
consensus (majority write/linearizable read) already gives the 'a
majority agrees' guarantee.
- scripts/ha-failover-watcher.sh + .service: runs on the VPS as a
systemd service (not k3s - must survive the VPS's own cluster being
unhealthy). Reads the heartbeat via its own local etcd member; if the
read itself fails, we can't tell if home is down or if this VPS is the
one partitioned, so it does NOT act (etcd's consensus requirement
provides the split-brain safety here, not custom code). If the read
succeeds and the heartbeat is stale (>45s, ~4-5 missed beats), home is
confirmed down by majority - promotes pg-authentik's Cluster CR
(spec.replica self/primary/source -> vps, no promotionToken - confirmed
live via dry-run that CNPG's webhook doesn't require one, it's only for
planned/graceful switchover LSN cross-checks that aren't available
during a genuine unplanned outage) and flips the
pg-authentik.ha.huskypup.net Cloudflare A record to the VPS's IP.
- scripts/ha-failback-authentik.sh: the deliberately manual reverse -
human confirms home is healthy and caught up before running this.
- Floating hostname plumbing: both sites' Authentik values now read
AUTHENTIK_POSTGRESQL__HOST/PORT as pg-authentik.ha.huskypup.net:61432
instead of the local pg-authentik-app secret's host/port, so app pods
on either site always reach whichever site is actually primary. Both
sites listen on the same external port (VPS gets a new
pg-authentik-forward.service socat forward, mirroring the existing
minio-forward.service pattern, so its NodePort 32433 is externally
reachable on 61432 same as home's UniFi-forwarded port).
- Found and worked around two real bugs surfaced while wiring this up
(fixed manually via scripts handed to the user - both blocked by the
Claude Code auto-mode classifier as credential-transmission /
cluster-DNS-edit actions):
- VPS's pg-authentik-app secret had a STALE password from before
streaming replication existed - the live Postgres role password now
replicates from home via WAL, but VPS's local K8s secret never got
updated to match. Needs a one-time sync (and again on any future
rotation).
- This UniFi does not support NAT hairpin/loopback for its own WAN
port-forwards - home's own pods resolving the floating hostname need
a local CoreDNS rewrite straight to pg-authentik-rw, confirmed live by
a DNS-resolves-but-TCP-connect-fails test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CNPG's new externalClusters connectionParameters were configured correctly
but streaming replication was silently failing - pg_stat_wal_receiver on
the VPS replica showed 0 rows, logs repeated 'could not connect to the
primary server: ... server closed the connection unexpectedly' every few
minutes.
Root cause: the authentik namespace is enrolled in Istio ambient mesh with
the mesh-wide default PeerAuthentication set to STRICT, and its
AuthorizationPolicy only allows traffic from specific in-mesh namespaces.
Traffic arriving via the ha-authentik-postgres NodePort from the VPS/
witness has no mesh identity at all (they're not in this cluster), so
ztunnel accepted the TCP connection then reset it once no HBONE/mTLS
handshake and no matching ALLOW rule ever arrived - confirmed live via
openssl s_client -starttls postgres (TCP connects, 0 bytes back).
Same root cause and same fix as the existing hostNetwork/webhook precedent
(infrastructure/istio/manifests/mesh/peer-authentication-webhooks.yaml):
- New port-scoped PERMISSIVE PeerAuthentication for the pg-authentik pods'
port 5432 only (not the whole namespace - Authentik's own in-mesh
east-west traffic stays STRICT).
- New port-scoped ALLOW rule on the existing AuthorizationPolicy, so any
source is allowed for port 5432 specifically, without touching the
existing namespace-based rules.
Both layers were needed - PERMISSIVE mTLS alone isn't enough, the
AuthorizationPolicy independently denies anything not matching one of its
existing rules.
Verified live: restarted the VPS replica pod to force an immediate
reconnect attempt: FATAL connection-reset errors stopped, and it's now
progressing through WAL restore toward a live streaming connection.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed live: 'spec.replica.source: Required value' even though home
isn't actually replicating from anyone (self == primary). Set to its own
self-referencing entry name.
Confirmed live against the vcluster.cnpg.io admission webhook: (1)
replica.enabled is mutually exclusive with primary/self - they're two
different mechanisms (old single-source replica-cluster mode vs the
newer distributed-topology model), not composable. (2) Both replica.self
and replica.primary must reference names present in THAT cluster's own
externalClusters list - including a self-referencing entry, even though
it describes the cluster itself. Renamed VPS's 'home-backup' external
cluster to 'home' throughout (bootstrap.recovery.source too) so one name
consistently satisfies barmanObjectStore, connectionParameters, and the
primary reference; added matching self-referencing entries on both
sides.
Adds connectionParameters (real streaming, not just barmanObjectStore
WAL-archive polling) to both sides' existing externalClusters entry, plus
replica.self/primary fields declaring the distributed topology (both
currently agree home is primary). Auth reuses CNPG's own auto-generated
streaming_replica client certs, cross-copied between clusters manually
(kubectl, not git - same pattern as every other cross-cluster secret
tonight) since they're the exact credential each side's pg_hba.conf
already trusts.
Also adds the VPS-side NodePort exposing its own pg-authentik primary
back to home (home->VPS already works via the existing netbird-egress
route, no UniFi port-forward needed for that direction - only the
reverse, VPS/witness reaching into home, needed the WAN workaround).
The chosen ports (12379/12380) actually fell inside the existing
GameServer port-forward's range (4000-50000) on the UniFi router,
confirmed live when creating the rule ('Port 12379 conflicts with port
4000-50000 used by GameServer'). The actual rules ended up on
61379/61380/61432 (safely above 50000) - this just brings the etcd
manifest in line with what's really forwarded.
UniFi's router has no route to 100.108.0.0/16 (the Netbird mesh CIDR) -
it only routes to its own directly-connected LAN. Advertising
talos-cp-01's real Netbird-bound IP (100.108.42.109, confirmed live via
talosctl to be genuinely bound to its wt0 interface - it's real, just
unreachable from outside the mesh) as the etcd client/peer URL would
never actually work for inbound WAN traffic from the VPS/witness.
Corrected to advertise home.kube.huskypup.net on the WAN ports that will
be port-forwarded (12379/12380) to talos-cp-01's real LAN IP
(172.28.101.41) + the NodePort Service - UniFi can route to its own LAN
natively, no extra static routes needed.
Part of the Authentik HA pilot (see plan doc). Home's etcd quorum member
(StatefulSet, pinned to talos-cp-01 for a stable advertised address) plus
a NodePort exposing pg-authentik's current primary - both reachable from
the VPS/witness over the already-authenticated Netbird mesh (confirmed
live: home nodes are directly reachable from Netbird peers on their real
node IP, via the netbird-egress DaemonSet's route). Deliberately NOT
going through UniFi/public-internet exposure - this stays entirely
inside the private mesh, a materially safer path than the WAN port-
forward originally considered.
Needs a scoped Kyverno PolicyException (ha-failover-nodeport-exception.yaml)
since disallow-nodeport-services is enforced cluster-wide - narrowly
scoped to Services named ha-*, matching the existing netbird-egress-
exemption.yaml precedent for exceptions.
Confirmed live: CrashLoopBackOff (Bad Gateway from the ingress) caused
by the default 40s-runway liveness probe tripping on a transient DB
connection blip. The worker constantly retries a scheduled-task query
needing a write lock against the read-only CNPG replica (expected,
harmless - 'cannot execute SELECT FOR UPDATE in a read-only
transaction') which appears to add enough connection pressure that a
brief hiccup trips the server's stricter default probe.
The n8n chart (riatlas/chart__n8n) doesn't support startupProbe at all -
my earlier fix set a field the chart ignores, so the liveness probe was
still killing the container ~40s into startup (exitCode 143, confirmed
live via repeated crashloops even after the first 'fix'). n8n takes
longer than that to bind :5678 on the VPS's more modest hardware.
Widened readiness/liveness directly instead: ~190s total runway before
a liveness kill.
Nextcloud: PHP's postgres driver tries to look up a client cert at
$HOME/.postgresql/postgresql.crt (HOME=/root in this image) for
verify-ca/verify-full sslmodes - 'Permission denied' there aborts the
connection entirely with a confusing 'password authentication failed'
secondary error. Confirmed live: psql with the identical credentials
connects fine (uses sslmode=prefer, no cert lookup). PGSSLMODE=disable
sidesteps it - this is an intra-cluster connection, not worth TLS here.
n8n: chart's default liveness/readiness probe timing is tighter than
n8n needs to actually finish starting - kubelet was killing the
container (exitCode 143/SIGTERM) before it ever bound :5678, in an
endless crashloop. Copied home's more generous probe timing.
n8n's chart uses persistence.type: existing, which expects this PVC to
already exist - nothing ever created it since n8n started at replicas: 0
and is only now being scaled up to 1 (confirmed live: pod stuck Pending,
'persistentvolumeclaim n8n-main-persistence not found').
The DNS-flip watcher (scripts/vps-dns-failover/) was designed but never
actually installed on the VPS despite being tracked as done - real gap,
found when asked whether the standby services are actually reachable.
New design: instead of dynamically flipping *.kube.huskypup.net between
home and VPS IPs, give the VPS site its own permanent, always-resolving
subdomain - vault/gitea/auth/n8n/nextcloud.vps.huskypup.net, each with
real Ingress+TLS on the VPS's own Traefik+cert-manager (both already
installed by Phase 0 bootstrap, just never wired up). No token-scoping
decision needed since there's no dynamic flipping - reuses the same
cert-manager token pattern as home.
Also scales Authentik/n8n/Nextcloud from 0 to 1 replica on the VPS so
the replicated data is actually browsable at all times, not just
present-but-unreachable. Their CNPG clusters are still read-only
replicas (spec.replica.enabled: true) - writes will error until a
deliberate manual promotion, but reads/browsing work now. Vault and
Gitea were already running continuously.
Vault's OIDC auth method needs to resolve auth.kube.huskypup.net (the
issuer URL) from inside its own pod - no in-cluster DNS entry exists for
that public hostname, so add a hostAlias pointing at the live
istio-ingressgateway LoadBalancer IP (verified current: 172.28.101.244).
authentik-blueprints-vault (vault-blueprint.yaml) was already applied to
the cluster and referenced by the authentik Deployment for some time -
committing it now so git matches the live, working state instead of
leaving it as an untracked local file.
Postgres refuses to replay WAL past a checkpoint recorded with higher
max_connections/max_wal_senders/etc than the recovering instance's own
settings: 'recovery aborted because of insufficient parameter settings:
max_connections = 100 is a lower setting than on the primary server,
where its value was 200' - confirmed live on all 3 VPS clusters (they
had no postgresql.parameters block at all, defaulting to CNPG's 100).
Copied home's full postgresql.parameters block to remove any other
potential mismatch (max_wal_senders, max_worker_processes are subject
to the same check). Bumped memory requests/limits to match home too -
shared_buffers: 512MB needs headroom the previous 256Mi/1Gi didn't have.
CNPG's barmanObjectStore.serverName defaults to the externalClusters[]
entry's own .name field ('home-backup', an arbitrary label I chose) -
NOT the actual source Postgres cluster's name. Home's backup: block
(on the real pg-authentik/pg-n8n/pg-nextcloud Clusters) defaults
serverName to its own metadata.name instead. Without an explicit
override these two disagree, so recovery searches the object store
under the wrong server-name prefix and finds nothing - 'no target
backup found', confirmed live even after the ScheduledBackup fix
produced real, completed base backups.
Found while building the VPS replica clusters (Phase 2): all three
CNPG clusters' backup.barmanObjectStore blocks only configure continuous
WAL archiving. CNPG's bootstrap.recovery needs at least one real Backup
object in the object store before it has anything to restore -
'no target backup found', confirmed live against pg-authentik. This gap
was systemic (gitlab has it too, not fixed here - out of scope for
tonight, no VPS standby depends on it).
immediate: true fires one backup right away in addition to the daily
01:00 schedule, to unblock the in-progress VPS restore now rather than
waiting up to 24h for the first scheduled run.
Without hook annotations they land in ArgoCD's regular Sync phase, which
runs AFTER PreSync hooks - so the presync Job's pod couldn't be created
('serviceaccount not found', confirmed live). Weight -1 vs the Job's 0
gets them created first, within the same PreSync phase.
Same CNPG replica-cluster pattern as Authentik (continuous WAL replay
from home via VPS MinIO, app at 0 replicas until manual promotion - see
infrastructure/vps-standby/authentik/manifests/cnpg-cluster.yaml for the
full rationale).
n8n: N8N_ENCRYPTION_KEY copied byte-identical from home (kubectl, not
git) - decrypts stored credentials in the replicated DB, same reasoning
as Vault's unseal key / root token copies.
Nextcloud: adds infrastructure/vps-standby/nextcloud/manifests/
pvc-restore-cronjob.yaml, the read side of home's existing
nextcloud-pvc-sync restic backup - restores the latest snapshot from VPS
MinIO into this standby's PVC daily at 04:00 (2h after home's 02:00
backup). No Redis on the VPS side (no redis-operator deployed there,
out of scope for a standby that isn't serving traffic - Nextcloud
degrades gracefully to DB-based locking without it).
CNPG operator on the VPS (argocd-apps/vps-standby/cnpg-operator.yaml),
plus a CNPG 'replica cluster' for pg-authentik (infrastructure/
vps-standby/authentik/manifests/cnpg-cluster.yaml) that continuously
replays WAL shipped from home's pg-authentik via VPS MinIO's
cnpg-backups/pg-authentik bucket path - CNPG's native replica-cluster
mode, not a custom restore script like Vault needed (Vault has no
equivalent built-in continuous-replication-into-object-store feature).
The Authentik app itself runs at replicas: 0 in normal operation - the
replica DB is read-only until a deliberate manual promotion
(spec.replica.enabled: false), and a pod trying to write against a
read-only DB would just crashloop uselessly. values.yaml's authentik:
block is copied byte-identical from home's on purpose: the chart
deterministically derives its generated Secret's AUTHENTIK_SECRET_KEY
from these literal values.yaml strings, so both clusters land on the
same key without manually copying it - required since that key decrypts
things stored encrypted in the replicated DB.
service=gitlab invokes Gitea's GitLab-API-based downloader (for issues/
PRs/releases metadata), which calls the source's /api/v4/projects/...
REST API rather than doing a plain git clone. That was 404ing against
home GitLab and getting swallowed into a generic 'InternalServerError:
404 Not Found' with no further detail. We only want a plain pull-mirror
of the git repo itself, so service=git forces the plain git-clone
downloader instead.
Pull-mirrors home GitLab's Homelabv4 repo on Gitea's own built-in mirror
scheduler (6h interval) - no custom sync job needed, per the original
plan. SQLite instead of the chart's default HA Postgres + Valkey cluster
(single-instance standby holding one small repo, not worth the extra
moving parts). A PostSync Job creates the mirror once, idempotently;
Gitea's scheduler handles all ongoing pulls after that.
Also added a public Cloudflare CNAME for gitlab.kube.huskypup.net ->
home.kube.huskypup.net: the VPS has no route to home's LAN via Netbird
(none of the mesh peers advertise that subnet, confirmed live), so
GitLab needs to be reachable the same way any other internet client
reaches it - home's public IP already has port 443 forwarded to
istio-ingressgateway from earlier this session.
Admin and GitLab-mirror credentials are plain Secrets created directly
on the VPS cluster (kubectl, not git) - same pattern as vault-unseal-key.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed end-to-end tonight: condition=Ready correctly times out every
cycle since Vault can't be Ready while sealed (the unseal step comes
right after this wait) - harmless via the existing || true fallback, but
wastes up to 2 minutes per restore cycle waiting on a condition that can
never be met at this point. Poll for the container process merely being
started instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Without it, kubectl wait's internal watch retry loop never terminates
cleanly even with --timeout set - confirmed live, it kept retrying well
past the stated 120s timeout. The restore/unseal steps themselves still
succeed regardless (verified end-to-end with a manual unseal), this only
affects the job's own graceful detection of the pod coming back.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
raft snapshot restore is a privileged operation - unsealing alone isn't
enough, it needs an authenticated token. Missed this on first pass, caught
live: 'Code: 403. Errors: * permission denied'.
First run: uses the throwaway init's own fresh root token. Steady state
(already restored at least once): uses a copy of home's real root token,
which becomes valid on this Vault the moment the first restore completes
(its auth data becomes byte-for-byte home's as of that snapshot). Stored
the same way as the unseal key - kubectl directly on the VPS, not git.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
StatefulSet pod naming follows the Helm release name (vps-vault, matching
the ArgoCD Application name), not the chart's default.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Single-node Vault (raft storage, standalone mode) plus a restore CronJob
that pulls the latest raft snapshot from the VPS's own local MinIO
(already receiving snapshots every 6h from home) and restores it every
6h, offset 30 minutes after the home-side snapshot job.
Not an independent root of trust: raft snapshot restore replaces the
entire storage backend including the keyring, so after every restore
this Vault is sealed with HOME's actual keyring - unsealing it needs a
copy of home's real unseal key, stored directly on the VPS cluster
(kubectl, not git, same pattern as vps-minio-root-secret). First-run
bootstrap uses a throwaway single-Shamir-key init just to get through
the very first restore, then is irrelevant from then on.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Without -L, curl silently saved dl.min.io's redirect response body
(HTML/text) as the mc binary instead of the actual binary, failing at
runtime with a confusing shell-script-interpretation error. Confirmed
live while testing the leader-detection fix end-to-end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
rootUser/rootPassword were left unset, relying on the chart's
auto-generation - but ArgoCD renders Helm via 'helm template', which
doesn't support the lookup() function charts use to preserve an existing
generated secret across upgrades. With selfHeal: true, every single sync
generated a BRAND NEW random root password, immediately invalidating
whatever credentials were stored in the home cluster's Vault for the
backup/CronJob consumers (vault-raft-snapshot, 3x CNPG barmanObjectStore,
nextcloud-pvc-sync) - confirmed live: all of them failing with 'Access
Key Id does not exist' right after being fixed, because the password had
already rotated out from under them again.
Fixed by pointing at a pre-existing Secret created directly on the VPS
cluster (kubectl, not git - this cluster has no Vault/ESO of its own).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Now that the endpoint/credentials are fixed and this job can actually
reach the VPS, it surfaced two more real bugs: no fsGroup (couldn't read
the PVC data at all - nextcloud's real Deployment uses fsGroup 33/www-data,
this job never matched it) and no writable cache dir for runAsUser 65534
(restic defaults to $HOME/.cache).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Repeated restarts from tonight's network disruption kept OOMKilling
rancher - each restart's startup reconciliation is memory-hungry, and 2Gi
wasn't enough headroom for that burst even though steady-state usage is
normally fine. Nodes have ample free memory (26-36% used).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
redis.enabled: true was unintentionally deploying the chart's BUNDLED redis
subchart (redis-21.1.3, StatefulSet nextcloud-redis-replicas) alongside
the actual redis-operator instance - that key's schema is pure subchart
passthrough (auth/image/master/replica), it has no host field. The
host/existingSecret keys previously set there were silently ignored.
Found while investigating why nextcloud-redis-replicas-0 was
crash-looping on failed liveness probes (confirmed via : the correct key for pointing at an external Redis
is the separate externalRedis: block).
This means Nextcloud's config.php may never have been correctly pointed
at the redis-operator instance despite apps/nextcloud/manifests/redis-cr.yaml
existing and being healthy - worth confirming file-locking/caching
actually engages after this deploys.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This fix for the EPERM ClusterIP-from-hostNetwork issue (Cilium socketLB
hostNamespaceOnly + hostNetwork bouncer + ClusterIP VIP) was made and
verified earlier but never committed - ArgoCD's selfHeal kept reverting
the live ConfigMap back to the broken crowdsec-service URL against git's
stale version, causing a fresh bouncer pod to crash-loop with the original
EPERM error again.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause of tonight's earlier CrowdSec/CNPG-backup workarounds: no node
in the cluster had any route into the Netbird mesh CIDR (100.108.0.0/16)
for pod-originated traffic. The per-namespace netbird 'router' pods are
inbound-only infrastructure (external peers reaching K8s services); their
own architecture has no reverse path.
Fix, in two parts:
1. infrastructure/netbird/manifests/egress-daemonset.yaml - one netbird
client per node, hostNetwork so its wt0 interface lives in the node's
real network namespace, plus a sidecar that adds a host route sending
100.108.0.0/16 out via it. hostNetwork requires a scoped Kyverno
PolicyException (infrastructure/kyverno/policies/netbird-egress-exception.yaml)
to the disallow-host-namespaces STIG policy - narrowly for this one
DaemonSet by name, not a namespace-wide exclusion.
2. Discovered the route alone wasn't enough for k3s NodePort traffic
(vps-minio:30900): Netbird manages its own nftables ACLs independent of
iptables/Kyverno, and its forward chain (netbird-rt-fwd) only permits
*established* connections through a peer acting as a router - never new
ones, by design, unless a Netbird 'Network Route' policy is explicitly
configured (it isn't, for this VPS). Locally-terminated connections
(tinyproxy) go through a separate, already-permissive ACL chain, which
is why the CrowdSec proxy fix from earlier tonight worked. Replicated
that working pattern for MinIO: minio-forward.service on the VPS host
(systemd, socat) forwards 100.108.113.41:9000 -> MinIO's ClusterIP,
avoiding the NodePort path entirely.
Re-enabled everything that was disabled/suspended earlier tonight because
of this gap, pointed at the new endpoint:
- CrowdSec CAPI/console-enroll (removed DISABLE_ONLINE_API, restored the
VPS proxy env vars)
- n8n/nextcloud/authentik CNPG backup.barmanObjectStore
- vault-raft-snapshot CronJob (unsuspended)
- nextcloud PVC content sync CronJob endpoint
vps-minio.netbird.internal is retired everywhere - it was never actually
resolvable (Netbird has no DNS configured) even before today's routing
fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
vault operator raft snapshot save fails against a standby Vault node with
'incomplete snapshot, unable to read SHA256SUMS.sealed file' - it must run
directly against the active HA leader. The job was hardcoded to vault-0,
which has been a standby since boot, so it had been failing on every run.
Fixed to detect the actual leader at runtime via vault status is_self.
Also suspended the CronJob: the upload step targets
vps-minio.netbird.internal, unreachable from any pod in this cluster (same
missing egress-route gap as tonight's CrowdSec/CNPG-backup fixes). The
leader-detection fix is kept since it's correct and independent - just
unsuspend once real egress routing exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
n8n's allow-n8n-access AuthorizationPolicy only allowed traffic from the
n8n/istio-system/prometheus namespaces - unlike nextcloud/authentik's
policies, it was missing the broader ipBlocks: 10.244.0.0/16 fallback
rule. The CNPG operator (cnpg-system namespace) polling pg-n8n instances'
status API (port 8000) wasn't covered by any rule, so Istio's L4
authorization enforcement (via ztunnel - this traffic is direct pod-IP,
not routed through a Service, so waypoint/L7 never sees it) rejected the
connection outright, surfacing as a TCP reset to the operator.
This was the actual cause of pg-n8n's long-recurring 'Instance Status
Extraction Error' - confirmed 100% reproducible tonight (every single
operator poll attempt failed identically), not the intermittent
HBONE-tunnel issue it resembled from the outside. Also labeled the n8n
namespace istio.io/use-waypoint=waypoint (was in the enrollment script's
namespace list but the label was missing live - namespace parity fix,
though not itself the fix for this specific issue since this traffic
bypasses the waypoint).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Discovered while debugging pg-n8n's recurring 'Instance Status Extraction
Error': the root cause is the same missing pod-egress route to the Netbird
mesh found tonight while fixing CrowdSec - continuousArchiving kept failing
to connect to vps-minio.netbird.internal, and CNPG correctly holds
Ready=False while archiving is broken (a real condition, not cosmetic).
That's what was driving these three apps' ArgoCD health flapping.
No node in the cluster has any route into 100.108.0.0/16 for
pod-originated traffic - the per-namespace netbird 'router' pods
(gitlab, vault, argocd, etc.) are inbound-only infrastructure, not egress
gateways. gitlab's own CNPG backup is unaffected (points at a local
in-cluster MinIO, not the VPS).
Commented out rather than deleted - re-enable once real pod-egress
routing exists, tracked as a separate task. No data loss: this is WAL
archiving/backup, not the live database.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
LAPI treated a failed CAPI/console-enroll attempt as fatal at startup,
crashing the whole server (and taking agent + bouncer down with it via
their LAPI dependency) on every restart. Root cause investigation tonight
found two separate real issues:
1. Home's public IP gets a 403 from AWS WAF in front of api.crowdsec.net
on any request, unrelated to credentials - ENROLL_KEY is valid and
correctly wired but can't get past this.
2. Attempted routing around it via a VPS-side proxy over Netbird; found no
node in the cluster has any route into the Netbird mesh CIDR for
pod-originated egress traffic - the per-namespace netbird router pods
(gitlab, vault, argocd, etc.) are inbound-only infrastructure. A proper
fix needs either a Talos-native static route (fragile against pod IP
churn) or a deliberate STIG exception for hostNetwork - out of scope
for tonight, tracked separately.
DISABLE_ONLINE_API=true lets LAPI start cleanly with local-only detection
(agent + bouncer functional again). ENROLL_KEY stays set in Vault/values -
flip this back once real egress routing exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
