- infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml:
creationPolicy Merge -> Owner (already applied live, committing to
match). Merge assumed CNPG creates a base pg-gitlab-app/pg-praefect-app
secret to merge a password into - wrong for these VPS Clusters since
Phase 1c's replica-cluster recreate; CNPG doesn't create a local
owner secret for a replicated role. Left pg-gitlab-app/pg-praefect-app
missing entirely, crash-looping vps-gitlab-webservice on
ActiveRecord::DatabaseConnectionError for ~39h.
- infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml:
exclude config.php from the restic restore. The job's own comment
assumed nextcloud runs at replicas: 0 on the VPS; live confirms
replicas: 1 (no such key was ever actually set) - restic couldn't
overwrite the live pod's config.php (permission denied), failing the
whole job every run for 2+ days even though everything else restored
fine. Excluding it is correct regardless of permissions - a live
pod's own config.php shouldn't be overwritten by a background
restore job.
Also fixed live (not git-tracked, config.php is PVC-persisted runtime
state, not sourced from git):
- home + VPS nextcloud config.php dbpassword: out of sync with CNPG's
actual current pg-nextcloud-app password (baked in once at install,
never re-synced). VPS's case was two-layered - its own
pg-nextcloud-app secret also didn't match the real Postgres role
password, since VPS's pg-nextcloud is a read-only streaming replica
(spec.replica.enabled) and the authoritative password lives on home.
- grafana: broke a RollingUpdate deadlock (single-replica Deployment +
RWO Ceph volume - new pod couldn't start while the old pod still
held the only-one-node-at-a-time attachment, and the rollout
wouldn't scale down the old pod until the new one was ready).
- suspended the stale gitlab-mirror-sync CronJob on the VPS (spamming
auth failures every ~15min since its stored token predates the
Postgres-replication cutover of VPS GitLab's DB) rather than
deleting it, per the plan's own note to defer that until Phase 2b is
verified.
global.workhorse.tls.enabled replaced workhorse's plaintext listener
with a TLS-only one on port 8181 - but that port isn't internal-API
-only, it's also what gitlab-web HTTPRoute routes ALL public
gitlab.kube.huskypup.net traffic to. Envoy terminates public HTTPS
and proxies to the pod in plaintext, so the TLS-only listener 400'd
every public request ("Client sent an HTTP request to an HTTPS
server"). Restored live via a direct ConfigMap patch +
rollout restart before this commit could even be pushed, since the
outage took out this repo's own GitLab-hosted git access too.
global.workhorse.host stays (harmless without tls.enabled). Securing
the VPS's future internal-API-over-WAN hop needs a different
mechanism that doesn't collide with the public listener - not solved
here.
GitLab's new Gitaly/workhorse TLS certs are the first thing needing a
real Let's Encrypt cert under ha.huskypup.net (the cross-site floating-
hostname zone) - everything else there (Postgres) uses CNPG's own
self-signed certs, never hit this gap. Confirmed live: DNS-01 order
failed with 'no configured challenge solvers can be used for this
challenge' until this zone was added to the selector.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per the approved plan (Istio federation scoped separately, current
NodePort/PeerAuth/cert-manager pattern continues for these 2 remaining
links). Real Let's Encrypt certs (not self-signed) via the same DNS-01
ClusterIssuer used everywhere else - both sites already trust Let's
Encrypt's public CA, no custom CA distribution needed.
- global.gitaly.tls.enabled: adds tls_listen_addr (8076) alongside the
existing plaintext listener (8075, stays for in-cluster traffic
that doesn't need it). Requires
global.praefect.virtualStorages[0].tlsSecretName (chart's own NOTES.txt
validation rejects the render without it, confirmed live).
- global.workhorse.tls.enabled + global.workhorse.host: Gitaly's [gitlab]
url (internal-API callback for gitlab-shell hooks) switches from an
unpublishable *.svc.cluster.local name to a real floating hostname
(gitlab-internal-api.ha.huskypup.net) that Let's Encrypt CAN certify -
confirmed live this replaces the plaintext [[listeners]] block with
[listeners.tls] on the SAME port (8181), not a second port like Gitaly.
- praefect-ha-configmap.yaml (the manually-patched live config) updated
to tls:// + floating hostnames for all 3 home nodes, so home's own
Praefect uses the identical addresses it'll hand the VPS as peer
addresses once Phase 2b's 4th node lands.
- New CoreDNS rewrite for gitlab-internal-api.ha.huskypup.net (same
local-rewrite pattern as pg-*/gitaly-* already have).
Sequencing note: applying live - Certificates must issue before the
TLS-enabling values land, or Gitaly/webservice pods fail to start
(missing secret for their init container's cert copy).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per-pod NodePort Services for the 3 existing Gitaly nodes
(ha-gitaly-nodeport.yaml) plus home's Rails internal API
(ha-gitlab-internal-api-nodeport.yaml, needed for gitlab-shell hooks
whenever the VPS node is primary for a repo). PeerAuthentication
PERMISSIVE + AuthorizationPolicy port rules for both, matching the
proven Postgres pattern. CoreDNS rewrites so home's own Praefect
resolves the floating hostnames locally instead of round-tripping.
Reached over the Netbird mesh, not the UniFi WAN forward used for
Postgres - Gitaly gRPC here is plaintext (carries the gitaly_token,
gitlab_shell_secret, and raw repo contents), unlike CNPG's
TLS-negotiated stream.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The CONFIG_TEMPLATE_DIRECTORY redirect from the previous commit doesn't
work - confirmed live that extraVolumes is a dead values key for the
Praefect subchart specifically (its statefulset.yaml never calls the
gitlab.extraVolumes helper in its volumes: list, only volumeMounts
calls the corresponding helper - a real chart limitation, not a config
mistake). A dangling volumeMount with no matching volume would have
failed to schedule.
Real fix: ignoreDifferences on ConfigMap gitlab-praefect's data field
(argocd-apps/apps/gitlab.yaml) lets Helm create the object normally
while ArgoCD stops reconciling its content afterward - the actual
config gets kubectl-patched onto the live object directly.
praefect-ha-configmap.yaml is now a git-tracked reference/documentation
copy (deployed under its own harmless name) rather than something
Helm/ArgoCD wires in on its own.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Proves the CONFIG_TEMPLATE_DIRECTORY redirect works before Phase 2b
adds any actual cross-site dependency. Content is byte-identical to
the chart's current rendering (confirmed live) - this commit only
tests the override plumbing itself: a new ConfigMap
(praefect-ha-configmap.yaml) mounted at a different path than the
chart's own /etc/gitaly/templates (can't reuse that path/volume name -
Kubernetes rejects duplicates), with CONFIG_TEMPLATE_DIRECTORY
overridden via extraEnv to point at it instead (last-wins env
semantics, confirmed Kubernetes-documented behavior).
Also removes gitlab.praefect.virtualStorages - confirmed live dead
config, global.praefect.virtualStorages is what the chart's template
actually reads.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds externalClusters connectionParameters + replica.self/primary/source
to all four Cluster CRs (home pg-gitlab/pg-praefect, VPS
pg-gitlab/pg-praefect), turning last commit's recovery-only replicas
into real CNPG streaming replicas - exact pattern already proven for
pg-authentik. TLS certs copied cross-cluster manually (kubectl, not
git, same as every other cross-cluster secret in this plan).
Also adds pg-gitlab-app/pg-praefect-app password sync (PushSecret home
-> Vault -> ExternalSecret VPS, Merge policy) - CNPG-generated
passwords stay in sync via WAL replication but the K8s Secret OBJECTS
never resync on their own, same gap already closed for pg-authentik-app.
VPS's standalone clusters were already destroyed and recreated as
recovery-mode replicas in the prior commit (with the user's explicit
help running the classifier-blocked kubectl delete) - this commit turns
on the actual streaming replication on top of that.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces VPS's independent standalone pg-gitlab (own initdb, own
root/Scooby test data - never meant to survive this, per the approved
plan) with a bootstrap.recovery replica of home's, matching the
pg-authentik precedent. New pg-praefect cluster on the VPS too (never
existed before - VPS ran Gitaly without Praefect).
Stage 1 of 2: recovery-only, no streaming replication config yet - that
needs this cluster's own CNPG-generated certs to exist first, which
only happens once CNPG actually creates the cluster. Stage 2 (next
commit) copies certs cross-cluster and adds externalClusters
connectionParameters + replica.self/primary/source to all four Cluster
CRs (home pg-gitlab/pg-praefect, VPS pg-gitlab/pg-praefect).
postgresql.parameters copied verbatim from home's manifests (must
match-or-exceed, confirmed live for pg-authentik). vps-minio-secret
created manually on the VPS (kubectl, not git - same as every other
cross-cluster secret in this plan) pointing at the VPS's own local
MinIO, which now holds home's pg-gitlab/pg-praefect base backups
(triggered manually via Backup CRs since neither had run yet).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PushSecret (home) + ExternalSecret (VPS) pairs for the four
chart-generated secrets whose values must match before Phase 1c's
Postgres replication goes live: gitlab-rails-secret (secret_key_base/
db_key_base/otp_key_base/openid_connect_signing_key, all in one
secrets.yml key - confirmed live, not four separate keys as first
assumed), gitlab-gitlab-shell-secret, gitlab-gitaly-secret,
gitlab-praefect-secret (the latter two also needed for Phase 2's
cross-site gRPC auth). Same pattern as
infrastructure/authentik/manifests/ha-postgres-app-pushsecret.yaml.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per the approved plan (jiggly-snacking-iverson.md, 'GitLab True
Cross-Site Replication' section) - replacing the git-mirror CronJob
with real CNPG streaming replication for pg-gitlab and pg-praefect,
reusing the exact pattern already proven for pg-authentik.
This commit is network plumbing only, no DB replication yet:
- Repoint both CNPG clusters' barmanObjectStore backup target from
in-cluster gitlab-minio-svc to VPS MinIO (also becomes the initial
seed source for the VPS's replicas in Phase 1c).
- New ha-gitlab-postgres/ha-praefect-postgres NodePort Services
(bypass CNPG's own -rw Services) on both home and VPS.
- New PeerAuthentication with portLevelMtls PERMISSIVE on the replication
ports - ambient mesh STRICT default would otherwise reset the
VPS/witness's connection before the Postgres TLS handshake starts.
- Extended allow-gitlab-access AuthorizationPolicy with a port-scoped
rule for the same traffic (no source.namespaces rule can match
external, non-mesh peers).
- CoreDNS rewrites for pg-gitlab.ha.huskypup.net / pg-praefect.ha.huskypup.net
so home's own pods reach the floating hostname locally instead of
round-tripping through the WAN forward (no NAT hairpin support).
Still needed before Phase 1b/1c: UniFi WAN port-forwards for external
ports 61442/61443 -> talos-cp-01:32442/32443, source-restricted to the
VPS + witness public IPs (manual, same as the existing 61432 rule).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GitLab's native pull-mirror feature is Premium/Ultimate-gated on this
chart/version - confirmed live: the project API rejects 'mirror' and
'import_url' as unrecognized params on this CE instance entirely.
Plain git equivalent instead: CronJob every 15min does
'git clone --mirror' from home's Scooby/Homelabv4 (read-only deploy
token) then 'git push --mirror' to the VPS copy (write_repository
PAT, since deploy tokens can't push - not a valid scope for them
either). Matches the original plan's 'Gitea pull mirror of the GitOps
source' intent now that real GitLab replaced Gitea. Both tokens in
Vault at secret/vps/gitlab-mirror, pulled via ExternalSecret.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
3 consecutive attempts with the first loosening (~8min grace) all made
genuine progress (booted, started listening on queues) but never got
to open the health-check port before being killed - box's sustained
background load (gitea, restic, barman, k3s - unrelated to GitLab) is
apparently enough to starve that specific thread. More patience, not
a different fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RollingUpdate's default surge kept creating a second pod during every
rollout even with 1 replica - on this CPU-overcommitted single-node
VPS that meant 2 concurrent Ruby boots starving each other, never
converging (load avg hit 39+ on 6 cores). Recreate guarantees only 1
pod exists at a time. No availability cost given minReplicas=maxReplicas=1
already.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GITLAB_OMNIBUS_CONFIG (gitlab_rails['omniauth_providers'] = [...]) is
an omnibus-Docker-image-only convention. This chart's CNG webservice
image never processes it - confirmed via gitlab/charts/gitlab/
templates/_omniauth.tpl, which only reads
global.appConfig.omniauth.providers (a list of {secret,key} refs, each
pointing at a Secret key holding a whole YAML-encoded provider block
loaded via Ruby's YAML.load_file). So GITLAB_OMNIBUS_CONFIG was always
a silent no-op on both home and VPS - neither ever actually had SSO
configured despite gitlab-oidc-secret existing and looking correct.
Yesterday's fix for 'FailedMount: references non-existent secret key:
provider' removed the providers: list entirely instead of populating
that key - stopped the crash, but also silently deleted the only real
OIDC config path on both sites (no error, login page just lost its SSO
button). This restores providers: on both, and gitlab-oidc-secret's
ExternalSecret template now actually renders a provider key containing
a real YAML provider block (with the templated client_id/secret
substituted in), matching what _omniauth.tpl expects.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
VPS box is chronically CPU-overcommitted (load avg ~22-26 on 6 cores,
confirmed live - unrelated pre-existing services: barman, k3s itself,
node/gunicorn apps). Chart-default liveness probes (~200s grace) were
SIGTERM-killing both webservice and sidekiq mid-boot (still inside
bootsnap's require), every restart resetting progress to zero and
never actually finishing. Loosened to a ~7min grace window and dropped
webservice to 1 worker process instead of 2, less to fork/boot.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ingress pointed at 'gitlab-webservice-default' but the chart release
name on the VPS is 'vps-gitlab', so the actual Service Helm creates is
'vps-gitlab-webservice-default' (confirmed via 'kubectl get svc -n
gitlab'). Home's GitLab isn't affected - it routes via Istio
VirtualService, not this Ingress, and its release name has no prefix.
This was the actual cause of the persistent 404 - webservice pod was
healthy and passing readiness checks the whole time, Traefik just had
no matching backend to route to.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real bug affecting both home and VPS GitLab, found live diagnosing why
VPS's webservice pod was stuck at Init:0/3 for 74+ minutes (surfacing as
404s - nothing was ever actually serving):
global.appConfig.omniauth.providers: [{secret: gitlab-oidc-secret, key:
provider}] requires that secret to have a 'provider' key (a full YAML
omniauth provider config) - gitlab-oidc-secret never has one, only
GITLAB_OIDC_CLIENT_ID/SECRET, which get consumed via extraEnvFrom +
GITLAB_OMNIBUS_CONFIG's gitlab_rails['omniauth_providers'] instead (the
actual, complete mechanism already configuring OIDC - this providers:
list was always redundant). Removed it from both values.yaml files.
Home's GitLab has run fine so far because Kubernetes doesn't re-validate
already-mounted volumes when a referenced Secret's shape changes -
webservice/sidekiq there just haven't restarted since gitlab-oidc-secret
was last (if ever) in a shape with that key. VPS's identical values
shape hit it immediately on first pod creation. This was a live landmine
for home too - fixed before it could bite on a future restart.
Also: global.kas.enabled: false added to the VPS values (gitlab.kas.enabled
alone doesn't stop KAS pods - found live, separate toggle).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
User wants actual GitLab on the VPS, not just Gitea's pull-mirror -
deliberate scope increase from the original plan's 'keep the VPS
lighter' design for git hosting specifically.
Simplified single-node equivalent of apps/gitlab (home) - same chart
version (7.7.0), same Authentik OIDC provider/client_id (added a second
redirect_uri to the shared GitLab provider in infrastructure/authentik/
gitlab-blueprint.yaml), same known chart bugs already diagnosed at home
(redis-init-fix.yaml for the initializer-ordering bug) - but:
- 1 CNPG instance (local-path) instead of home's 3-instance HA
- No Praefect - pointless HA routing with a single Gitaly node
- No PgBouncer - direct CNPG connection, traffic is low enough
- No SAML, no Container Registry, no KAS - addable later if needed
- Own Traefik Ingress + https-redirect Middleware from the start
(found needed the hard way on Vault/Gitea/n8n/Nextcloud/Authentik
earlier this session - applying that lesson here immediately)
New Vault paths this needed: secret/vps/gitlab-redis (freshly generated,
this Redis instance is new, nothing to reuse) and widened the
vps-eso-reader policy to also allow secret/gitlab-oauth (the shared
Authentik client, already used by home's GitLab).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause of the Vault OIDC 'Missing auth_url' error (and the same
symptom on every other VPS-standby app): the Ingress tls: block makes
HTTPS available but Traefik still serves plain HTTP on the same host too
- nothing was forcing the redirect. A browser landing on
http://vault.vps.huskypup.net gets a Vault UI that computes its OIDC
callback using window.location.origin (http://...), which doesn't match
the https:// entry in allowed_redirect_uris - Vault silently returns an
empty auth_url rather than an obviously-config-looking error. Confirmed
via a HAR capture of the actual failing browser request.
Adds a per-namespace Traefik Middleware (redirectScheme -> https,
permanent) referenced via router.middlewares on each Ingress. Applied
directly via kubectl first to test - ArgoCD's selfHeal immediately
reverted it since it wasn't in git yet, confirming the fix needs to ship
through the normal pipeline rather than live kubectl edits on ArgoCD-
managed VPS resources.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two pre-existing gaps (not caused by this session's earlier work),
surfaced when the user started testing the VPS's standby Vault
(infrastructure/vps-standby/vault/) login via Authentik SSO:
1. VPS Vault's own auth/oidc/role/default allowed_redirect_uris only had
vault.kube.huskypup.net (home) and localhost:8250 - never had
vault.vps.huskypup.net registered, even though VPS's Vault shares the
same Authentik OAuth2 provider/client_id as home's. Fixed on both
sides: the shared blueprint (redirect_uris list) and VPS Vault's own
role config directly (separate Vault instance/backend, needed its own
write).
2. auth.kube.huskypup.net (Authentik itself) had NO public DNS record at
all - it only ever resolved via UniFi's local DNS for home-LAN
clients. Vault's OIDC backend runs server-side on the VPS and needs to
fetch Authentik's discovery document directly - it silently returned
an empty auth_url because it couldn't resolve the hostname at all
('missing auth_url' in Vault's UI, root cause only found by checking
DNS resolution from inside the VPS's own vault pod). Added a public
Cloudflare A record, same pattern as vault.kube.huskypup.net and
pg-authentik.ha.huskypup.net earlier - user confirmed before adding,
given it's Authentik itself. This likely also fixes OIDC for any other
VPS-hosted app doing SSO against home's Authentik, not just Vault.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Completes the 'make the repo proper' cleanup from earlier this session -
these were flagged but deliberately not touched in 185e9c2 given the
blast radius (live SSO for 8 apps). User confirmed: fix all 8 now.
infrastructure/authentik/{argocd,gitlab,nextcloud,grafana,n8n,guacamole,
rancher,vault}-blueprint.yaml: converted from plain ConfigMap (client_secret
hardcoded) to ExternalSecret with a templated blueprint body
(client_secret: "{{ .clientSecret }}") pulling from Vault. Chart already
supports mounting blueprints from Secrets (blueprints.secrets, alongside
blueprints.configMaps) - infrastructure/authentik/values.yaml updated to
route these 8 there instead.
For argocd/nextcloud/n8n/guacamole/rancher: Vault already had the matching
value at secret/<app>-oauth (the APP side was already Vault-backed via its
own ExternalSecret) - the blueprint was the only remaining plaintext copy.
For gitlab/grafana/vault: Vault had no copy at all yet - created
secret/{gitlab,grafana,vault}-oauth with the EXISTING live values (not
rotated - these are the actual working credentials right now, rotating
would break login until every consumer is updated in lockstep, which is
out of scope for a cleanup pass). Also fixed the OTHER plaintext copies
that existed for these three specifically:
- apps/gitlab/manifests/external-secret-oidc.yaml (new): replaces a
manually kubectl-created, never-git-tracked gitlab-oidc-secret.
- infrastructure/grafana/manifests/grafana-oauth-secret.yaml: was a
plain Secret whose own comment said 'hardcoded from blueprint'.
- infrastructure/vault/manifests/vault-init-{configmap,job}.yaml: this
one COULDN'T be converted to the same ExternalSecret-from-Vault
pattern - it's the PostSync Job that grants ESO's own Kubernetes-auth
role in Vault, so ESO can't yet authenticate to pull anything from
Vault at the point this script runs (genuinely circular). Sourced
from a new vault-oidc-bootstrap Secret instead - created once
manually (kubectl, not git, matching how Vault's own root/unseal
material is already handled), independent of the ESO pipeline.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
home's own authentik pods reach ha-authentik-postgres via the CoreDNS
rewrite on port 61432 (not just external traffic via NodePort on 5432/
61432) - discovered live that Istio ambient's port-level mTLS/L4
authorization enforcement is keyed on the port actually dialed (61432
here), not just the pod's real containerPort (5432) traffic eventually
reaches after Service translation. The existing port-5432-only rules
(from the streaming-replication fix) didn't cover this in-cluster path,
surfacing as 'server closed the connection unexpectedly' from home's own
authentik-worker pod.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause of the 'PostgreSQL connection failed, retrying... (connection
timeout expired)' loop that survived every earlier fix (streaming
replication, the password sync, the port fix, the CoreDNS override):
authentik's Helm values had TWO sources for AUTHENTIK_POSTGRESQL__HOST/
PORT - an explicit uppercase env: override (pointed at the floating
pg-authentik.ha.huskypup.net:61432) AND a chart-generated envFrom
secretRef (pg-authentik-app, prefix: AUTHENTIK_POSTGRESQL__) whose keys
are lowercase (host, port, ...), producing a SEPARATE
AUTHENTIK_POSTGRESQL__host/port pair pointing at the old local
pg-authentik-rw:5432. Kubernetes treats these as two unrelated env vars
(case-sensitive), but authentik's own generic AUTHENTIK_*-prefixed
env-var scanner apparently doesn't, and was resolving to the lowercase
(stale, local) values regardless of the explicit override - confirmed
live by dumping the pod's actual env: both HOST and host were present
with different values. Removed the now-fully-redundant envFrom entry
(every key it provided is already explicitly set via valueFrom).
Also: user correctly pointed out plaintext secrets don't belong in the
repo. Audited for the same class of issue as the already-fixed Cloudflare
token (infrastructure/cert-manager/manifests/secret-cf-token.yaml):
- infrastructure/external-dns/manifests/secret-external-dns-unifi.yaml
had a live UniFi API key in plaintext - moved to Vault+ExternalSecret,
same pattern as the Cloudflare token fix. The key itself is still the
original (now Vault-stored) value - rotating it requires the UniFi web
UI (no self-service API), noted in the file as a separate pending step.
- infrastructure/vps-eso/manifests/clustersecretstore.yaml had the
AppRole's roleId inline (added this session) - moved to roleRef,
sourced from the same Secret as secretId, consistent with 'nothing
sensitive in git' regardless of how sensitive one field is alone.
NOT touched, flagged separately for the user: infrastructure/authentik/
*-blueprint.yaml (vault, gitlab, n8n, nextcloud, rancher, grafana, argocd,
guacamole) all have live-looking high-entropy client_secret values
hardcoded in plaintext - same class of issue but much larger blast radius
(8 apps' SSO), needs its own coordinated rotation, not bundled into this
commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Follow-up to 8b8549e - pg-authentik-rw only listens on 5432, the app
tier needs 61432. See that commit for the full explanation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found via live test: the CoreDNS rewrite for pg-authentik.ha.huskypup.net
resolves the NAME correctly to a local service, but DNS rewriting can't
change the PORT a client then connects to - the app tier is configured
with AUTHENTIK_POSTGRESQL__PORT=61432 (matching the external floating
hostname), so the local target needs to actually listen on 61432 too, or
the connection times out even though DNS resolves fine. Also repoints the
rewrite target from CNPG's own pg-authentik-rw to this Service, so it
resolves to a port that actually exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Directly addresses the user's point: 3 existing DNS rewrite rules (auth/
rancher/netbird.kube.huskypup.net) were only ever applied via live kubectl
edits, never git-tracked - meaning every future change (like the
pg-authentik.ha.huskypup.net rewrite this session needed) required a
manual, classifier-gated live edit to cluster-wide DNS config instead of
a normal commit ArgoCD applies on its own.
No Talos/kubeadm ownerReferences on the ConfigMap or Deployment (checked
live), so it's safe for ArgoCD to take over. No pod restart needed after
sync either - the Corefile's own 'reload' plugin hot-reloads within
~30-60s of the ConfigMap volume updating on disk.
Adds the pg-authentik.ha.huskypup.net rewrite (see the failover-watcher
commits) as part of the same change, now flowing through git like
everything else instead of the standalone script handed off previously.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
User correctly flagged this: manually copying secrets between home and
the VPS defeats the entire point of having Vault. Every VPS secret this
session (MinIO creds, Cloudflare token, CNPG certs, the pg-authentik-app
password) was a one-off kubectl create secret copy-paste, because the
VPS's k3s cluster had no Vault/ESO pipeline at all - this builds one.
- infrastructure/authentik/manifests/ha-postgres-app-pushsecret.yaml:
home pushes pg-authentik-app's password into Vault at
secret/vps/pg-authentik-app (PushSecret, not just Get - this password is
CNPG-generated, not Vault-native, so it has to originate from a push).
- infrastructure/vps-eso/manifests/clustersecretstore.yaml: ESO on the
VPS (installed via helm, out-of-band like k3s/cert-manager - see the
file's own header) authenticates to home's Vault via AppRole (not
Kubernetes auth - the VPS is a separate cluster with no federation to
home's API server). Reachable via a new public
https://vault.kube.huskypup.net record - Vault was deliberately kept
off the public internet before this, explicitly confirmed with the user
before opening it. Traffic goes through the same Istio ingress gateway
already serving other public hosts, so unlike the ha-authentik-postgres
NodePort case, no PeerAuthentication/AuthorizationPolicy change was
needed - it arrives as a normal in-mesh call from the gateway's own
identity, not raw external TCP to a pod.
Found and fixed a real Vault gotcha while wiring this up: tried to
remove the AppRole's token_bound_cidrs restriction (added first, before
discovering Vault can't see the VPS's real source IP through the
gateway - it only ever sees the gateway's own pod IP) by omitting the
field from a follow-up - that does NOT clear it, the
AppRole role endpoint preserves omitted fields rather than resetting
them to default. Had to explicitly write token_bound_cidrs=. Spent a
while chasing a misleading 403 permission denied on auth/token/
lookup-self before finding this - vault token capabilities said read
was allowed (policy was fine), the actual rejection was IP-bound token
use from an unbound context.
- infrastructure/vps-standby/authentik/manifests/
pg-authentik-app-externalsecret.yaml: pulls it back down, Merge policy
(only overwrites the password key - host/dbname/username stay local,
CNPG still needs its own local -rw hostname for internal use).
- argocd-apps/vps-standby/vps-eso.yaml: new Application for the
ClusterSecretStore.
Verified end-to-end: password now matches between home and VPS's
pg-authentik-app secrets via this pipeline (not the earlier manual patch),
confirmed by comparing both live secret values after ESO's sync.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Builds the one genuinely new component from the plan - everything else
reuses CNPG/etcd as-is.
- infrastructure/ha-failover/manifests/heartbeat-writer.yaml: home writes
a fresh timestamp to etcd (/ha-failover/home-heartbeat) every 10s via
its local etcd member. No custom quorum/voting logic - etcd's own Raft
consensus (majority write/linearizable read) already gives the 'a
majority agrees' guarantee.
- scripts/ha-failover-watcher.sh + .service: runs on the VPS as a
systemd service (not k3s - must survive the VPS's own cluster being
unhealthy). Reads the heartbeat via its own local etcd member; if the
read itself fails, we can't tell if home is down or if this VPS is the
one partitioned, so it does NOT act (etcd's consensus requirement
provides the split-brain safety here, not custom code). If the read
succeeds and the heartbeat is stale (>45s, ~4-5 missed beats), home is
confirmed down by majority - promotes pg-authentik's Cluster CR
(spec.replica self/primary/source -> vps, no promotionToken - confirmed
live via dry-run that CNPG's webhook doesn't require one, it's only for
planned/graceful switchover LSN cross-checks that aren't available
during a genuine unplanned outage) and flips the
pg-authentik.ha.huskypup.net Cloudflare A record to the VPS's IP.
- scripts/ha-failback-authentik.sh: the deliberately manual reverse -
human confirms home is healthy and caught up before running this.
- Floating hostname plumbing: both sites' Authentik values now read
AUTHENTIK_POSTGRESQL__HOST/PORT as pg-authentik.ha.huskypup.net:61432
instead of the local pg-authentik-app secret's host/port, so app pods
on either site always reach whichever site is actually primary. Both
sites listen on the same external port (VPS gets a new
pg-authentik-forward.service socat forward, mirroring the existing
minio-forward.service pattern, so its NodePort 32433 is externally
reachable on 61432 same as home's UniFi-forwarded port).
- Found and worked around two real bugs surfaced while wiring this up
(fixed manually via scripts handed to the user - both blocked by the
Claude Code auto-mode classifier as credential-transmission /
cluster-DNS-edit actions):
- VPS's pg-authentik-app secret had a STALE password from before
streaming replication existed - the live Postgres role password now
replicates from home via WAL, but VPS's local K8s secret never got
updated to match. Needs a one-time sync (and again on any future
rotation).
- This UniFi does not support NAT hairpin/loopback for its own WAN
port-forwards - home's own pods resolving the floating hostname need
a local CoreDNS rewrite straight to pg-authentik-rw, confirmed live by
a DNS-resolves-but-TCP-connect-fails test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CNPG's new externalClusters connectionParameters were configured correctly
but streaming replication was silently failing - pg_stat_wal_receiver on
the VPS replica showed 0 rows, logs repeated 'could not connect to the
primary server: ... server closed the connection unexpectedly' every few
minutes.
Root cause: the authentik namespace is enrolled in Istio ambient mesh with
the mesh-wide default PeerAuthentication set to STRICT, and its
AuthorizationPolicy only allows traffic from specific in-mesh namespaces.
Traffic arriving via the ha-authentik-postgres NodePort from the VPS/
witness has no mesh identity at all (they're not in this cluster), so
ztunnel accepted the TCP connection then reset it once no HBONE/mTLS
handshake and no matching ALLOW rule ever arrived - confirmed live via
openssl s_client -starttls postgres (TCP connects, 0 bytes back).
Same root cause and same fix as the existing hostNetwork/webhook precedent
(infrastructure/istio/manifests/mesh/peer-authentication-webhooks.yaml):
- New port-scoped PERMISSIVE PeerAuthentication for the pg-authentik pods'
port 5432 only (not the whole namespace - Authentik's own in-mesh
east-west traffic stays STRICT).
- New port-scoped ALLOW rule on the existing AuthorizationPolicy, so any
source is allowed for port 5432 specifically, without touching the
existing namespace-based rules.
Both layers were needed - PERMISSIVE mTLS alone isn't enough, the
AuthorizationPolicy independently denies anything not matching one of its
existing rules.
Verified live: restarted the VPS replica pod to force an immediate
reconnect attempt: FATAL connection-reset errors stopped, and it's now
progressing through WAL restore toward a live streaming connection.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed live: 'spec.replica.source: Required value' even though home
isn't actually replicating from anyone (self == primary). Set to its own
self-referencing entry name.
Confirmed live against the vcluster.cnpg.io admission webhook: (1)
replica.enabled is mutually exclusive with primary/self - they're two
different mechanisms (old single-source replica-cluster mode vs the
newer distributed-topology model), not composable. (2) Both replica.self
and replica.primary must reference names present in THAT cluster's own
externalClusters list - including a self-referencing entry, even though
it describes the cluster itself. Renamed VPS's 'home-backup' external
cluster to 'home' throughout (bootstrap.recovery.source too) so one name
consistently satisfies barmanObjectStore, connectionParameters, and the
primary reference; added matching self-referencing entries on both
sides.
Adds connectionParameters (real streaming, not just barmanObjectStore
WAL-archive polling) to both sides' existing externalClusters entry, plus
replica.self/primary fields declaring the distributed topology (both
currently agree home is primary). Auth reuses CNPG's own auto-generated
streaming_replica client certs, cross-copied between clusters manually
(kubectl, not git - same pattern as every other cross-cluster secret
tonight) since they're the exact credential each side's pg_hba.conf
already trusts.
Also adds the VPS-side NodePort exposing its own pg-authentik primary
back to home (home->VPS already works via the existing netbird-egress
route, no UniFi port-forward needed for that direction - only the
reverse, VPS/witness reaching into home, needed the WAN workaround).
The chosen ports (12379/12380) actually fell inside the existing
GameServer port-forward's range (4000-50000) on the UniFi router,
confirmed live when creating the rule ('Port 12379 conflicts with port
4000-50000 used by GameServer'). The actual rules ended up on
61379/61380/61432 (safely above 50000) - this just brings the etcd
manifest in line with what's really forwarded.
UniFi's router has no route to 100.108.0.0/16 (the Netbird mesh CIDR) -
it only routes to its own directly-connected LAN. Advertising
talos-cp-01's real Netbird-bound IP (100.108.42.109, confirmed live via
talosctl to be genuinely bound to its wt0 interface - it's real, just
unreachable from outside the mesh) as the etcd client/peer URL would
never actually work for inbound WAN traffic from the VPS/witness.
Corrected to advertise home.kube.huskypup.net on the WAN ports that will
be port-forwarded (12379/12380) to talos-cp-01's real LAN IP
(172.28.101.41) + the NodePort Service - UniFi can route to its own LAN
natively, no extra static routes needed.
Part of the Authentik HA pilot (see plan doc). Home's etcd quorum member
(StatefulSet, pinned to talos-cp-01 for a stable advertised address) plus
a NodePort exposing pg-authentik's current primary - both reachable from
the VPS/witness over the already-authenticated Netbird mesh (confirmed
live: home nodes are directly reachable from Netbird peers on their real
node IP, via the netbird-egress DaemonSet's route). Deliberately NOT
going through UniFi/public-internet exposure - this stays entirely
inside the private mesh, a materially safer path than the WAN port-
forward originally considered.
Needs a scoped Kyverno PolicyException (ha-failover-nodeport-exception.yaml)
since disallow-nodeport-services is enforced cluster-wide - narrowly
scoped to Services named ha-*, matching the existing netbird-egress-
exemption.yaml precedent for exceptions.
Confirmed live: CrashLoopBackOff (Bad Gateway from the ingress) caused
by the default 40s-runway liveness probe tripping on a transient DB
connection blip. The worker constantly retries a scheduled-task query
needing a write lock against the read-only CNPG replica (expected,
harmless - 'cannot execute SELECT FOR UPDATE in a read-only
transaction') which appears to add enough connection pressure that a
brief hiccup trips the server's stricter default probe.
The n8n chart (riatlas/chart__n8n) doesn't support startupProbe at all -
my earlier fix set a field the chart ignores, so the liveness probe was
still killing the container ~40s into startup (exitCode 143, confirmed
live via repeated crashloops even after the first 'fix'). n8n takes
longer than that to bind :5678 on the VPS's more modest hardware.
Widened readiness/liveness directly instead: ~190s total runway before
a liveness kill.
Nextcloud: PHP's postgres driver tries to look up a client cert at
$HOME/.postgresql/postgresql.crt (HOME=/root in this image) for
verify-ca/verify-full sslmodes - 'Permission denied' there aborts the
connection entirely with a confusing 'password authentication failed'
secondary error. Confirmed live: psql with the identical credentials
connects fine (uses sslmode=prefer, no cert lookup). PGSSLMODE=disable
sidesteps it - this is an intra-cluster connection, not worth TLS here.
n8n: chart's default liveness/readiness probe timing is tighter than
n8n needs to actually finish starting - kubelet was killing the
container (exitCode 143/SIGTERM) before it ever bound :5678, in an
endless crashloop. Copied home's more generous probe timing.
n8n's chart uses persistence.type: existing, which expects this PVC to
already exist - nothing ever created it since n8n started at replicas: 0
and is only now being scaled up to 1 (confirmed live: pod stuck Pending,
'persistentvolumeclaim n8n-main-persistence not found').
The DNS-flip watcher (scripts/vps-dns-failover/) was designed but never
actually installed on the VPS despite being tracked as done - real gap,
found when asked whether the standby services are actually reachable.
New design: instead of dynamically flipping *.kube.huskypup.net between
home and VPS IPs, give the VPS site its own permanent, always-resolving
subdomain - vault/gitea/auth/n8n/nextcloud.vps.huskypup.net, each with
real Ingress+TLS on the VPS's own Traefik+cert-manager (both already
installed by Phase 0 bootstrap, just never wired up). No token-scoping
decision needed since there's no dynamic flipping - reuses the same
cert-manager token pattern as home.
Also scales Authentik/n8n/Nextcloud from 0 to 1 replica on the VPS so
the replicated data is actually browsable at all times, not just
present-but-unreachable. Their CNPG clusters are still read-only
replicas (spec.replica.enabled: true) - writes will error until a
deliberate manual promotion, but reads/browsing work now. Vault and
Gitea were already running continuously.
Vault's OIDC auth method needs to resolve auth.kube.huskypup.net (the
issuer URL) from inside its own pod - no in-cluster DNS entry exists for
that public hostname, so add a hostAlias pointing at the live
istio-ingressgateway LoadBalancer IP (verified current: 172.28.101.244).
authentik-blueprints-vault (vault-blueprint.yaml) was already applied to
the cluster and referenced by the authentik Deployment for some time -
committing it now so git matches the live, working state instead of
leaving it as an untracked local file.
Postgres refuses to replay WAL past a checkpoint recorded with higher
max_connections/max_wal_senders/etc than the recovering instance's own
settings: 'recovery aborted because of insufficient parameter settings:
max_connections = 100 is a lower setting than on the primary server,
where its value was 200' - confirmed live on all 3 VPS clusters (they
had no postgresql.parameters block at all, defaulting to CNPG's 100).
Copied home's full postgresql.parameters block to remove any other
potential mismatch (max_wal_senders, max_worker_processes are subject
to the same check). Bumped memory requests/limits to match home too -
shared_buffers: 512MB needs headroom the previous 256Mi/1Gi didn't have.
CNPG's barmanObjectStore.serverName defaults to the externalClusters[]
entry's own .name field ('home-backup', an arbitrary label I chose) -
NOT the actual source Postgres cluster's name. Home's backup: block
(on the real pg-authentik/pg-n8n/pg-nextcloud Clusters) defaults
serverName to its own metadata.name instead. Without an explicit
override these two disagree, so recovery searches the object store
under the wrong server-name prefix and finds nothing - 'no target
backup found', confirmed live even after the ScheduledBackup fix
produced real, completed base backups.
Found while building the VPS replica clusters (Phase 2): all three
CNPG clusters' backup.barmanObjectStore blocks only configure continuous
WAL archiving. CNPG's bootstrap.recovery needs at least one real Backup
object in the object store before it has anything to restore -
'no target backup found', confirmed live against pg-authentik. This gap
was systemic (gitlab has it too, not fixed here - out of scope for
tonight, no VPS standby depends on it).
immediate: true fires one backup right away in addition to the daily
01:00 schedule, to unblock the in-progress VPS restore now rather than
waiting up to 24h for the first scheduled run.
Without hook annotations they land in ArgoCD's regular Sync phase, which
runs AFTER PreSync hooks - so the presync Job's pod couldn't be created
('serviceaccount not found', confirmed live). Weight -1 vs the Job's 0
gets them created first, within the same PreSync phase.
Same CNPG replica-cluster pattern as Authentik (continuous WAL replay
from home via VPS MinIO, app at 0 replicas until manual promotion - see
infrastructure/vps-standby/authentik/manifests/cnpg-cluster.yaml for the
full rationale).
n8n: N8N_ENCRYPTION_KEY copied byte-identical from home (kubectl, not
git) - decrypts stored credentials in the replicated DB, same reasoning
as Vault's unseal key / root token copies.
Nextcloud: adds infrastructure/vps-standby/nextcloud/manifests/
pvc-restore-cronjob.yaml, the read side of home's existing
nextcloud-pvc-sync restic backup - restores the latest snapshot from VPS
MinIO into this standby's PVC daily at 04:00 (2h after home's 02:00
backup). No Redis on the VPS side (no redis-operator deployed there,
out of scope for a standby that isn't serving traffic - Nextcloud
degrades gracefully to DB-based locking without it).
