User wants actual GitLab on the VPS, not just Gitea's pull-mirror -
deliberate scope increase from the original plan's 'keep the VPS
lighter' design for git hosting specifically.
Simplified single-node equivalent of apps/gitlab (home) - same chart
version (7.7.0), same Authentik OIDC provider/client_id (added a second
redirect_uri to the shared GitLab provider in infrastructure/authentik/
gitlab-blueprint.yaml), same known chart bugs already diagnosed at home
(redis-init-fix.yaml for the initializer-ordering bug) - but:
- 1 CNPG instance (local-path) instead of home's 3-instance HA
- No Praefect - pointless HA routing with a single Gitaly node
- No PgBouncer - direct CNPG connection, traffic is low enough
- No SAML, no Container Registry, no KAS - addable later if needed
- Own Traefik Ingress + https-redirect Middleware from the start
(found needed the hard way on Vault/Gitea/n8n/Nextcloud/Authentik
earlier this session - applying that lesson here immediately)
New Vault paths this needed: secret/vps/gitlab-redis (freshly generated,
this Redis instance is new, nothing to reuse) and widened the
vps-eso-reader policy to also allow secret/gitlab-oauth (the shared
Authentik client, already used by home's GitLab).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause of the Vault OIDC 'Missing auth_url' error (and the same
symptom on every other VPS-standby app): the Ingress tls: block makes
HTTPS available but Traefik still serves plain HTTP on the same host too
- nothing was forcing the redirect. A browser landing on
http://vault.vps.huskypup.net gets a Vault UI that computes its OIDC
callback using window.location.origin (http://...), which doesn't match
the https:// entry in allowed_redirect_uris - Vault silently returns an
empty auth_url rather than an obviously-config-looking error. Confirmed
via a HAR capture of the actual failing browser request.
Adds a per-namespace Traefik Middleware (redirectScheme -> https,
permanent) referenced via router.middlewares on each Ingress. Applied
directly via kubectl first to test - ArgoCD's selfHeal immediately
reverted it since it wasn't in git yet, confirming the fix needs to ship
through the normal pipeline rather than live kubectl edits on ArgoCD-
managed VPS resources.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause of the 'PostgreSQL connection failed, retrying... (connection
timeout expired)' loop that survived every earlier fix (streaming
replication, the password sync, the port fix, the CoreDNS override):
authentik's Helm values had TWO sources for AUTHENTIK_POSTGRESQL__HOST/
PORT - an explicit uppercase env: override (pointed at the floating
pg-authentik.ha.huskypup.net:61432) AND a chart-generated envFrom
secretRef (pg-authentik-app, prefix: AUTHENTIK_POSTGRESQL__) whose keys
are lowercase (host, port, ...), producing a SEPARATE
AUTHENTIK_POSTGRESQL__host/port pair pointing at the old local
pg-authentik-rw:5432. Kubernetes treats these as two unrelated env vars
(case-sensitive), but authentik's own generic AUTHENTIK_*-prefixed
env-var scanner apparently doesn't, and was resolving to the lowercase
(stale, local) values regardless of the explicit override - confirmed
live by dumping the pod's actual env: both HOST and host were present
with different values. Removed the now-fully-redundant envFrom entry
(every key it provided is already explicitly set via valueFrom).
Also: user correctly pointed out plaintext secrets don't belong in the
repo. Audited for the same class of issue as the already-fixed Cloudflare
token (infrastructure/cert-manager/manifests/secret-cf-token.yaml):
- infrastructure/external-dns/manifests/secret-external-dns-unifi.yaml
had a live UniFi API key in plaintext - moved to Vault+ExternalSecret,
same pattern as the Cloudflare token fix. The key itself is still the
original (now Vault-stored) value - rotating it requires the UniFi web
UI (no self-service API), noted in the file as a separate pending step.
- infrastructure/vps-eso/manifests/clustersecretstore.yaml had the
AppRole's roleId inline (added this session) - moved to roleRef,
sourced from the same Secret as secretId, consistent with 'nothing
sensitive in git' regardless of how sensitive one field is alone.
NOT touched, flagged separately for the user: infrastructure/authentik/
*-blueprint.yaml (vault, gitlab, n8n, nextcloud, rancher, grafana, argocd,
guacamole) all have live-looking high-entropy client_secret values
hardcoded in plaintext - same class of issue but much larger blast radius
(8 apps' SSO), needs its own coordinated rotation, not bundled into this
commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
User correctly flagged this: manually copying secrets between home and
the VPS defeats the entire point of having Vault. Every VPS secret this
session (MinIO creds, Cloudflare token, CNPG certs, the pg-authentik-app
password) was a one-off kubectl create secret copy-paste, because the
VPS's k3s cluster had no Vault/ESO pipeline at all - this builds one.
- infrastructure/authentik/manifests/ha-postgres-app-pushsecret.yaml:
home pushes pg-authentik-app's password into Vault at
secret/vps/pg-authentik-app (PushSecret, not just Get - this password is
CNPG-generated, not Vault-native, so it has to originate from a push).
- infrastructure/vps-eso/manifests/clustersecretstore.yaml: ESO on the
VPS (installed via helm, out-of-band like k3s/cert-manager - see the
file's own header) authenticates to home's Vault via AppRole (not
Kubernetes auth - the VPS is a separate cluster with no federation to
home's API server). Reachable via a new public
https://vault.kube.huskypup.net record - Vault was deliberately kept
off the public internet before this, explicitly confirmed with the user
before opening it. Traffic goes through the same Istio ingress gateway
already serving other public hosts, so unlike the ha-authentik-postgres
NodePort case, no PeerAuthentication/AuthorizationPolicy change was
needed - it arrives as a normal in-mesh call from the gateway's own
identity, not raw external TCP to a pod.
Found and fixed a real Vault gotcha while wiring this up: tried to
remove the AppRole's token_bound_cidrs restriction (added first, before
discovering Vault can't see the VPS's real source IP through the
gateway - it only ever sees the gateway's own pod IP) by omitting the
field from a follow-up - that does NOT clear it, the
AppRole role endpoint preserves omitted fields rather than resetting
them to default. Had to explicitly write token_bound_cidrs=. Spent a
while chasing a misleading 403 permission denied on auth/token/
lookup-self before finding this - vault token capabilities said read
was allowed (policy was fine), the actual rejection was IP-bound token
use from an unbound context.
- infrastructure/vps-standby/authentik/manifests/
pg-authentik-app-externalsecret.yaml: pulls it back down, Merge policy
(only overwrites the password key - host/dbname/username stay local,
CNPG still needs its own local -rw hostname for internal use).
- argocd-apps/vps-standby/vps-eso.yaml: new Application for the
ClusterSecretStore.
Verified end-to-end: password now matches between home and VPS's
pg-authentik-app secrets via this pipeline (not the earlier manual patch),
confirmed by comparing both live secret values after ESO's sync.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Builds the one genuinely new component from the plan - everything else
reuses CNPG/etcd as-is.
- infrastructure/ha-failover/manifests/heartbeat-writer.yaml: home writes
a fresh timestamp to etcd (/ha-failover/home-heartbeat) every 10s via
its local etcd member. No custom quorum/voting logic - etcd's own Raft
consensus (majority write/linearizable read) already gives the 'a
majority agrees' guarantee.
- scripts/ha-failover-watcher.sh + .service: runs on the VPS as a
systemd service (not k3s - must survive the VPS's own cluster being
unhealthy). Reads the heartbeat via its own local etcd member; if the
read itself fails, we can't tell if home is down or if this VPS is the
one partitioned, so it does NOT act (etcd's consensus requirement
provides the split-brain safety here, not custom code). If the read
succeeds and the heartbeat is stale (>45s, ~4-5 missed beats), home is
confirmed down by majority - promotes pg-authentik's Cluster CR
(spec.replica self/primary/source -> vps, no promotionToken - confirmed
live via dry-run that CNPG's webhook doesn't require one, it's only for
planned/graceful switchover LSN cross-checks that aren't available
during a genuine unplanned outage) and flips the
pg-authentik.ha.huskypup.net Cloudflare A record to the VPS's IP.
- scripts/ha-failback-authentik.sh: the deliberately manual reverse -
human confirms home is healthy and caught up before running this.
- Floating hostname plumbing: both sites' Authentik values now read
AUTHENTIK_POSTGRESQL__HOST/PORT as pg-authentik.ha.huskypup.net:61432
instead of the local pg-authentik-app secret's host/port, so app pods
on either site always reach whichever site is actually primary. Both
sites listen on the same external port (VPS gets a new
pg-authentik-forward.service socat forward, mirroring the existing
minio-forward.service pattern, so its NodePort 32433 is externally
reachable on 61432 same as home's UniFi-forwarded port).
- Found and worked around two real bugs surfaced while wiring this up
(fixed manually via scripts handed to the user - both blocked by the
Claude Code auto-mode classifier as credential-transmission /
cluster-DNS-edit actions):
- VPS's pg-authentik-app secret had a STALE password from before
streaming replication existed - the live Postgres role password now
replicates from home via WAL, but VPS's local K8s secret never got
updated to match. Needs a one-time sync (and again on any future
rotation).
- This UniFi does not support NAT hairpin/loopback for its own WAN
port-forwards - home's own pods resolving the floating hostname need
a local CoreDNS rewrite straight to pg-authentik-rw, confirmed live by
a DNS-resolves-but-TCP-connect-fails test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed live against the vcluster.cnpg.io admission webhook: (1)
replica.enabled is mutually exclusive with primary/self - they're two
different mechanisms (old single-source replica-cluster mode vs the
newer distributed-topology model), not composable. (2) Both replica.self
and replica.primary must reference names present in THAT cluster's own
externalClusters list - including a self-referencing entry, even though
it describes the cluster itself. Renamed VPS's 'home-backup' external
cluster to 'home' throughout (bootstrap.recovery.source too) so one name
consistently satisfies barmanObjectStore, connectionParameters, and the
primary reference; added matching self-referencing entries on both
sides.
Adds connectionParameters (real streaming, not just barmanObjectStore
WAL-archive polling) to both sides' existing externalClusters entry, plus
replica.self/primary fields declaring the distributed topology (both
currently agree home is primary). Auth reuses CNPG's own auto-generated
streaming_replica client certs, cross-copied between clusters manually
(kubectl, not git - same pattern as every other cross-cluster secret
tonight) since they're the exact credential each side's pg_hba.conf
already trusts.
Also adds the VPS-side NodePort exposing its own pg-authentik primary
back to home (home->VPS already works via the existing netbird-egress
route, no UniFi port-forward needed for that direction - only the
reverse, VPS/witness reaching into home, needed the WAN workaround).
Confirmed live: CrashLoopBackOff (Bad Gateway from the ingress) caused
by the default 40s-runway liveness probe tripping on a transient DB
connection blip. The worker constantly retries a scheduled-task query
needing a write lock against the read-only CNPG replica (expected,
harmless - 'cannot execute SELECT FOR UPDATE in a read-only
transaction') which appears to add enough connection pressure that a
brief hiccup trips the server's stricter default probe.
The n8n chart (riatlas/chart__n8n) doesn't support startupProbe at all -
my earlier fix set a field the chart ignores, so the liveness probe was
still killing the container ~40s into startup (exitCode 143, confirmed
live via repeated crashloops even after the first 'fix'). n8n takes
longer than that to bind :5678 on the VPS's more modest hardware.
Widened readiness/liveness directly instead: ~190s total runway before
a liveness kill.
Nextcloud: PHP's postgres driver tries to look up a client cert at
$HOME/.postgresql/postgresql.crt (HOME=/root in this image) for
verify-ca/verify-full sslmodes - 'Permission denied' there aborts the
connection entirely with a confusing 'password authentication failed'
secondary error. Confirmed live: psql with the identical credentials
connects fine (uses sslmode=prefer, no cert lookup). PGSSLMODE=disable
sidesteps it - this is an intra-cluster connection, not worth TLS here.
n8n: chart's default liveness/readiness probe timing is tighter than
n8n needs to actually finish starting - kubelet was killing the
container (exitCode 143/SIGTERM) before it ever bound :5678, in an
endless crashloop. Copied home's more generous probe timing.
n8n's chart uses persistence.type: existing, which expects this PVC to
already exist - nothing ever created it since n8n started at replicas: 0
and is only now being scaled up to 1 (confirmed live: pod stuck Pending,
'persistentvolumeclaim n8n-main-persistence not found').
The DNS-flip watcher (scripts/vps-dns-failover/) was designed but never
actually installed on the VPS despite being tracked as done - real gap,
found when asked whether the standby services are actually reachable.
New design: instead of dynamically flipping *.kube.huskypup.net between
home and VPS IPs, give the VPS site its own permanent, always-resolving
subdomain - vault/gitea/auth/n8n/nextcloud.vps.huskypup.net, each with
real Ingress+TLS on the VPS's own Traefik+cert-manager (both already
installed by Phase 0 bootstrap, just never wired up). No token-scoping
decision needed since there's no dynamic flipping - reuses the same
cert-manager token pattern as home.
Also scales Authentik/n8n/Nextcloud from 0 to 1 replica on the VPS so
the replicated data is actually browsable at all times, not just
present-but-unreachable. Their CNPG clusters are still read-only
replicas (spec.replica.enabled: true) - writes will error until a
deliberate manual promotion, but reads/browsing work now. Vault and
Gitea were already running continuously.
Postgres refuses to replay WAL past a checkpoint recorded with higher
max_connections/max_wal_senders/etc than the recovering instance's own
settings: 'recovery aborted because of insufficient parameter settings:
max_connections = 100 is a lower setting than on the primary server,
where its value was 200' - confirmed live on all 3 VPS clusters (they
had no postgresql.parameters block at all, defaulting to CNPG's 100).
Copied home's full postgresql.parameters block to remove any other
potential mismatch (max_wal_senders, max_worker_processes are subject
to the same check). Bumped memory requests/limits to match home too -
shared_buffers: 512MB needs headroom the previous 256Mi/1Gi didn't have.
CNPG's barmanObjectStore.serverName defaults to the externalClusters[]
entry's own .name field ('home-backup', an arbitrary label I chose) -
NOT the actual source Postgres cluster's name. Home's backup: block
(on the real pg-authentik/pg-n8n/pg-nextcloud Clusters) defaults
serverName to its own metadata.name instead. Without an explicit
override these two disagree, so recovery searches the object store
under the wrong server-name prefix and finds nothing - 'no target
backup found', confirmed live even after the ScheduledBackup fix
produced real, completed base backups.
Without hook annotations they land in ArgoCD's regular Sync phase, which
runs AFTER PreSync hooks - so the presync Job's pod couldn't be created
('serviceaccount not found', confirmed live). Weight -1 vs the Job's 0
gets them created first, within the same PreSync phase.
Same CNPG replica-cluster pattern as Authentik (continuous WAL replay
from home via VPS MinIO, app at 0 replicas until manual promotion - see
infrastructure/vps-standby/authentik/manifests/cnpg-cluster.yaml for the
full rationale).
n8n: N8N_ENCRYPTION_KEY copied byte-identical from home (kubectl, not
git) - decrypts stored credentials in the replicated DB, same reasoning
as Vault's unseal key / root token copies.
Nextcloud: adds infrastructure/vps-standby/nextcloud/manifests/
pvc-restore-cronjob.yaml, the read side of home's existing
nextcloud-pvc-sync restic backup - restores the latest snapshot from VPS
MinIO into this standby's PVC daily at 04:00 (2h after home's 02:00
backup). No Redis on the VPS side (no redis-operator deployed there,
out of scope for a standby that isn't serving traffic - Nextcloud
degrades gracefully to DB-based locking without it).
CNPG operator on the VPS (argocd-apps/vps-standby/cnpg-operator.yaml),
plus a CNPG 'replica cluster' for pg-authentik (infrastructure/
vps-standby/authentik/manifests/cnpg-cluster.yaml) that continuously
replays WAL shipped from home's pg-authentik via VPS MinIO's
cnpg-backups/pg-authentik bucket path - CNPG's native replica-cluster
mode, not a custom restore script like Vault needed (Vault has no
equivalent built-in continuous-replication-into-object-store feature).
The Authentik app itself runs at replicas: 0 in normal operation - the
replica DB is read-only until a deliberate manual promotion
(spec.replica.enabled: false), and a pod trying to write against a
read-only DB would just crashloop uselessly. values.yaml's authentik:
block is copied byte-identical from home's on purpose: the chart
deterministically derives its generated Secret's AUTHENTIK_SECRET_KEY
from these literal values.yaml strings, so both clusters land on the
same key without manually copying it - required since that key decrypts
things stored encrypted in the replicated DB.
service=gitlab invokes Gitea's GitLab-API-based downloader (for issues/
PRs/releases metadata), which calls the source's /api/v4/projects/...
REST API rather than doing a plain git clone. That was 404ing against
home GitLab and getting swallowed into a generic 'InternalServerError:
404 Not Found' with no further detail. We only want a plain pull-mirror
of the git repo itself, so service=git forces the plain git-clone
downloader instead.
Pull-mirrors home GitLab's Homelabv4 repo on Gitea's own built-in mirror
scheduler (6h interval) - no custom sync job needed, per the original
plan. SQLite instead of the chart's default HA Postgres + Valkey cluster
(single-instance standby holding one small repo, not worth the extra
moving parts). A PostSync Job creates the mirror once, idempotently;
Gitea's scheduler handles all ongoing pulls after that.
Also added a public Cloudflare CNAME for gitlab.kube.huskypup.net ->
home.kube.huskypup.net: the VPS has no route to home's LAN via Netbird
(none of the mesh peers advertise that subnet, confirmed live), so
GitLab needs to be reachable the same way any other internet client
reaches it - home's public IP already has port 443 forwarded to
istio-ingressgateway from earlier this session.
Admin and GitLab-mirror credentials are plain Secrets created directly
on the VPS cluster (kubectl, not git) - same pattern as vault-unseal-key.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed end-to-end tonight: condition=Ready correctly times out every
cycle since Vault can't be Ready while sealed (the unseal step comes
right after this wait) - harmless via the existing || true fallback, but
wastes up to 2 minutes per restore cycle waiting on a condition that can
never be met at this point. Poll for the container process merely being
started instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Without it, kubectl wait's internal watch retry loop never terminates
cleanly even with --timeout set - confirmed live, it kept retrying well
past the stated 120s timeout. The restore/unseal steps themselves still
succeed regardless (verified end-to-end with a manual unseal), this only
affects the job's own graceful detection of the pod coming back.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
raft snapshot restore is a privileged operation - unsealing alone isn't
enough, it needs an authenticated token. Missed this on first pass, caught
live: 'Code: 403. Errors: * permission denied'.
First run: uses the throwaway init's own fresh root token. Steady state
(already restored at least once): uses a copy of home's real root token,
which becomes valid on this Vault the moment the first restore completes
(its auth data becomes byte-for-byte home's as of that snapshot). Stored
the same way as the unseal key - kubectl directly on the VPS, not git.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
StatefulSet pod naming follows the Helm release name (vps-vault, matching
the ArgoCD Application name), not the chart's default.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Single-node Vault (raft storage, standalone mode) plus a restore CronJob
that pulls the latest raft snapshot from the VPS's own local MinIO
(already receiving snapshots every 6h from home) and restores it every
6h, offset 30 minutes after the home-side snapshot job.
Not an independent root of trust: raft snapshot restore replaces the
entire storage backend including the keyring, so after every restore
this Vault is sealed with HOME's actual keyring - unsealing it needs a
copy of home's real unseal key, stored directly on the VPS cluster
(kubectl, not git, same pattern as vps-minio-root-secret). First-run
bootstrap uses a throwaway single-Shamir-key init just to get through
the very first restore, then is irrelevant from then on.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
rootUser/rootPassword were left unset, relying on the chart's
auto-generation - but ArgoCD renders Helm via 'helm template', which
doesn't support the lookup() function charts use to preserve an existing
generated secret across upgrades. With selfHeal: true, every single sync
generated a BRAND NEW random root password, immediately invalidating
whatever credentials were stored in the home cluster's Vault for the
backup/CronJob consumers (vault-raft-snapshot, 3x CNPG barmanObjectStore,
nextcloud-pvc-sync) - confirmed live: all of them failing with 'Access
Key Id does not exist' right after being fixed, because the password had
already rotated out from under them again.
Fixed by pointing at a pre-existing Secret created directly on the VPS
cluster (kubectl, not git - this cluster has no Vault/ESO of its own).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Foundation for a DR/backup path using an always-on VPS as a second
ArgoCD-managed cluster, plus DB/backup standardization work that fell
out of it:
- vps-standby ArgoCD cluster destination + AppProject, MinIO backup
receiver, VPS bootstrap script (k3s, Netbird, cert-manager)
- Dual-site DNS failover watcher + home-IP DDNS CronJob, Cloudflare
token moved out of git into Vault+ExternalSecret
- Nextcloud migrated from ad-hoc MariaDB to CNPG + redis-operator
(matches n8n/Authentik/GitLab's backup-native pattern)
- Authentik's CNPG manifests moved into the actual ArgoCD-synced
manifests/ path (were present but never wired into the sync path)
- Vault raft-snapshot CronJob, CNPG barmanObjectStore backups
(Authentik/n8n/Nextcloud), Nextcloud file-PVC restic sync - all
targeting the new VPS MinIO receiver
See VPS Warm-Standby plan doc for full design rationale.
