- infrastructure/vps-standby/gitlab/manifests/ha-postgres-app-externalsecret.yaml:
creationPolicy Merge -> Owner (already applied live, committing to
match). Merge assumed CNPG creates a base pg-gitlab-app/pg-praefect-app
secret to merge a password into - wrong for these VPS Clusters since
Phase 1c's replica-cluster recreate; CNPG doesn't create a local
owner secret for a replicated role. Left pg-gitlab-app/pg-praefect-app
missing entirely, crash-looping vps-gitlab-webservice on
ActiveRecord::DatabaseConnectionError for ~39h.
- infrastructure/vps-standby/nextcloud/manifests/pvc-restore-cronjob.yaml:
exclude config.php from the restic restore. The job's own comment
assumed nextcloud runs at replicas: 0 on the VPS; live confirms
replicas: 1 (no such key was ever actually set) - restic couldn't
overwrite the live pod's config.php (permission denied), failing the
whole job every run for 2+ days even though everything else restored
fine. Excluding it is correct regardless of permissions - a live
pod's own config.php shouldn't be overwritten by a background
restore job.
Also fixed live (not git-tracked, config.php is PVC-persisted runtime
state, not sourced from git):
- home + VPS nextcloud config.php dbpassword: out of sync with CNPG's
actual current pg-nextcloud-app password (baked in once at install,
never re-synced). VPS's case was two-layered - its own
pg-nextcloud-app secret also didn't match the real Postgres role
password, since VPS's pg-nextcloud is a read-only streaming replica
(spec.replica.enabled) and the authoritative password lives on home.
- grafana: broke a RollingUpdate deadlock (single-replica Deployment +
RWO Ceph volume - new pod couldn't start while the old pod still
held the only-one-node-at-a-time attachment, and the rollout
wouldn't scale down the old pod until the new one was ready).
- suspended the stale gitlab-mirror-sync CronJob on the VPS (spamming
auth failures every ~15min since its stored token predates the
Postgres-replication cutover of VPS GitLab's DB) rather than
deleting it, per the plan's own note to defer that until Phase 2b is
verified.
Adds externalClusters connectionParameters + replica.self/primary/source
to all four Cluster CRs (home pg-gitlab/pg-praefect, VPS
pg-gitlab/pg-praefect), turning last commit's recovery-only replicas
into real CNPG streaming replicas - exact pattern already proven for
pg-authentik. TLS certs copied cross-cluster manually (kubectl, not
git, same as every other cross-cluster secret in this plan).
Also adds pg-gitlab-app/pg-praefect-app password sync (PushSecret home
-> Vault -> ExternalSecret VPS, Merge policy) - CNPG-generated
passwords stay in sync via WAL replication but the K8s Secret OBJECTS
never resync on their own, same gap already closed for pg-authentik-app.
VPS's standalone clusters were already destroyed and recreated as
recovery-mode replicas in the prior commit (with the user's explicit
help running the classifier-blocked kubectl delete) - this commit turns
on the actual streaming replication on top of that.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces VPS's independent standalone pg-gitlab (own initdb, own
root/Scooby test data - never meant to survive this, per the approved
plan) with a bootstrap.recovery replica of home's, matching the
pg-authentik precedent. New pg-praefect cluster on the VPS too (never
existed before - VPS ran Gitaly without Praefect).
Stage 1 of 2: recovery-only, no streaming replication config yet - that
needs this cluster's own CNPG-generated certs to exist first, which
only happens once CNPG actually creates the cluster. Stage 2 (next
commit) copies certs cross-cluster and adds externalClusters
connectionParameters + replica.self/primary/source to all four Cluster
CRs (home pg-gitlab/pg-praefect, VPS pg-gitlab/pg-praefect).
postgresql.parameters copied verbatim from home's manifests (must
match-or-exceed, confirmed live for pg-authentik). vps-minio-secret
created manually on the VPS (kubectl, not git - same as every other
cross-cluster secret in this plan) pointing at the VPS's own local
MinIO, which now holds home's pg-gitlab/pg-praefect base backups
(triggered manually via Backup CRs since neither had run yet).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PushSecret (home) + ExternalSecret (VPS) pairs for the four
chart-generated secrets whose values must match before Phase 1c's
Postgres replication goes live: gitlab-rails-secret (secret_key_base/
db_key_base/otp_key_base/openid_connect_signing_key, all in one
secrets.yml key - confirmed live, not four separate keys as first
assumed), gitlab-gitlab-shell-secret, gitlab-gitaly-secret,
gitlab-praefect-secret (the latter two also needed for Phase 2's
cross-site gRPC auth). Same pattern as
infrastructure/authentik/manifests/ha-postgres-app-pushsecret.yaml.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per the approved plan (jiggly-snacking-iverson.md, 'GitLab True
Cross-Site Replication' section) - replacing the git-mirror CronJob
with real CNPG streaming replication for pg-gitlab and pg-praefect,
reusing the exact pattern already proven for pg-authentik.
This commit is network plumbing only, no DB replication yet:
- Repoint both CNPG clusters' barmanObjectStore backup target from
in-cluster gitlab-minio-svc to VPS MinIO (also becomes the initial
seed source for the VPS's replicas in Phase 1c).
- New ha-gitlab-postgres/ha-praefect-postgres NodePort Services
(bypass CNPG's own -rw Services) on both home and VPS.
- New PeerAuthentication with portLevelMtls PERMISSIVE on the replication
ports - ambient mesh STRICT default would otherwise reset the
VPS/witness's connection before the Postgres TLS handshake starts.
- Extended allow-gitlab-access AuthorizationPolicy with a port-scoped
rule for the same traffic (no source.namespaces rule can match
external, non-mesh peers).
- CoreDNS rewrites for pg-gitlab.ha.huskypup.net / pg-praefect.ha.huskypup.net
so home's own pods reach the floating hostname locally instead of
round-tripping through the WAN forward (no NAT hairpin support).
Still needed before Phase 1b/1c: UniFi WAN port-forwards for external
ports 61442/61443 -> talos-cp-01:32442/32443, source-restricted to the
VPS + witness public IPs (manual, same as the existing 61432 rule).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GitLab's native pull-mirror feature is Premium/Ultimate-gated on this
chart/version - confirmed live: the project API rejects 'mirror' and
'import_url' as unrecognized params on this CE instance entirely.
Plain git equivalent instead: CronJob every 15min does
'git clone --mirror' from home's Scooby/Homelabv4 (read-only deploy
token) then 'git push --mirror' to the VPS copy (write_repository
PAT, since deploy tokens can't push - not a valid scope for them
either). Matches the original plan's 'Gitea pull mirror of the GitOps
source' intent now that real GitLab replaced Gitea. Both tokens in
Vault at secret/vps/gitlab-mirror, pulled via ExternalSecret.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
3 consecutive attempts with the first loosening (~8min grace) all made
genuine progress (booted, started listening on queues) but never got
to open the health-check port before being killed - box's sustained
background load (gitea, restic, barman, k3s - unrelated to GitLab) is
apparently enough to starve that specific thread. More patience, not
a different fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RollingUpdate's default surge kept creating a second pod during every
rollout even with 1 replica - on this CPU-overcommitted single-node
VPS that meant 2 concurrent Ruby boots starving each other, never
converging (load avg hit 39+ on 6 cores). Recreate guarantees only 1
pod exists at a time. No availability cost given minReplicas=maxReplicas=1
already.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GITLAB_OMNIBUS_CONFIG (gitlab_rails['omniauth_providers'] = [...]) is
an omnibus-Docker-image-only convention. This chart's CNG webservice
image never processes it - confirmed via gitlab/charts/gitlab/
templates/_omniauth.tpl, which only reads
global.appConfig.omniauth.providers (a list of {secret,key} refs, each
pointing at a Secret key holding a whole YAML-encoded provider block
loaded via Ruby's YAML.load_file). So GITLAB_OMNIBUS_CONFIG was always
a silent no-op on both home and VPS - neither ever actually had SSO
configured despite gitlab-oidc-secret existing and looking correct.
Yesterday's fix for 'FailedMount: references non-existent secret key:
provider' removed the providers: list entirely instead of populating
that key - stopped the crash, but also silently deleted the only real
OIDC config path on both sites (no error, login page just lost its SSO
button). This restores providers: on both, and gitlab-oidc-secret's
ExternalSecret template now actually renders a provider key containing
a real YAML provider block (with the templated client_id/secret
substituted in), matching what _omniauth.tpl expects.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
VPS box is chronically CPU-overcommitted (load avg ~22-26 on 6 cores,
confirmed live - unrelated pre-existing services: barman, k3s itself,
node/gunicorn apps). Chart-default liveness probes (~200s grace) were
SIGTERM-killing both webservice and sidekiq mid-boot (still inside
bootsnap's require), every restart resetting progress to zero and
never actually finishing. Loosened to a ~7min grace window and dropped
webservice to 1 worker process instead of 2, less to fork/boot.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ingress pointed at 'gitlab-webservice-default' but the chart release
name on the VPS is 'vps-gitlab', so the actual Service Helm creates is
'vps-gitlab-webservice-default' (confirmed via 'kubectl get svc -n
gitlab'). Home's GitLab isn't affected - it routes via Istio
VirtualService, not this Ingress, and its release name has no prefix.
This was the actual cause of the persistent 404 - webservice pod was
healthy and passing readiness checks the whole time, Traefik just had
no matching backend to route to.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real bug affecting both home and VPS GitLab, found live diagnosing why
VPS's webservice pod was stuck at Init:0/3 for 74+ minutes (surfacing as
404s - nothing was ever actually serving):
global.appConfig.omniauth.providers: [{secret: gitlab-oidc-secret, key:
provider}] requires that secret to have a 'provider' key (a full YAML
omniauth provider config) - gitlab-oidc-secret never has one, only
GITLAB_OIDC_CLIENT_ID/SECRET, which get consumed via extraEnvFrom +
GITLAB_OMNIBUS_CONFIG's gitlab_rails['omniauth_providers'] instead (the
actual, complete mechanism already configuring OIDC - this providers:
list was always redundant). Removed it from both values.yaml files.
Home's GitLab has run fine so far because Kubernetes doesn't re-validate
already-mounted volumes when a referenced Secret's shape changes -
webservice/sidekiq there just haven't restarted since gitlab-oidc-secret
was last (if ever) in a shape with that key. VPS's identical values
shape hit it immediately on first pod creation. This was a live landmine
for home too - fixed before it could bite on a future restart.
Also: global.kas.enabled: false added to the VPS values (gitlab.kas.enabled
alone doesn't stop KAS pods - found live, separate toggle).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
User wants actual GitLab on the VPS, not just Gitea's pull-mirror -
deliberate scope increase from the original plan's 'keep the VPS
lighter' design for git hosting specifically.
Simplified single-node equivalent of apps/gitlab (home) - same chart
version (7.7.0), same Authentik OIDC provider/client_id (added a second
redirect_uri to the shared GitLab provider in infrastructure/authentik/
gitlab-blueprint.yaml), same known chart bugs already diagnosed at home
(redis-init-fix.yaml for the initializer-ordering bug) - but:
- 1 CNPG instance (local-path) instead of home's 3-instance HA
- No Praefect - pointless HA routing with a single Gitaly node
- No PgBouncer - direct CNPG connection, traffic is low enough
- No SAML, no Container Registry, no KAS - addable later if needed
- Own Traefik Ingress + https-redirect Middleware from the start
(found needed the hard way on Vault/Gitea/n8n/Nextcloud/Authentik
earlier this session - applying that lesson here immediately)
New Vault paths this needed: secret/vps/gitlab-redis (freshly generated,
this Redis instance is new, nothing to reuse) and widened the
vps-eso-reader policy to also allow secret/gitlab-oauth (the shared
Authentik client, already used by home's GitLab).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Scooby Husky