Adds externalClusters connectionParameters + replica.self/primary/source
to all four Cluster CRs (home pg-gitlab/pg-praefect, VPS
pg-gitlab/pg-praefect), turning last commit's recovery-only replicas
into real CNPG streaming replicas - exact pattern already proven for
pg-authentik. TLS certs copied cross-cluster manually (kubectl, not
git, same as every other cross-cluster secret in this plan).
Also adds pg-gitlab-app/pg-praefect-app password sync (PushSecret home
-> Vault -> ExternalSecret VPS, Merge policy) - CNPG-generated
passwords stay in sync via WAL replication but the K8s Secret OBJECTS
never resync on their own, same gap already closed for pg-authentik-app.
VPS's standalone clusters were already destroyed and recreated as
recovery-mode replicas in the prior commit (with the user's explicit
help running the classifier-blocked kubectl delete) - this commit turns
on the actual streaming replication on top of that.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces VPS's independent standalone pg-gitlab (own initdb, own
root/Scooby test data - never meant to survive this, per the approved
plan) with a bootstrap.recovery replica of home's, matching the
pg-authentik precedent. New pg-praefect cluster on the VPS too (never
existed before - VPS ran Gitaly without Praefect).
Stage 1 of 2: recovery-only, no streaming replication config yet - that
needs this cluster's own CNPG-generated certs to exist first, which
only happens once CNPG actually creates the cluster. Stage 2 (next
commit) copies certs cross-cluster and adds externalClusters
connectionParameters + replica.self/primary/source to all four Cluster
CRs (home pg-gitlab/pg-praefect, VPS pg-gitlab/pg-praefect).
postgresql.parameters copied verbatim from home's manifests (must
match-or-exceed, confirmed live for pg-authentik). vps-minio-secret
created manually on the VPS (kubectl, not git - same as every other
cross-cluster secret in this plan) pointing at the VPS's own local
MinIO, which now holds home's pg-gitlab/pg-praefect base backups
(triggered manually via Backup CRs since neither had run yet).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PushSecret (home) + ExternalSecret (VPS) pairs for the four
chart-generated secrets whose values must match before Phase 1c's
Postgres replication goes live: gitlab-rails-secret (secret_key_base/
db_key_base/otp_key_base/openid_connect_signing_key, all in one
secrets.yml key - confirmed live, not four separate keys as first
assumed), gitlab-gitlab-shell-secret, gitlab-gitaly-secret,
gitlab-praefect-secret (the latter two also needed for Phase 2's
cross-site gRPC auth). Same pattern as
infrastructure/authentik/manifests/ha-postgres-app-pushsecret.yaml.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per the approved plan (jiggly-snacking-iverson.md, 'GitLab True
Cross-Site Replication' section) - replacing the git-mirror CronJob
with real CNPG streaming replication for pg-gitlab and pg-praefect,
reusing the exact pattern already proven for pg-authentik.
This commit is network plumbing only, no DB replication yet:
- Repoint both CNPG clusters' barmanObjectStore backup target from
in-cluster gitlab-minio-svc to VPS MinIO (also becomes the initial
seed source for the VPS's replicas in Phase 1c).
- New ha-gitlab-postgres/ha-praefect-postgres NodePort Services
(bypass CNPG's own -rw Services) on both home and VPS.
- New PeerAuthentication with portLevelMtls PERMISSIVE on the replication
ports - ambient mesh STRICT default would otherwise reset the
VPS/witness's connection before the Postgres TLS handshake starts.
- Extended allow-gitlab-access AuthorizationPolicy with a port-scoped
rule for the same traffic (no source.namespaces rule can match
external, non-mesh peers).
- CoreDNS rewrites for pg-gitlab.ha.huskypup.net / pg-praefect.ha.huskypup.net
so home's own pods reach the floating hostname locally instead of
round-tripping through the WAN forward (no NAT hairpin support).
Still needed before Phase 1b/1c: UniFi WAN port-forwards for external
ports 61442/61443 -> talos-cp-01:32442/32443, source-restricted to the
VPS + witness public IPs (manual, same as the existing 61432 rule).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GitLab's native pull-mirror feature is Premium/Ultimate-gated on this
chart/version - confirmed live: the project API rejects 'mirror' and
'import_url' as unrecognized params on this CE instance entirely.
Plain git equivalent instead: CronJob every 15min does
'git clone --mirror' from home's Scooby/Homelabv4 (read-only deploy
token) then 'git push --mirror' to the VPS copy (write_repository
PAT, since deploy tokens can't push - not a valid scope for them
either). Matches the original plan's 'Gitea pull mirror of the GitOps
source' intent now that real GitLab replaced Gitea. Both tokens in
Vault at secret/vps/gitlab-mirror, pulled via ExternalSecret.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GITLAB_OMNIBUS_CONFIG (gitlab_rails['omniauth_providers'] = [...]) is
an omnibus-Docker-image-only convention. This chart's CNG webservice
image never processes it - confirmed via gitlab/charts/gitlab/
templates/_omniauth.tpl, which only reads
global.appConfig.omniauth.providers (a list of {secret,key} refs, each
pointing at a Secret key holding a whole YAML-encoded provider block
loaded via Ruby's YAML.load_file). So GITLAB_OMNIBUS_CONFIG was always
a silent no-op on both home and VPS - neither ever actually had SSO
configured despite gitlab-oidc-secret existing and looking correct.
Yesterday's fix for 'FailedMount: references non-existent secret key:
provider' removed the providers: list entirely instead of populating
that key - stopped the crash, but also silently deleted the only real
OIDC config path on both sites (no error, login page just lost its SSO
button). This restores providers: on both, and gitlab-oidc-secret's
ExternalSecret template now actually renders a provider key containing
a real YAML provider block (with the templated client_id/secret
substituted in), matching what _omniauth.tpl expects.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ingress pointed at 'gitlab-webservice-default' but the chart release
name on the VPS is 'vps-gitlab', so the actual Service Helm creates is
'vps-gitlab-webservice-default' (confirmed via 'kubectl get svc -n
gitlab'). Home's GitLab isn't affected - it routes via Istio
VirtualService, not this Ingress, and its release name has no prefix.
This was the actual cause of the persistent 404 - webservice pod was
healthy and passing readiness checks the whole time, Traefik just had
no matching backend to route to.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
User wants actual GitLab on the VPS, not just Gitea's pull-mirror -
deliberate scope increase from the original plan's 'keep the VPS
lighter' design for git hosting specifically.
Simplified single-node equivalent of apps/gitlab (home) - same chart
version (7.7.0), same Authentik OIDC provider/client_id (added a second
redirect_uri to the shared GitLab provider in infrastructure/authentik/
gitlab-blueprint.yaml), same known chart bugs already diagnosed at home
(redis-init-fix.yaml for the initializer-ordering bug) - but:
- 1 CNPG instance (local-path) instead of home's 3-instance HA
- No Praefect - pointless HA routing with a single Gitaly node
- No PgBouncer - direct CNPG connection, traffic is low enough
- No SAML, no Container Registry, no KAS - addable later if needed
- Own Traefik Ingress + https-redirect Middleware from the start
(found needed the hard way on Vault/Gitea/n8n/Nextcloud/Authentik
earlier this session - applying that lesson here immediately)
New Vault paths this needed: secret/vps/gitlab-redis (freshly generated,
this Redis instance is new, nothing to reuse) and widened the
vps-eso-reader policy to also allow secret/gitlab-oauth (the shared
Authentik client, already used by home's GitLab).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
