Per the approved plan (Istio federation scoped separately, current
NodePort/PeerAuth/cert-manager pattern continues for these 2 remaining
links). Real Let's Encrypt certs (not self-signed) via the same DNS-01
ClusterIssuer used everywhere else - both sites already trust Let's
Encrypt's public CA, no custom CA distribution needed.
- global.gitaly.tls.enabled: adds tls_listen_addr (8076) alongside the
existing plaintext listener (8075, stays for in-cluster traffic
that doesn't need it). Requires
global.praefect.virtualStorages[0].tlsSecretName (chart's own NOTES.txt
validation rejects the render without it, confirmed live).
- global.workhorse.tls.enabled + global.workhorse.host: Gitaly's [gitlab]
url (internal-API callback for gitlab-shell hooks) switches from an
unpublishable *.svc.cluster.local name to a real floating hostname
(gitlab-internal-api.ha.huskypup.net) that Let's Encrypt CAN certify -
confirmed live this replaces the plaintext [[listeners]] block with
[listeners.tls] on the SAME port (8181), not a second port like Gitaly.
- praefect-ha-configmap.yaml (the manually-patched live config) updated
to tls:// + floating hostnames for all 3 home nodes, so home's own
Praefect uses the identical addresses it'll hand the VPS as peer
addresses once Phase 2b's 4th node lands.
- New CoreDNS rewrite for gitlab-internal-api.ha.huskypup.net (same
local-rewrite pattern as pg-*/gitaly-* already have).
Sequencing note: applying live - Certificates must issue before the
TLS-enabling values land, or Gitaly/webservice pods fail to start
(missing secret for their init container's cert copy).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per-pod NodePort Services for the 3 existing Gitaly nodes
(ha-gitaly-nodeport.yaml) plus home's Rails internal API
(ha-gitlab-internal-api-nodeport.yaml, needed for gitlab-shell hooks
whenever the VPS node is primary for a repo). PeerAuthentication
PERMISSIVE + AuthorizationPolicy port rules for both, matching the
proven Postgres pattern. CoreDNS rewrites so home's own Praefect
resolves the floating hostnames locally instead of round-tripping.
Reached over the Netbird mesh, not the UniFi WAN forward used for
Postgres - Gitaly gRPC here is plaintext (carries the gitaly_token,
gitlab_shell_secret, and raw repo contents), unlike CNPG's
TLS-negotiated stream.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The CONFIG_TEMPLATE_DIRECTORY redirect from the previous commit doesn't
work - confirmed live that extraVolumes is a dead values key for the
Praefect subchart specifically (its statefulset.yaml never calls the
gitlab.extraVolumes helper in its volumes: list, only volumeMounts
calls the corresponding helper - a real chart limitation, not a config
mistake). A dangling volumeMount with no matching volume would have
failed to schedule.
Real fix: ignoreDifferences on ConfigMap gitlab-praefect's data field
(argocd-apps/apps/gitlab.yaml) lets Helm create the object normally
while ArgoCD stops reconciling its content afterward - the actual
config gets kubectl-patched onto the live object directly.
praefect-ha-configmap.yaml is now a git-tracked reference/documentation
copy (deployed under its own harmless name) rather than something
Helm/ArgoCD wires in on its own.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Proves the CONFIG_TEMPLATE_DIRECTORY redirect works before Phase 2b
adds any actual cross-site dependency. Content is byte-identical to
the chart's current rendering (confirmed live) - this commit only
tests the override plumbing itself: a new ConfigMap
(praefect-ha-configmap.yaml) mounted at a different path than the
chart's own /etc/gitaly/templates (can't reuse that path/volume name -
Kubernetes rejects duplicates), with CONFIG_TEMPLATE_DIRECTORY
overridden via extraEnv to point at it instead (last-wins env
semantics, confirmed Kubernetes-documented behavior).
Also removes gitlab.praefect.virtualStorages - confirmed live dead
config, global.praefect.virtualStorages is what the chart's template
actually reads.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds externalClusters connectionParameters + replica.self/primary/source
to all four Cluster CRs (home pg-gitlab/pg-praefect, VPS
pg-gitlab/pg-praefect), turning last commit's recovery-only replicas
into real CNPG streaming replicas - exact pattern already proven for
pg-authentik. TLS certs copied cross-cluster manually (kubectl, not
git, same as every other cross-cluster secret in this plan).
Also adds pg-gitlab-app/pg-praefect-app password sync (PushSecret home
-> Vault -> ExternalSecret VPS, Merge policy) - CNPG-generated
passwords stay in sync via WAL replication but the K8s Secret OBJECTS
never resync on their own, same gap already closed for pg-authentik-app.
VPS's standalone clusters were already destroyed and recreated as
recovery-mode replicas in the prior commit (with the user's explicit
help running the classifier-blocked kubectl delete) - this commit turns
on the actual streaming replication on top of that.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PushSecret (home) + ExternalSecret (VPS) pairs for the four
chart-generated secrets whose values must match before Phase 1c's
Postgres replication goes live: gitlab-rails-secret (secret_key_base/
db_key_base/otp_key_base/openid_connect_signing_key, all in one
secrets.yml key - confirmed live, not four separate keys as first
assumed), gitlab-gitlab-shell-secret, gitlab-gitaly-secret,
gitlab-praefect-secret (the latter two also needed for Phase 2's
cross-site gRPC auth). Same pattern as
infrastructure/authentik/manifests/ha-postgres-app-pushsecret.yaml.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per the approved plan (jiggly-snacking-iverson.md, 'GitLab True
Cross-Site Replication' section) - replacing the git-mirror CronJob
with real CNPG streaming replication for pg-gitlab and pg-praefect,
reusing the exact pattern already proven for pg-authentik.
This commit is network plumbing only, no DB replication yet:
- Repoint both CNPG clusters' barmanObjectStore backup target from
in-cluster gitlab-minio-svc to VPS MinIO (also becomes the initial
seed source for the VPS's replicas in Phase 1c).
- New ha-gitlab-postgres/ha-praefect-postgres NodePort Services
(bypass CNPG's own -rw Services) on both home and VPS.
- New PeerAuthentication with portLevelMtls PERMISSIVE on the replication
ports - ambient mesh STRICT default would otherwise reset the
VPS/witness's connection before the Postgres TLS handshake starts.
- Extended allow-gitlab-access AuthorizationPolicy with a port-scoped
rule for the same traffic (no source.namespaces rule can match
external, non-mesh peers).
- CoreDNS rewrites for pg-gitlab.ha.huskypup.net / pg-praefect.ha.huskypup.net
so home's own pods reach the floating hostname locally instead of
round-tripping through the WAN forward (no NAT hairpin support).
Still needed before Phase 1b/1c: UniFi WAN port-forwards for external
ports 61442/61443 -> talos-cp-01:32442/32443, source-restricted to the
VPS + witness public IPs (manual, same as the existing 61432 rule).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GITLAB_OMNIBUS_CONFIG (gitlab_rails['omniauth_providers'] = [...]) is
an omnibus-Docker-image-only convention. This chart's CNG webservice
image never processes it - confirmed via gitlab/charts/gitlab/
templates/_omniauth.tpl, which only reads
global.appConfig.omniauth.providers (a list of {secret,key} refs, each
pointing at a Secret key holding a whole YAML-encoded provider block
loaded via Ruby's YAML.load_file). So GITLAB_OMNIBUS_CONFIG was always
a silent no-op on both home and VPS - neither ever actually had SSO
configured despite gitlab-oidc-secret existing and looking correct.
Yesterday's fix for 'FailedMount: references non-existent secret key:
provider' removed the providers: list entirely instead of populating
that key - stopped the crash, but also silently deleted the only real
OIDC config path on both sites (no error, login page just lost its SSO
button). This restores providers: on both, and gitlab-oidc-secret's
ExternalSecret template now actually renders a provider key containing
a real YAML provider block (with the templated client_id/secret
substituted in), matching what _omniauth.tpl expects.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Completes the 'make the repo proper' cleanup from earlier this session -
these were flagged but deliberately not touched in 185e9c2 given the
blast radius (live SSO for 8 apps). User confirmed: fix all 8 now.
infrastructure/authentik/{argocd,gitlab,nextcloud,grafana,n8n,guacamole,
rancher,vault}-blueprint.yaml: converted from plain ConfigMap (client_secret
hardcoded) to ExternalSecret with a templated blueprint body
(client_secret: "{{ .clientSecret }}") pulling from Vault. Chart already
supports mounting blueprints from Secrets (blueprints.secrets, alongside
blueprints.configMaps) - infrastructure/authentik/values.yaml updated to
route these 8 there instead.
For argocd/nextcloud/n8n/guacamole/rancher: Vault already had the matching
value at secret/<app>-oauth (the APP side was already Vault-backed via its
own ExternalSecret) - the blueprint was the only remaining plaintext copy.
For gitlab/grafana/vault: Vault had no copy at all yet - created
secret/{gitlab,grafana,vault}-oauth with the EXISTING live values (not
rotated - these are the actual working credentials right now, rotating
would break login until every consumer is updated in lockstep, which is
out of scope for a cleanup pass). Also fixed the OTHER plaintext copies
that existed for these three specifically:
- apps/gitlab/manifests/external-secret-oidc.yaml (new): replaces a
manually kubectl-created, never-git-tracked gitlab-oidc-secret.
- infrastructure/grafana/manifests/grafana-oauth-secret.yaml: was a
plain Secret whose own comment said 'hardcoded from blueprint'.
- infrastructure/vault/manifests/vault-init-{configmap,job}.yaml: this
one COULDN'T be converted to the same ExternalSecret-from-Vault
pattern - it's the PostSync Job that grants ESO's own Kubernetes-auth
role in Vault, so ESO can't yet authenticate to pull anything from
Vault at the point this script runs (genuinely circular). Sourced
from a new vault-oidc-bootstrap Secret instead - created once
manually (kubectl, not git, matching how Vault's own root/unseal
material is already handled), independent of the ESO pipeline.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes NameError: uninitialized constant Gitlab::Redis::ALL_CLASSES
caused by 7_prometheus_metrics.rb loading before 7_redis.rb
(alphabetically 'p' < 'r'). The ConfigMap provides a 0_redis_early.rb
initializer that requires gitlab/redis before other initializers run.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
