global.workhorse.tls.enabled replaced workhorse's plaintext listener
with a TLS-only one on port 8181 - but that port isn't internal-API
-only, it's also what gitlab-web HTTPRoute routes ALL public
gitlab.kube.huskypup.net traffic to. Envoy terminates public HTTPS
and proxies to the pod in plaintext, so the TLS-only listener 400'd
every public request ("Client sent an HTTP request to an HTTPS
server"). Restored live via a direct ConfigMap patch +
rollout restart before this commit could even be pushed, since the
outage took out this repo's own GitLab-hosted git access too.
global.workhorse.host stays (harmless without tls.enabled). Securing
the VPS's future internal-API-over-WAN hop needs a different
mechanism that doesn't collide with the public listener - not solved
here.
Per the approved plan (Istio federation scoped separately, current
NodePort/PeerAuth/cert-manager pattern continues for these 2 remaining
links). Real Let's Encrypt certs (not self-signed) via the same DNS-01
ClusterIssuer used everywhere else - both sites already trust Let's
Encrypt's public CA, no custom CA distribution needed.
- global.gitaly.tls.enabled: adds tls_listen_addr (8076) alongside the
existing plaintext listener (8075, stays for in-cluster traffic
that doesn't need it). Requires
global.praefect.virtualStorages[0].tlsSecretName (chart's own NOTES.txt
validation rejects the render without it, confirmed live).
- global.workhorse.tls.enabled + global.workhorse.host: Gitaly's [gitlab]
url (internal-API callback for gitlab-shell hooks) switches from an
unpublishable *.svc.cluster.local name to a real floating hostname
(gitlab-internal-api.ha.huskypup.net) that Let's Encrypt CAN certify -
confirmed live this replaces the plaintext [[listeners]] block with
[listeners.tls] on the SAME port (8181), not a second port like Gitaly.
- praefect-ha-configmap.yaml (the manually-patched live config) updated
to tls:// + floating hostnames for all 3 home nodes, so home's own
Praefect uses the identical addresses it'll hand the VPS as peer
addresses once Phase 2b's 4th node lands.
- New CoreDNS rewrite for gitlab-internal-api.ha.huskypup.net (same
local-rewrite pattern as pg-*/gitaly-* already have).
Sequencing note: applying live - Certificates must issue before the
TLS-enabling values land, or Gitaly/webservice pods fail to start
(missing secret for their init container's cert copy).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The CONFIG_TEMPLATE_DIRECTORY redirect from the previous commit doesn't
work - confirmed live that extraVolumes is a dead values key for the
Praefect subchart specifically (its statefulset.yaml never calls the
gitlab.extraVolumes helper in its volumes: list, only volumeMounts
calls the corresponding helper - a real chart limitation, not a config
mistake). A dangling volumeMount with no matching volume would have
failed to schedule.
Real fix: ignoreDifferences on ConfigMap gitlab-praefect's data field
(argocd-apps/apps/gitlab.yaml) lets Helm create the object normally
while ArgoCD stops reconciling its content afterward - the actual
config gets kubectl-patched onto the live object directly.
praefect-ha-configmap.yaml is now a git-tracked reference/documentation
copy (deployed under its own harmless name) rather than something
Helm/ArgoCD wires in on its own.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Proves the CONFIG_TEMPLATE_DIRECTORY redirect works before Phase 2b
adds any actual cross-site dependency. Content is byte-identical to
the chart's current rendering (confirmed live) - this commit only
tests the override plumbing itself: a new ConfigMap
(praefect-ha-configmap.yaml) mounted at a different path than the
chart's own /etc/gitaly/templates (can't reuse that path/volume name -
Kubernetes rejects duplicates), with CONFIG_TEMPLATE_DIRECTORY
overridden via extraEnv to point at it instead (last-wins env
semantics, confirmed Kubernetes-documented behavior).
Also removes gitlab.praefect.virtualStorages - confirmed live dead
config, global.praefect.virtualStorages is what the chart's template
actually reads.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GITLAB_OMNIBUS_CONFIG (gitlab_rails['omniauth_providers'] = [...]) is
an omnibus-Docker-image-only convention. This chart's CNG webservice
image never processes it - confirmed via gitlab/charts/gitlab/
templates/_omniauth.tpl, which only reads
global.appConfig.omniauth.providers (a list of {secret,key} refs, each
pointing at a Secret key holding a whole YAML-encoded provider block
loaded via Ruby's YAML.load_file). So GITLAB_OMNIBUS_CONFIG was always
a silent no-op on both home and VPS - neither ever actually had SSO
configured despite gitlab-oidc-secret existing and looking correct.
Yesterday's fix for 'FailedMount: references non-existent secret key:
provider' removed the providers: list entirely instead of populating
that key - stopped the crash, but also silently deleted the only real
OIDC config path on both sites (no error, login page just lost its SSO
button). This restores providers: on both, and gitlab-oidc-secret's
ExternalSecret template now actually renders a provider key containing
a real YAML provider block (with the templated client_id/secret
substituted in), matching what _omniauth.tpl expects.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real bug affecting both home and VPS GitLab, found live diagnosing why
VPS's webservice pod was stuck at Init:0/3 for 74+ minutes (surfacing as
404s - nothing was ever actually serving):
global.appConfig.omniauth.providers: [{secret: gitlab-oidc-secret, key:
provider}] requires that secret to have a 'provider' key (a full YAML
omniauth provider config) - gitlab-oidc-secret never has one, only
GITLAB_OIDC_CLIENT_ID/SECRET, which get consumed via extraEnvFrom +
GITLAB_OMNIBUS_CONFIG's gitlab_rails['omniauth_providers'] instead (the
actual, complete mechanism already configuring OIDC - this providers:
list was always redundant). Removed it from both values.yaml files.
Home's GitLab has run fine so far because Kubernetes doesn't re-validate
already-mounted volumes when a referenced Secret's shape changes -
webservice/sidekiq there just haven't restarted since gitlab-oidc-secret
was last (if ever) in a shape with that key. VPS's identical values
shape hit it immediately on first pod creation. This was a live landmine
for home too - fixed before it could bite on a future restart.
Also: global.kas.enabled: false added to the VPS values (gitlab.kas.enabled
alone doesn't stop KAS pods - found live, separate toggle).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Gitaly CPU limit was 250m causing throttling on git operations (485ms
p99 to webservice). Increased to 1 core with 100m request. Also added
workerProcesses: 3 to webservice for better request concurrency
(was default 2).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Webservice was at 94% memory (1875Mi/2Gi) with only 50m CPU request,
causing CPU throttling and slow first requests after idle periods.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scooby Husky