RollingUpdate's default surge kept creating a second pod during every
rollout even with 1 replica - on this CPU-overcommitted single-node
VPS that meant 2 concurrent Ruby boots starving each other, never
converging (load avg hit 39+ on 6 cores). Recreate guarantees only 1
pod exists at a time. No availability cost given minReplicas=maxReplicas=1
already.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GITLAB_OMNIBUS_CONFIG (gitlab_rails['omniauth_providers'] = [...]) is
an omnibus-Docker-image-only convention. This chart's CNG webservice
image never processes it - confirmed via gitlab/charts/gitlab/
templates/_omniauth.tpl, which only reads
global.appConfig.omniauth.providers (a list of {secret,key} refs, each
pointing at a Secret key holding a whole YAML-encoded provider block
loaded via Ruby's YAML.load_file). So GITLAB_OMNIBUS_CONFIG was always
a silent no-op on both home and VPS - neither ever actually had SSO
configured despite gitlab-oidc-secret existing and looking correct.
Yesterday's fix for 'FailedMount: references non-existent secret key:
provider' removed the providers: list entirely instead of populating
that key - stopped the crash, but also silently deleted the only real
OIDC config path on both sites (no error, login page just lost its SSO
button). This restores providers: on both, and gitlab-oidc-secret's
ExternalSecret template now actually renders a provider key containing
a real YAML provider block (with the templated client_id/secret
substituted in), matching what _omniauth.tpl expects.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
VPS box is chronically CPU-overcommitted (load avg ~22-26 on 6 cores,
confirmed live - unrelated pre-existing services: barman, k3s itself,
node/gunicorn apps). Chart-default liveness probes (~200s grace) were
SIGTERM-killing both webservice and sidekiq mid-boot (still inside
bootsnap's require), every restart resetting progress to zero and
never actually finishing. Loosened to a ~7min grace window and dropped
webservice to 1 worker process instead of 2, less to fork/boot.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ingress pointed at 'gitlab-webservice-default' but the chart release
name on the VPS is 'vps-gitlab', so the actual Service Helm creates is
'vps-gitlab-webservice-default' (confirmed via 'kubectl get svc -n
gitlab'). Home's GitLab isn't affected - it routes via Istio
VirtualService, not this Ingress, and its release name has no prefix.
This was the actual cause of the persistent 404 - webservice pod was
healthy and passing readiness checks the whole time, Traefik just had
no matching backend to route to.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real bug affecting both home and VPS GitLab, found live diagnosing why
VPS's webservice pod was stuck at Init:0/3 for 74+ minutes (surfacing as
404s - nothing was ever actually serving):
global.appConfig.omniauth.providers: [{secret: gitlab-oidc-secret, key:
provider}] requires that secret to have a 'provider' key (a full YAML
omniauth provider config) - gitlab-oidc-secret never has one, only
GITLAB_OIDC_CLIENT_ID/SECRET, which get consumed via extraEnvFrom +
GITLAB_OMNIBUS_CONFIG's gitlab_rails['omniauth_providers'] instead (the
actual, complete mechanism already configuring OIDC - this providers:
list was always redundant). Removed it from both values.yaml files.
Home's GitLab has run fine so far because Kubernetes doesn't re-validate
already-mounted volumes when a referenced Secret's shape changes -
webservice/sidekiq there just haven't restarted since gitlab-oidc-secret
was last (if ever) in a shape with that key. VPS's identical values
shape hit it immediately on first pod creation. This was a live landmine
for home too - fixed before it could bite on a future restart.
Also: global.kas.enabled: false added to the VPS values (gitlab.kas.enabled
alone doesn't stop KAS pods - found live, separate toggle).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
User wants actual GitLab on the VPS, not just Gitea's pull-mirror -
deliberate scope increase from the original plan's 'keep the VPS
lighter' design for git hosting specifically.
Simplified single-node equivalent of apps/gitlab (home) - same chart
version (7.7.0), same Authentik OIDC provider/client_id (added a second
redirect_uri to the shared GitLab provider in infrastructure/authentik/
gitlab-blueprint.yaml), same known chart bugs already diagnosed at home
(redis-init-fix.yaml for the initializer-ordering bug) - but:
- 1 CNPG instance (local-path) instead of home's 3-instance HA
- No Praefect - pointless HA routing with a single Gitaly node
- No PgBouncer - direct CNPG connection, traffic is low enough
- No SAML, no Container Registry, no KAS - addable later if needed
- Own Traefik Ingress + https-redirect Middleware from the start
(found needed the hard way on Vault/Gitea/n8n/Nextcloud/Authentik
earlier this session - applying that lesson here immediately)
New Vault paths this needed: secret/vps/gitlab-redis (freshly generated,
this Redis instance is new, nothing to reuse) and widened the
vps-eso-reader policy to also allow secret/gitlab-oauth (the shared
Authentik client, already used by home's GitLab).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
