#!/bin/bash # Verify GitLab Redis password rotation automation is working set -euo pipefail echo "====================================================================" echo "GitLab Redis Password Rotation Automation - Verification Script" echo "====================================================================" echo "" # Colors for output GREEN='\033[0;32m' RED='\033[0;31m' YELLOW='\033[1;33m' NC='\033[0m' # No Color check_pass() { echo -e "${GREEN}✓${NC} $1" } check_fail() { echo -e "${RED}✗${NC} $1" } check_warn() { echo -e "${YELLOW}⚠${NC} $1" } ERRORS=0 # Check 1: CronJob exists echo "Checking CronJob deployment..." if kubectl get cronjob -n gitlab redis-secret-monitor >/dev/null 2>&1; then check_pass "CronJob 'redis-secret-monitor' is deployed" # Get schedule SCHEDULE=$(kubectl get cronjob -n gitlab redis-secret-monitor -o jsonpath='{.spec.schedule}') echo " Schedule: $SCHEDULE" # Get last successful run LAST_SUCCESS=$(kubectl get cronjob -n gitlab redis-secret-monitor -o jsonpath='{.status.lastSuccessfulTime}' 2>/dev/null || echo "Never") echo " Last successful run: $LAST_SUCCESS" else check_fail "CronJob 'redis-secret-monitor' not found" ERRORS=$((ERRORS + 1)) fi echo "" # Check 2: RBAC resources exist echo "Checking RBAC resources..." if kubectl get serviceaccount -n gitlab redis-restart-sa >/dev/null 2>&1; then check_pass "ServiceAccount 'redis-restart-sa' exists" else check_fail "ServiceAccount 'redis-restart-sa' not found" ERRORS=$((ERRORS + 1)) fi if kubectl get role -n gitlab redis-restart-role >/dev/null 2>&1; then check_pass "Role 'redis-restart-role' exists" else check_fail "Role 'redis-restart-role' not found" ERRORS=$((ERRORS + 1)) fi if kubectl get rolebinding -n gitlab redis-restart-binding >/dev/null 2>&1; then check_pass "RoleBinding 'redis-restart-binding' exists" else check_fail "RoleBinding 'redis-restart-binding' not found" ERRORS=$((ERRORS + 1)) fi echo "" # Check 3: StatefulSet has tracking annotation echo "Checking StatefulSet annotations..." SECRET_VERSION=$(kubectl get statefulset -n gitlab redis-gitlab -o jsonpath='{.spec.template.metadata.annotations.secret-version/redis-password}' 2>/dev/null || echo "") if [ -n "$SECRET_VERSION" ]; then check_pass "StatefulSet has tracking annotation" echo " Tracked secret version: $SECRET_VERSION" else check_warn "StatefulSet missing tracking annotation (will be added on first CronJob run)" fi RESTART_TIME=$(kubectl get statefulset -n gitlab redis-gitlab -o jsonpath='{.spec.template.metadata.annotations.restarted-at}' 2>/dev/null || echo "") if [ -n "$RESTART_TIME" ]; then echo " Last restart: $RESTART_TIME" fi echo "" # Check 4: Redis is running echo "Checking Redis pod status..." if kubectl get pod -n gitlab redis-gitlab-0 >/dev/null 2>&1; then REDIS_STATUS=$(kubectl get pod -n gitlab redis-gitlab-0 -o jsonpath='{.status.phase}') if [ "$REDIS_STATUS" = "Running" ]; then check_pass "Redis pod is Running" else check_fail "Redis pod status: $REDIS_STATUS" ERRORS=$((ERRORS + 1)) fi else check_fail "Redis pod 'redis-gitlab-0' not found" ERRORS=$((ERRORS + 1)) fi echo "" # Check 5: Redis authentication works echo "Testing Redis authentication..." if RESULT=$(kubectl exec -n gitlab redis-gitlab-0 -- redis-cli -a "$(kubectl get secret -n gitlab redis-gitlab-secret -o jsonpath='{.data.password}' | base64 -d)" ping 2>&1 | grep PONG); then check_pass "Redis authentication successful (PONG)" else check_fail "Redis authentication failed" ERRORS=$((ERRORS + 1)) fi echo "" # Check 6: GitLab Sidekiq is healthy echo "Checking GitLab Sidekiq status..." SIDEKIQ_READY=$(kubectl get pods -n gitlab -l app=sidekiq -o jsonpath='{.items[*].status.containerStatuses[*].ready}' | grep -o "true" | wc -l) SIDEKIQ_TOTAL=$(kubectl get pods -n gitlab -l app=sidekiq --no-headers | wc -l) if [ "$SIDEKIQ_READY" -eq "$SIDEKIQ_TOTAL" ] && [ "$SIDEKIQ_TOTAL" -gt 0 ]; then check_pass "Sidekiq pods are healthy ($SIDEKIQ_READY/$SIDEKIQ_TOTAL ready)" else check_warn "Sidekiq pods: $SIDEKIQ_READY/$SIDEKIQ_TOTAL ready" fi echo "" # Check 7: Recent job executions echo "Checking recent CronJob executions..." JOB_COUNT=$(kubectl get jobs -n gitlab -l app=redis-secret-monitor --no-headers 2>/dev/null | wc -l) if [ "$JOB_COUNT" -gt 0 ]; then check_pass "Found $JOB_COUNT recent job execution(s)" # Show last job LAST_JOB=$(kubectl get jobs -n gitlab -l app=redis-secret-monitor --sort-by=.metadata.creationTimestamp -o name 2>/dev/null | tail -1) if [ -n "$LAST_JOB" ]; then echo "" echo " Last job logs:" kubectl logs -n gitlab "$LAST_JOB" 2>/dev/null | sed 's/^/ /' fi else check_warn "No recent job executions (CronJob may not have run yet)" fi echo "" # Check 8: Secret version comparison echo "Checking secret version synchronization..." CURRENT_SECRET_VERSION=$(kubectl get secret -n gitlab redis-gitlab-secret -o jsonpath='{.metadata.resourceVersion}') TRACKED_SECRET_VERSION=$(kubectl get statefulset -n gitlab redis-gitlab -o jsonpath='{.spec.template.metadata.annotations.secret-version/redis-password}' 2>/dev/null || echo "") echo " Current secret version: $CURRENT_SECRET_VERSION" echo " Tracked version in StatefulSet: $TRACKED_SECRET_VERSION" if [ "$CURRENT_SECRET_VERSION" = "$TRACKED_SECRET_VERSION" ]; then check_pass "Secret versions are synchronized" elif [ -z "$TRACKED_SECRET_VERSION" ]; then check_warn "StatefulSet not yet tracking secret version (first sync pending)" else check_warn "Secret versions differ - restart pending on next CronJob run" echo " Next scheduled run will synchronize versions" fi echo "" # Summary echo "====================================================================" if [ $ERRORS -eq 0 ]; then echo -e "${GREEN}✓ All checks passed!${NC}" echo "" echo "Redis password rotation automation is properly configured." echo "The system will automatically restart Redis when passwords rotate." else echo -e "${RED}✗ $ERRORS check(s) failed${NC}" echo "" echo "Please review the errors above and ensure all automation" echo "components are properly deployed." exit 1 fi echo "===================================================================="