Bitnami Sealed Secrets for Production

Date: 2026-03-13 Status: accepted

Context

Production secrets (JWT_SECRET, SESSION_SECRET, KEYCLOAK_CLIENT_SECRET, POSTGRES_PASSWORD, KEYCLOAK_ADMIN_PASSWORD) were previously stored as plain-text placeholders in values.prod.yaml or passed via --set flags. This approach has several problems:

  1. ArgoCD compatibility: ArgoCD uses helm template (no cluster access), so Helm lookup() always returns empty and randAlphaNum produces different values every render — causing constant drift or missing secret keys.
  2. GitOps gap: Secrets couldn’t be committed to Git, breaking the GitOps principle that the repo is the single source of truth.
  3. Manual bootstrapping: Operators had to kubectl create secret manually before ArgoCD could sync — error-prone and not auditable.

Decision

Adopt Bitnami Sealed Secrets for all production secrets. The Helm chart conditionally renders either a plain Secret (local dev) or a SealedSecret CRD (production) based on sealedSecrets.enabled.

  • The sealed-secrets-controller runs in the prod cluster and holds the private key
  • kubeseal encrypts secret values against the controller’s public key
  • Encrypted values are stored in values.prod.yaml under sealedSecrets.encryptedData — safe to commit
  • The controller decrypts SealedSecret resources into standard K8s Secret objects at runtime
  • All deployments reference the same luckyplans-secrets Secret name — zero deployment changes

Secrets inventory

KeyConsumer(s)
JWT_SECRETapi-gateway
SESSION_SECRETapi-gateway
KEYCLOAK_CLIENT_SECRETapi-gateway
POSTGRES_PASSWORDpostgresql, keycloak
KEYCLOAK_ADMIN_PASSWORDkeycloak

Helper scripts

  • infrastructure/scripts/install-sealed-secrets.sh — one-time controller installation + key backup
  • infrastructure/scripts/seal-secrets.sh — generate and seal all secrets, output ready-to-paste YAML

Consequences

Easier:

  • Secrets are version-controlled (encrypted) — full GitOps compliance
  • ArgoCD syncs deterministically — no more drift or missing keys
  • Secret rotation is scriptable and auditable

Harder:

  • Sealed values are cluster-bound — re-sealing needed if the controller’s key pair changes or you migrate clusters
  • Controller’s private key must be backed up securely — if lost, all sealed secrets are undecryptable
  • kubeseal CLI is required for operators who generate/rotate secrets

No change:

  • Local dev workflow is completely unaffected (plain secrets in values.yaml)
  • Deployment templates are unchanged — they still reference luckyplans-secrets