Secrets Management¶

Before You Read¶

This guide explains how to manage secrets operationally. For the architecture behind this, see Security Model.

Architecture Summary¶

GCP Secret Manager (source of truth)
    ↓ (synced by ESO every N minutes)
Kubernetes Secret (namespace-scoped)
    ↓ (mounted as env vars or volume)
Pod

Critical rule: Never commit secret values to Git. Only ExternalSecret CRDs (which reference secret names, not values) are in Git.

Adding a New Secret¶

Step 1: Create the Secret in GCP Secret Manager¶

Use Terraform (preferred) or gcloud CLI.

Via Terraform (add to infrastructure-management/projects/orofi-{env}/secrets.tf):

module "my_new_secret" {
  source     = "../../modules/secretmanager"
  project_id = local.project_id
  secret_id  = "${local.env}-my-new-secret"
}

Apply:

cd infrastructure-management/projects/orofi-dev
terraform plan
terraform apply

Via gcloud CLI (quick for ad-hoc):

# Create the secret (empty)
gcloud secrets create dev-my-new-secret \
  --project=orofi-dev-cloud \
  --replication-policy=automatic

# Add a version (the actual value)
echo -n "my-secret-value" | gcloud secrets versions add dev-my-new-secret \
  --data-file=- \
  --project=orofi-dev-cloud

Step 2: Grant Access to the Service Account¶

The microservice's GCP service account must have roles/secretmanager.secretAccessor on the new secret.

Via Terraform (add to modules/service-accounts call for that service):

module "microservice_identity_sa" {
  source       = "../../modules/service-accounts"
  ...
  secret_ids = [
    "existing-secret-1",
    "existing-secret-2",
    "dev-my-new-secret",  # ← add this
  ]
}

Step 3: Create an ExternalSecret Resource¶

Add an ExternalSecret to the service's Kubernetes manifests (in infrastructure-configuration):

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-new-secret
  namespace: microservice-identity
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: gcp-secret-manager   # cluster-wide SecretStore, references {env}-ext-secrets-manager SA
    kind: ClusterSecretStore
  target:
    name: my-new-secret        # name of the K8s Secret to create
    creationPolicy: Owner
  data:
    - secretKey: MY_KEY        # key in the K8s Secret
      remoteRef:
        key: dev-my-new-secret # name in GCP Secret Manager

Step 4: Mount the Secret in the Pod¶

In the Helm chart values.yaml for the service, add:

env:
  - name: MY_SECRET_ENV_VAR
    valueFrom:
      secretKeyRef:
        name: my-new-secret
        key: MY_KEY

Or as a volume mount for files (e.g., certificates):

volumes:
  - name: my-cert
    secret:
      secretName: my-new-secret
volumeMounts:
  - name: my-cert
    mountPath: /etc/certs
    readOnly: true

Updating a Secret Value¶

Rotate the Value in GCP¶

# Add a new version (old version remains accessible until disabled)
echo -n "new-secret-value" | gcloud secrets versions add {env}-my-secret \
  --data-file=- \
  --project=orofi-{env}-cloud

Wait for ESO to Sync¶

The External Secrets Operator syncs on the refreshInterval defined in the ExternalSecret (typically 1h). To force immediate sync:

# Annotate the ExternalSecret to trigger a resync
kubectl annotate externalsecret my-new-secret \
  -n microservice-identity \
  force-sync=$(date +%s) \
  --overwrite

Restart Pods to Pick Up New Secret¶

If the secret is mounted as an environment variable (not a volume), pods must restart to pick up the new value:

kubectl rollout restart deployment/microservice-identity -n microservice-identity
kubectl rollout status deployment/microservice-identity -n microservice-identity

Volume mounts vs env vars

Secrets mounted as volumes are updated automatically without a pod restart (Kubernetes refreshes volume content). Secrets injected as environment variables require a pod restart.

Secret Rotation for Database Passwords¶

Database passwords require extra steps because both the GCP secret and the Cloud SQL user password must be updated atomically.

# 1. Generate a new password
NEW_PASSWORD=$(openssl rand -base64 32)

# 2. Update Cloud SQL user password
gcloud sql users set-password {username} \
  --host=% \
  --instance=orofi-{env}-cloud-{env}-oro-mysql-instance \
  --password="$NEW_PASSWORD" \
  --project=orofi-{env}-cloud

# 3. Update the secret in GCP Secret Manager
echo -n "$NEW_PASSWORD" | gcloud secrets versions add {env}-microservice-{name}-db-connection \
  --data-file=- \
  --project=orofi-{env}-cloud

# 4. Force ESO resync
kubectl annotate externalsecret microservice-{name}-db-connection \
  -n microservice-{name} \
  force-sync=$(date +%s) --overwrite

# 5. Restart the service to use new credentials
kubectl rollout restart deployment/microservice-{name} -n microservice-{name}

Secret Inventory by Service¶

Shared Secrets (all services receive these)¶

Secret Name	Purpose
`{env}-shared-microservice-secrets`	Shared configuration values
`{env}-redis-auth-password`	Redis AUTH password

Per-Service Secrets¶

Service	Secret Name
`api-gateway-public`	`{env}-api-gateway-public-secret`
`api-gateway-account`	`{env}-api-gateway-account-secret`
`api-gateway-oro`	`{env}-api-gateway-oro-secret`
`api-gateway-admin-dashboard`	`{env}-api-gateway-admin-dashboard-secret`
`microservice-communication`	`{env}-microservice-communication-secret`
`microservice-identity`	`{env}-microservice-identity-secret`
`microservice-monolith`	`{env}-microservice-monolith-secret`
`microservice-analytics`	`{env}-microservice-analytics-secret`

Database Connection Secrets¶

Secret Name	Used By
`{env}-microservice-communication-db-connection`	`microservice-communication`
`{env}-microservice-identity-db-connection`	`microservice-identity`
`{env}-microservice-monolith-db-connection`	`microservice-monolith`
`{env}-microservice-analytics-db-connection`	`microservice-analytics`
`{env}-flyway-admin-db-connection`	Migration runner

Platform Secrets¶

Secret Name	Used By
`{env}-mongodb-connection`	Mongo Express, MongoDB Operator
`{env}-kafka-secrets`	Kafka SASL credentials
`{env}-oauth2-secrets`	OAuth2 Proxy (Kafka UI, Mongo Express)
`{env}-argocd-secrets`	ArgoCD configuration
`{env}-grafana-secrets`	Grafana
`{env}-slack-webhooks`	Alerting
`{env}-firebase-secret`	Firebase Admin SDK

Authentication Secrets¶

Secret Name	Used By
`microservice-identity-jwt-private-key-secret`	JWT signing
`microservice-identity-apikey-private-key-secret`	API key signing
`microservice-identity-encryption-search-hash-pepper-key-secret`	HMAC pepper
`{env}-admin-dashboard-gateway-apikey`	Admin gateway authentication
`{env}-oro-gateway-apikey`	Oro gateway authentication
`{env}-public-gateway-apikey`	Public gateway authentication
`{env}-account-gateway-apikey`	Account gateway authentication

Mobile Secrets¶

Secret Name	Used By
`{env}-appstore-api-key`	iOS App Store releases
`{env}-appstore-cert`	iOS signing certificate
`{env}-appstore-profile`	iOS provisioning profile