Secrets Audit

Audit date: 2026-04-05 Reviewer: Platform Engineer (CIAM Platform Agent) Ticket: platform#9, [SECURITY] Secrets Manager: Production Credential Inventory Scope: All cryptographic secrets and sensitive credentials used in the Olympus production stack

SOC2 evidence artifact for CC6.1 (logical access to production credentials).

1. Full Secret Inventory

All nine production secrets are sourced from GitHub Actions Secrets via deploy.yml. No secret has a hardcoded default or :-fallback value. If any secret is absent from the GitHub repository secrets, deploy.yml will write an empty string to .env, and the affected service will refuse to start (see Section 4).

No fallback values (:-default) are present for any of the above variables in compose.prod.yml or deploy.yml.

2. Log Configuration Audit, leak_sensitive_values

Kratos v26.2.0 can log session tokens and identity credentials if log.leak_sensitive_values: true. All four production Kratos configs have been inspected. The field is explicitly set to false in all files.

Hydra does not have an equivalent leak_sensitive_values field. No Hydra logging risk.

3. Dev Placeholder Audit

The following dev placeholder patterns were searched across all files mounted in production (platform/prod/):

• PLEASE-CHANGE-ME-I-AM-VERY-INSECURE
• ciam-hydra-secret-change-me
• iam-hydra-pairwise-salt
• secret-cookie (partial match)
• insecure

Result: No dev placeholder values found in any file under platform/prod/. All Kratos and Hydra YAML configs in prod/ delegate every secret to environment variable substitution. No inline secret values are present in any production-mounted config file.

Evidence: platform/prod/ciam-hydra/hydra.yml, platform/prod/iam-hydra/hydra.yml contain only structural configuration (cookie mode, OIDC subject types) with comments indicating the env vars that override secrets. platform/prod/ciam-kratos/kratos.yml and platform/prod/iam-kratos/kratos.yml contain only comments referencing the env vars; secrets are fully delegated.

4. Startup Behavior Under Empty Secrets

Test Methodology

Empirical tests were executed on 2026-04-05 using Podman v5.8.1 (podman-machine-default, applehv). Images used: docker.io/oryd/kratos:v26.2.0 and docker.io/oryd/hydra:v26.2.0, the exact images used in production (compose.prod.yml).

Kratos: Empty SECRETS_COOKIE and SECRETS_CIPHER

Command run:

podman run --rm \
  -v kratos-test-config:/etc/kratos:ro \
  -e DSN='sqlite:///tmp/test.db?_fk=true' \
  -e SECRETS_COOKIE='' \
  -e SECRETS_CIPHER='' \
  docker.io/oryd/kratos:v26.2.0 \
  serve -c /etc/kratos/kratos.yml

Observed output (exit code 1):

The configuration contains values or keys which are invalid:
secrets: map[cipher:<nil> cookie:<nil>]
         ^-- validation failed

secrets.cookie: <nil>
                ^-- expected array, but got null

secrets.cipher: <nil>
                ^-- expected array, but got null

level=error msg=Unable to instantiate configuration.
Error: validation failed

Verdict: PASS. Kratos v26.2.0 refuses to start with empty cookie or cipher secrets. The container exits immediately with code 1 and a clear error message identifying the missing secrets. Silent use of empty secrets is not possible.

Hydra: Empty SECRETS_SYSTEM

Command run:

podman run --rm \
  -v kratos-test-config:/etc/hydra:ro \
  -e DSN='memory' \
  -e SECRETS_SYSTEM='' \
  -e OIDC_SUBJECT_IDENTIFIERS_PAIRWISE_SALT='' \
  -e URLS_SELF_ISSUER='http://localhost:4444/' \
  docker.io/oryd/hydra:v26.2.0 \
  serve -c /etc/hydra/hydra.yml all

Observed output (exit code 1):

The configuration contains values or keys which are invalid:
secrets.system: <nil>
                ^-- expected array, but got null

level=error msg=Unable to instantiate configuration.
Error: validation failed

Verdict: PASS. Hydra v26.2.0 refuses to start with an empty system secret. The container exits immediately with code 1 and a clear error message. Silent use of empty secrets is not possible.

Init-Container Shim

Not required. Both Kratos and Hydra enforce secret presence at startup validation. Adding an init-container would be redundant. The existing compose structure (bare ${VAR} substitution, no :-default fallback) is sufficient to prevent silent startup with missing secrets.

SDK / ENCRYPTION_KEY

The SDK (@olympusoss/sdk) validates ENCRYPTION_KEY on first call to encrypt() or decrypt() not at module load. This is the deliberate design (see athena#111): eager module-level validation broke the container build stage because next build imports SDK-dependent API routes without ENCRYPTION_KEY set. Deferring validation to call time allows builds to succeed while still catching missing keys at runtime before any actual encryption is attempted.

Behavior with missing ENCRYPTION_KEY:

• Container starts and passes healthchecks (startup itself does not throw)
• First call to encrypt() or decrypt() throws with the exact error: [SDK] ENCRYPTION_KEY environment variable is required but not set. Generate a key with: openssl rand -base64 32
• Athena emits an ERROR-level log at startup if ENCRYPTION_KEY is absent (via src/instrumentation.ts)

Startup warning: Athena logs an ERROR on startup when ENCRYPTION_KEY is missing:

ERROR: ENCRYPTION_KEY is not set, SDK encryption operations will throw on first call.
Set ENCRYPTION_KEY before serving requests.

This log is the operator signal. It does not block startup, but it surfaces the misconfiguration before any user request fails. Monitor for this log pattern in production.

Deploy guard: deploy.yml sources ENCRYPTION_KEY from secrets.ENCRYPTION_KEY with no :-default fallback. If the GitHub Secret is absent, the container receives an empty string, instrumentation.ts emits the ERROR log, and the first settings API call throws. The deployment health check will detect the service is not responding to settings requests and the deployment is considered failed.

Verdict: DEFERRED VALIDATION. The SDK validates on first use, not on import. The startup ERROR log (Athena instrumentation.ts) is the early-warning mechanism. The build-stage constraint (athena#111) is why module-level validation was removed. See sdk/src/crypto.ts for the deferred validation implementation.

5. Findings Summary

6. Secret Generation Commands

Use these commands to generate each secret type. Never reuse secrets across environments. Never commit production secret values to source control.

Store all outputs directly in GitHub Actions Secrets. Never write them to files or shell history.

7. Understanding Pairwise Salts

CIAM_HYDRA_PAIRWISE_SALT and IAM_HYDRA_PAIRWISE_SALT control how Hydra computes the OIDC sub (subject) claim for each OAuth2 client.

What pairwise subjects do: When pairwise subject identifiers are enabled, Hydra derives a different sub value for the same user depending on which OAuth2 client they are authenticating to. For example, a user with internal identity abc-123 will appear as xG7q... to one client and mPw9... to a different client. This is an OIDC privacy mechanism that prevents clients from cross-correlating users by sub.

Why the salt matters: The pairwise derivation is HMAC(salt, user_id || client_id). Rotating the salt changes the sub claim for every user in every OAuth2 client. Any client that has stored the sub claim as a user identifier will fail to match existing records, effectively breaking all existing user associations in those clients.

Rotation rule: Do not rotate pairwise salts unless you have a coordinated migration plan for all affected OAuth2 clients. Rotating a Kratos cookie secret invalidates sessions (low operational impact). Rotating a pairwise salt breaks user identity associations in downstream systems (high operational impact).

8. Secret Rotation Procedures

Rotating Kratos Cookie or Cipher Secrets

Cookie secrets and cipher secrets can be added to the existing array rather than replacing it. Kratos supports multiple secrets simultaneously, it uses the first for new signatures/encryptions and accepts all others for verification/decryption. This enables zero-downtime rotation.

Procedure:

1. Generate a new secret: openssl rand -hex 32
2. In GitHub Actions Secrets, prepend the new value to the existing secret, Kratos expects a comma-separated list or a YAML array depending on config format
3. Run the deploy workflow, existing sessions remain valid while signed with either key
4. After one session TTL period (typically 1 hour for CIAM, configurable), remove the old value
5. Run deploy again, rotation complete

Effect: No sessions invalidated. Users remain logged in throughout the rotation.

Rotating Hydra System Secret

Hydra system secret rotation invalidates all outstanding OAuth2 tokens.

Procedure:

1. Generate a new secret: openssl rand -hex 32
2. Schedule a maintenance window if token invalidation will affect users
3. Update the GitHub Actions Secret
4. Run the deploy workflow, Hydra restarts with the new secret; all existing tokens become invalid
5. OAuth2 clients will receive token errors on their next API call and must re-authenticate

Effect: All outstanding access tokens, refresh tokens, and ID tokens are invalidated immediately.

Rotating ENCRYPTION_KEY (SDK)

Rotating ENCRYPTION_KEY requires a migration step. All encrypted values in the olympus database must be re-encrypted with the new key before the old key is removed.

Procedure:

1. Generate a new key: openssl rand -base64 32
2. Run the re-encryption migration script against the production database:

   DATABASE_URL=<prod_url> OLD_ENCRYPTION_KEY=<current> ENCRYPTION_KEY=<new> \
     bun run src/migrate-encryption-key.ts

3. The migration script re-encrypts all encrypted=true rows in all settings tables
4. After migration completes successfully, update the GitHub Actions Secret to the new value
5. Run the deploy workflow, containers restart with the new key

Effect: No service interruption if migration completes before deployment. Services with the old key and the migration running concurrently will produce decrypt errors for any settings written after migration starts, plan for a brief maintenance window or drain traffic before migrating.

Rotating Pairwise Salts

Do not rotate pairwise salts unless you have a coordinated migration plan. See Section 7 for the impact. Contact the team lead before rotating any pairwise salt in production.

9. ENCRYPTION_KEY Scope: Shared Across CIAM and IAM

ENCRYPTION_KEY has no domain prefix (CIAM_ or IAM_). This is intentional: it is a cross-domain SDK secret, shared by all four SDK-consuming containers, ciam-hera, iam-hera, ciam-athena, and iam-athena.

Current behavior

All four containers read the same single ENCRYPTION_KEY value. Any setting encrypted in ciam_settings and any setting encrypted in iam_settings use the same AES-256-GCM key. Encrypted values stored in one domain cannot be decrypted in the other domain (tables are separate), but the same cryptographic key material is in use across both.

Implications of the shared key

Operational rule

Do not interpret the absence of a domain prefix as a mistake. ENCRYPTION_KEY is shared by design for the current single-key architecture. If this changes in a future release, this document and deploy.yml will be updated simultaneously. Any PR that introduces domain-specific encryption keys must update this document as part of the PR checklist.

10. Maintenance Policy

This document must be updated whenever a new secret is added to deploy.yml. Any PR that introduces a new secrets.* reference in deploy.yml must include a corresponding update to this document as part of the PR checklist. This is a required step, not optional.

The accepted residual risk is that a CI linter is not in place to enforce this procedurally. The PR review process is the gate. This risk is rated Low given the current team size and review cadence.