Break-glass admin access

Scenario: your SSO provider is down. Or Kratos's DB is corrupt. Normal admin access doesn't work. You need to get back in.

A "break-glass" account is the answer: a high-privilege identity, rarely used, with credentials stored offline. Open it only in true emergencies.

Design

Account specifics

Dedicated identity, separate from any normal user.
Email: breakglass@your-domain.com (or similar, not associated with any individual).
Role: highest privilege (root admin).
Password: 30+ char random, generated once.
MFA: separate authenticator on a dedicated device.

Storage

Password and MFA seed: in a sealed envelope in a physical safe.
OR: in your team's 1Password emergency access vault.
OR: split secret (Shamir's) across N team members, M needed.

NOT stored in your normal secrets vault, that's compromised in some scenarios where you need break-glass.

Provisioning

# One-time setup
NEW_PASSWORD=$(openssl rand -base64 30)
curl -X POST $KRATOS_ADMIN/admin/identities \
  -H "Content-Type: application/json" \
  -d "{
    \"schema_id\": \"default\",
    \"traits\": {
      \"email\": \"breakglass@your-domain.com\",
      \"role\": \"admin\"
    },
    \"credentials\": {
      \"password\": { \"config\": { \"password\": \"$NEW_PASSWORD\" } }
    }
  }"

# Print password (only once):
echo "Break-glass password: $NEW_PASSWORD"
# Save to safe / 1Password.

# Enroll TOTP via Athena UI (manual)
# Save QR / seed.

Detection

Every break-glass login MUST be alarming:

-- In your audit hook:
INSERT INTO break_glass_events (identity_id, ip, user_agent, created_at)
VALUES ($identity_id, $ip, $ua, NOW());

Trigger:

Alert to all admins: "Break-glass account just signed in from [IP]. Confirm this is authorized."
Page on-call.
Slack/Teams security channel.

The signal: this should NEVER happen by accident. If it happens, you have a real incident, either expected use or unauthorized access.

Usage runbook

# Break-glass usage

When normal SSO is unavailable AND you need emergency admin access:

1. Get authorization. At least 2 senior people must agree this is needed.
2. Retrieve credentials from safe / 1Password.
3. Log in.
4. Document:
   - Who authorized.
   - What you did.
   - Time you logged in / logged out.
5. After resolution:
   - Rotate password.
   - Re-seal credentials.
   - Post-mortem within 48h.

Print and post in the on-call runbook.

Rotation

After every use:

New password.
New MFA seed.
Re-seal.

This way, even if the previous credentials were observed during use, they're invalid.

Quarterly: rotate even if unused (in case of unknown leak).

Multiple accounts

For larger teams, multiple break-glass accounts:

One per region / cluster.
One per service category.

Each with its own credentials, owners, runbook.

Don't use for non-emergency

The break-glass is for emergencies. Don't:

Use for routine admin (use your normal admin account).
Test by logging in.
Share with anyone not in the authorized list.

Each use should be a notable event.

Network isolation

Some teams restrict break-glass login to a specific IP (jump host, VPN). Caddy:

@breakglass path /admin/* {
  remote_ip 10.0.0.0/8
}

If the attacker doesn't have network access, the password alone isn't enough.

Logging and accountability

Beyond audit:

Every action of the break-glass account is logged at maximum verbosity.
All sessions recorded (if possible, ttyrec for SSH, screen recording for web).
Reviewed by security team within 24h of use.

What about offline emergencies?

If Kratos / DB is completely down, even break-glass login fails. The fallback is:

Database superuser (postgres user) via direct DB access.
Modify identities table directly, set password hash.

This requires:

Postgres password.
Access to the host.

Store these analogously offline. Document the SQL to "raise an emergency admin" in your runbook.

Compliance angle

Auditors love break-glass. It demonstrates:

You've planned for emergencies.
High-privilege access is controlled and logged.
Rotation is in place.

Document the process. Audit log entries during quarterly drills.

Testing

Quarterly drill:

Pretend SSO is broken.
Retrieve credentials.
Log in.
Log out.
Verify the alert fired.
Rotate.

Practice catches gaps (forgotten safe combination, MFA device dead battery, etc.).

Break-glass admin access

On this page