Bulk audit log export

For compliance, security analytics, or just backups, you'll want to export the entire audit log periodically. Streaming approach to avoid memory issues.

Daily export script

#!/bin/bash
# /etc/cron.daily/audit-export
set -e

DATE=$(date +%Y-%m-%d)
TARGET="s3://olympus-audit-archive/$DATE.csv.gz"

psql -d olympus -c "
COPY (
  SELECT
    id, created_at, identity_id, actor_id, actor_type, event_type, outcome,
    source_ip, user_agent, metadata
  FROM security_audit
  WHERE created_at >= '$DATE' AND created_at < ('$DATE'::date + INTERVAL '1 day')
  ORDER BY created_at
) TO STDOUT WITH CSV HEADER
" | gzip | aws s3 cp - "$TARGET"

# Optional: write a manifest
echo "{ \"date\": \"$DATE\", \"size\": $(aws s3 ls $TARGET --summarize | tail -1 | awk '{print $3}'), \"rows\": ... }" > /var/lib/olympus/last-export.json

0 5 * * * deploy /etc/cron.daily/audit-export >> /var/log/olympus/audit-export.log 2>&1

Daily at 05:00. Old days compressed.

To Object Storage with Object Lock

For tamper-evident archival:

aws s3 cp - "$TARGET" \
  --object-lock-mode COMPLIANCE \
  --object-lock-retain-until-date $(date -d "+7 years" -I)

Once written, S3 won't allow modification for 7 years. Even root can't delete (in compliance mode).

Streaming to SIEM

If you have a SIEM that accepts JSONL:

psql -d olympus -c "
COPY (
  SELECT row_to_json(t) FROM (
    SELECT id, created_at, identity_id, actor_id, event_type, outcome, source_ip, metadata
    FROM security_audit
    WHERE created_at >= NOW() - INTERVAL '1 day'
  ) t
) TO STDOUT
" | curl -X POST $SPLUNK_HEC \
    -H "Authorization: Splunk $TOKEN" \
    -H "Content-Type: application/x-ndjson" \
    --data-binary @-

Real-time delivery to SIEM. Each line is an event.

Better: use Vector for continuous streaming, not batched.

Real-time via Debezium

For sub-second latency:

# debezium connector
name: olympus-audit-cdc
config:
  connector.class: io.debezium.connector.postgresql.PostgresConnector
  database.hostname: ciam-postgres
  database.dbname: olympus
  table.include.list: public.security_audit
  topic.prefix: olympus

Postgres logical replication → Kafka → consumers.

Heavier infra; better latency. For most: daily batch is enough.

Privacy considerations

Audit data is PII-heavy (IPs, identity_ids, user agents).

If exporting to external SIEM:

Encrypt in transit (TLS).
Encrypt at rest in destination.
Access controls in SIEM match Olympus's.
Retention matches your policy.

Document in your privacy policy.

Anonymized export

For analysts / researchers who don't need PII:

COPY (
  SELECT 
    event_type, outcome, 
    DATE_TRUNC('hour', created_at) AS hour,  -- aggregated, not exact
    digest(identity_id::text, 'sha256') AS hashed_user,
    NULL AS source_ip,
    NULL AS user_agent
  FROM security_audit
  WHERE created_at >= ...
) TO STDOUT WITH CSV;

Less granular but usable for stats.

Manifest

For each export, write a manifest:

{
  "date": "2026-05-13",
  "rows": 12345,
  "first_id": "01HQ-...",
  "last_id": "01HQ-...",
  "size_bytes": 1234567,
  "format": "csv.gz",
  "fields": ["id", "created_at", "identity_id", ...],
  "exported_at": "2026-05-14T05:00:00Z",
  "exporter_version": "v1"
}

Helps consumers know what they have.

Checksum

For integrity:

sha256sum exported.csv.gz > exported.csv.gz.sha256
aws s3 cp exported.csv.gz.sha256 s3://...

Verify before processing.

Restore from archive

If main DB loses data (unlikely but possible), restore audit log from archive:

aws s3 cp s3://olympus-audit-archive/$DATE.csv.gz - | gunzip | psql -c "COPY security_audit FROM STDIN WITH CSV HEADER"

Doesn't restore the present state, but provides historical record.

Periodic verification

# Weekly: verify last 7 days of exports exist
for i in $(seq 1 7); do
  DATE=$(date -d "-$i days" +%Y-%m-%d)
  aws s3 ls s3://olympus-audit-archive/$DATE.csv.gz || alert "Missing: $DATE"
done

Detects silently failing exports.

Compliance value

Auditors love regular exports:

Demonstrates retention.
Demonstrates immutability.
Demonstrates "we have what we say we have."

Make exports a documented practice.

Bulk audit log export

On this page