Grafana + Loki + Prometheus
Self-hosted observability with the Grafana stack
For operators who prefer to self-host their observability or want to keep PII off vendor SaaS, the Grafana stack (Loki + Prometheus + Grafana) is a complete alternative.
Stack components
| Tool | Purpose |
|---|---|
| Loki | Log aggregation. Indexes by labels, not full-text, cheap to run. |
| Prometheus | Metrics. Pull-based scraping. |
| Grafana | Dashboards. Queries Loki/Prometheus/etc. |
| Promtail | Log shipper from your VPS to Loki. |
| Alertmanager | Alert routing (email, PagerDuty, Slack). |
Deployment options
Self-hosted
Run the stack on a separate VPS (don't co-locate with Olympus, observability should survive an Olympus outage):
# observability/compose.yml
services:
loki:
image: grafana/loki:latest
volumes:
- loki-data:/loki
ports: ["3100:3100"]
prometheus:
image: prom/prometheus:latest
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- prom-data:/prometheus
ports: ["9090:9090"]
grafana:
image: grafana/grafana:latest
volumes:
- grafana-data:/var/lib/grafana
ports: ["3000:3000"]Grafana Cloud
Or use Grafana Cloud (managed). Free tier: 10k metrics, 50GB logs, 50GB traces.
On the Olympus host
Run Promtail to ship logs:
# Add to compose.prod.yml
promtail:
image: grafana/promtail:latest
volumes:
- ./promtail-config.yml:/etc/promtail/config.yml
- /var/log/containers:/var/log/containers:ro
command: -config.file=/etc/promtail/config.ymlpromtail-config.yml:
clients:
- url: https://your-loki/loki/api/v1/push
basic_auth:
username: <user>
password: <password>
scrape_configs:
- job_name: olympus
static_configs:
- targets: [localhost]
labels:
job: olympus
__path__: /var/log/containers/*.log
pipeline_stages:
- json:
expressions:
type: type
event: event
- labels:
type:
event:Metrics scraping
Kratos / Hydra expose Prometheus metrics at /metrics. Scrape:
# prometheus.yml
scrape_configs:
- job_name: ciam-kratos
static_configs:
- targets: [ciam-kratos:5001]
metrics_path: /metrics
- job_name: ciam-hydra
static_configs:
- targets: [ciam-hydra:5003]
metrics_path: /metrics
# ... per serviceDashboards
Import or build Grafana dashboards:
- Service health, uptime, request rate, latency from Prometheus.
- Auth flow funnel, Loki queries on
event:registration_*,event:login_*. - Brute-force activity,
event:lockout_*over time. - Database, Postgres exporter (
postgres_exporter) → Prometheus → dashboard.
Pre-made Grafana dashboards exist for Ory Kratos/Hydra; check grafana.com/grafana/dashboards for the latest.
Alertmanager
# alert-rules.yml
groups:
- name: olympus
rules:
- alert: HighFailedLoginRate
expr: rate(olympus_login_attempts_failed[5m]) > 10
annotations:
summary: "Login failure rate > 10/sec"
- alert: LockoutsApplied
expr: increase(olympus_lockouts_total[1h]) > 100
annotations:
summary: "Many lockouts in last hour, possible attack"Route to Slack, PagerDuty, email.
Cost
Self-hosted: cost of the observability VPS only. ~$10-20/mo on Hetzner.
Grafana Cloud: free tier covers small Olympus deployments.