sentry-automq

Self-hosted Sentry with the Kafka broker replaced by AutoMQ — a Kafka-protocol-compatible broker that stores stream data in S3-compatible object storage instead of local disk.

This is a minimal, compose-only variant of getsentry/self-hosted. The full Sentry stack (Sentry, Snuba, ClickHouse, PostgreSQL, relay, nginx, symbolicator, taskbroker, …) runs exactly as upstream — only the kafka service changes. It keeps the same service name, so every component talks to kafka:9092 unchanged.

based on: getsentry/self-hosted 26.8.0 (2026-08-17) — service list and
          consumer commands match that tag exactly, plus seaweedfs-tier
changes:  kafka → automqinc/automq:1.7.2 (external S3/R2 only),
          seaweedfs cloud-tiered to R2 via seaweedfs-tier sidecar,
          volumes made non-external (no install script needed),
          per-service CPU/RAM limits in docker-compose.limits.yml
license:  see LICENSE.md (upstream self-hosted license)

Why AutoMQ

The bundled Kafka keeps its data in a local volume with a 3-hour retention just to avoid filling the disk. AutoMQ writes the same data to object storage; local disk only holds a bounded WAL + block cache. Point it at external object storage (Oracle Cloud Object Storage, AWS S3, …) and the message bus uses essentially zero local disk.

ClickHouse (event data, default 90-day retention) remains the largest local disk consumer. This swap addresses the message bus only.

Quick start

Requires Docker Compose v2. COMPOSE_FILE in .env merges the resource limits, so plain docker compose commands need no -f flags.

cp .env.example .env

Then edit .env before running anything. These have no defaults, and compose refuses to start until they are set — you get an interpolation error naming the variable, not a running stack with placeholder credentials:

• AUTOMQ_S3_ENDPOINT, AUTOMQ_S3_ACCESS_KEY, AUTOMQ_S3_SECRET_KEY
• R2_ENDPOINT, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY

Create the three buckets first (automq-data, automq-ops, sentry-seaweedfs-tier) — nothing in this stack creates them.

# 1. start the backing services (builds sentry/clickhouse/cron images on first run)
docker compose up -d postgres redis kafka memcached pgbouncer clickhouse

# 2. create/migrate the database (first boot only, or after upgrades)
docker compose run --rm web upgrade

# 3. start everything
docker compose up -d --wait

Sentry is then at http://<host>:9000 (SENTRY_BIND in .env). The first run creates the admin user in step 2.

Object storage

Everything durable goes to S3-compatible object storage. There is no bundled MinIO — set the endpoints in .env and create the buckets yourself first.

AutoMQ (the message bus)

AUTOMQ_S3_ENDPOINT=https://<ACCOUNT_ID>.r2.cloudflarestorage.com
AUTOMQ_S3_REGION=auto
AUTOMQ_S3_ACCESS_KEY=<key>
AUTOMQ_S3_SECRET_KEY=<secret>

Create both buckets before first boot — AutoMQ does not create them and fails with NoSuchBucketException:

• automq-data (or set AUTOMQ_S3_DATA_BUCKET)
• automq-ops (or set AUTOMQ_S3_OPS_BUCKET)

R2 requires region=auto. For Oracle Cloud Object Storage use the S3-compatible endpoint (...compat.objectstorage...) with its real region name and an S3 API key from your customer secret key.

SeaweedFS cloud tier (nodestore + profiles)

SeaweedFS stays the local S3 endpoint Sentry talks to at seaweedfs:8333 — it backs SENTRY_NODESTORE and filestore.profiles. The nodestore is a hot path (read and written per event), so it is deliberately not pointed straight at R2. Instead the seaweedfs-tier sidecar periodically pushes sealed volumes out to R2:

R2_ENDPOINT=https://<ACCOUNT_ID>.r2.cloudflarestorage.com
R2_ACCESS_KEY_ID=<key>
R2_SECRET_ACCESS_KEY=<secret>
R2_TIER_BUCKET=sentry-seaweedfs-tier

Create sentry-seaweedfs-tier yourself; it must not be one of the AutoMQ buckets. Backend config lives in seaweedfs/master.toml, with every value overridden from .env (SeaweedFS maps config keys to WEED_-prefixed env vars with dots replaced by underscores).

Three things that are easy to get wrong:

• region must be auto. R2 aliases only empty and us-east-1 to it — the upstream SeaweedFS scaffold's us-east-2 will not work.
• storage_class must be STANDARD. R2 has no infrequent-access class, and the upstream scaffold defaults to STANDARD_IA.
• Never leave the credentials empty in master.toml. The seaweedfs container already sets AWS_ACCESS_KEY_ID=sentry for its own S3 gateway; an empty key lets the AWS SDK credential chain present that to R2, failing with an auth error that looks nothing like a config mistake.

What this buys: only sealed (full, quiet) volumes move, so active data stays local and reads on a tiered volume proxy to R2. It bounds long-term disk growth rather than shrinking the working set. volume.tier.compact runs in the same sidecar loop and is not optional — deleted files do not reclaim remote space, so without it R2 usage grows forever under Sentry's retention deletes.

The sidecar sleeps before its first run, so nothing is tiered for the first SEAWEEDFS_TIER_INTERVAL (6h by default) after up — deliberate, so tiering does not compete with an initial backfill. Quiet logs early on are expected, not a fault.

After changing seaweedfs/master.toml you must restart the container, or tiering fails with "no storage backends configured". Verify manually before trusting the cron:

docker compose exec seaweedfs sh -c \
  'echo "lock;volume.tier.upload -dest s3.r2 -fullPercent=95 -quietFor=1m;unlock" | weed shell -master=localhost:9333'

feature-complete profile

.env.example ships COMPOSE_PROFILES=feature-complete, which adds vroom, uptime-checker, launchpad and the transactions/metrics/spans consumers (29 extra containers). Drop it for a smaller stack.

Resource limits

docker-compose.limits.yml sets a CPU weight and RAM ceiling on every service. It is merged automatically via COMPOSE_FILE in .env; delete that line to run with no limits at all.

Defaults are sized for 6 vCPU / 24 GB running feature-complete (58 containers). Be aware of the shape of that host:

• At feature-complete the stack idles at roughly 18–19 GB, leaving ~5 GB of headroom on 24 GB for ClickHouse to burst into during queries and merges.
• Both cpus and mem_limit ceilings intentionally sum to more than the host has. They are ceilings, not reservations — the job is to stop one service monopolising the box, not to partition it. cpu_shares decides priority when the CPU is genuinely saturated.
• Upstream's stated minimum is 4 CPU / 16 GB for the default profile; feature-complete adds 29 containers on top of that. 6 cores handles it, but CPU is still the tighter of the two resources — if the stack feels slow before RAM is exhausted, add cores rather than memory.

Going smaller. On 4 vCPU the stack still runs, but CPU becomes the binding constraint. Lower these together — each pairs a container cap with the in-process setting that drives it:

SENTRY_TASKWORKER_CONCURRENCY=2   # with TASKWORKER_MEM=1536m
LAUNCHPAD_TASKWORKER_CONCURRENCY=2
WEB_UWSGI_WORKERS=2               # with WEB_MEM=2g
CLICKHOUSE_MEM=4g                 # with CLICKHOUSE_MAX_MEMORY_BYTES=2147483648

and halve the *_CPUS ceilings. That brings steady state to roughly 16–17 GB.

Going bigger. On 8 vCPU / 28 GB, raise CLICKHOUSE_MEM=6g with CLICKHOUSE_MAX_MEMORY_BYTES=4294967296 and lift the *_CPUS ceilings proportionally — ClickHouse is the component that most repays extra headroom.

The numbers are estimates, not measurements. Take a real baseline and set each limit to about 2× observed peak RSS:

docker stats --no-stream --format '{{.Name}}\t{{.MemUsage}}\t{{.CPUPerc}}' | sort -k2 -h

mem_limit is a hard OOM-kill, so where a limit has a matching in-process setting, raise both or neither:

Limit	Must move with
KAFKA_MEM	AUTOMQ_HEAP_OPTS (heap + MaxDirectMemorySize)
CLICKHOUSE_MEM	CLICKHOUSE_MAX_MEMORY_BYTES
REDIS_MEM	REDIS_MAXMEMORY
WEB_MEM	WEB_UWSGI_WORKERS
TASKWORKER_MEM	SENTRY_TASKWORKER_CONCURRENCY

Check nothing is being killed after a soak:

for c in $(docker compose ps -q); do
  docker inspect --format '{{.Name}} oom={{.State.OOMKilled}} restarts={{.RestartCount}}' $c
done | grep -v 'oom=false restarts=0'

Cluster id

.env.example ships a valid default. Regenerate for a fresh cluster:

docker run --rm automqinc/automq:1.7.2 /bin/bash -c "/opt/automq/kafka/bin/kafka-storage.sh random-uuid"

Operations

• Logs: docker compose logs -f kafka (broker), docker compose logs -f web
• Stop: docker compose down
• Full reset (wipes all data volumes): docker compose down -v
• Tiering activity: docker compose logs -f seaweedfs-tier
• Force a tier run now, without waiting for the interval: docker compose exec seaweedfs sh -c 'echo "lock;volume.tier.upload -dest s3.r2 -fullPercent=95 -quietFor=1h;unlock" | weed shell -master=localhost:9333'

Before removing the R2 tier backend, run volume.tier.download for any already-tiered volumes. Dropping the backend first makes that data unreachable — it is the one step here that is not a clean revert.

Upgrading

The *_IMAGE vars in .env are pinned to a tagged release (26.8.0), not nightly. Bump all eight to the same new tag together — a mixed set is not a combination upstream tests — then:

docker compose pull
docker compose run --rm web upgrade
docker compose up -d --wait

Read the release notes between your current tag and the target before upgrading: self-hosted regularly adds and removes containers, and a removal will leave orphans running until you reconcile both docker-compose.yml and docker-compose.limits.yml.

Diff against the release tag, not master. master carries unreleased changes — as of 26.8.0 it had already bumped ClickHouse 25.3 → 25.8 and removed process-segments. Adopting those would put you on a combination upstream has not shipped, which is the same problem as tracking nightly. Use raw.githubusercontent.com/getsentry/self-hosted/<tag>/docker-compose.yml.

To track changes to the compose file itself, diff docker-compose.yml against getsentry/self-hosted and carry forward this repo's own divergence: the kafka block, the seaweedfs cloud-tier env and mount, and the seaweedfs-tier service.

docker-compose.limits.yml needs no reconciliation — it only sets resource keys on services by name, so upstream changes to a service's command or environment merge cleanly. It does need a new entry whenever upstream adds a service.

Verified

• AutoMQ 1.7.2 boots as a single-node KRaft broker with the exact override set in this compose and reports healthy.
• Kafka protocol end-to-end: topic create, produce, consume (__consumer_offsets present) — the same calls Snuba's consumers and Sentry's producers make.
• docker compose config validates the merged stack (58 services): every service carries a mem_limit, ClickHouse keeps both config.d mounts, and web/redis keep their commands and healthchecks through the override.

Not yet verified

The stack has not been booted against real R2 from this checkout. Two things to confirm on first run:

• SeaweedFS loads [storage.backend.s3.r2] from /etc/seaweedfs/master.toml — the failure mode is "no storage backends configured" at tier time. Use the manual volume.tier.upload above before trusting the sidecar.
• UWSGI_WORKERS actually lowers the worker count. Sentry passes SENTRY_WEB_OPTIONS programmatically, which may win over the env var. Check with docker compose logs web | grep -c 'spawned uWSGI worker' — if it prints 3 rather than 2, set "workers": 2 in sentry/sentry.conf.py instead.