Psyduck V2 - A Data Analyst's Pokemon Experimental Tool

Configuration

Copy the .env.example file to .env and fill in the required environment variables.
Copy config/example.config.json to config/config.json and fill in the required configurations.
Copy dashboard/dashboard_config.example.json to dashboard/dashboard_config.json and customize your dashboard settings (map name, navbar links, etc.).
Ensure in Golbat you have the webhook with /webhook. Example: 127.0.0.1:5050/webhook

Docker Setup

Install Docker Desktop
docker network create psyduckv2_network
docker compose up -d

If you want to run python natively:

Install Python 3.11
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python3 psyduckv2.py

This option requires you to have a MySQL database running on your local/external machine and manually set the DB_HOST environment variable to the correct IP address.

Change DB_PORT to 3307 if you are using the docker-compose setup and didn't touch the docker-compose.yml file.

Docker stand alone container options:

Run docker compose up psyduckv2_db to start the database container
Run docker compose up psyduckv2_pma to start the phpmyadmin container
Run docker compose up psyduckv2_app to start the app container

Redis Tuning (Recommended for Production)

Redis doesn’t run well on VPS setups that rely on virtual memory. For large-scale webhook feeding into PsyduckV2, it needs guaranteed physical RAM to perform reliably.

To ensure Redis performs well under high load, apply the following system-level and Redis config optimizations:

System-Level Optimizations

Add these to your system's startup or provisioning scripts:

# Allow memory overcommit (recommended for Redis)
sysctl -w vm.overcommit_memory=1

# Reduce swapping tendency
sysctl -w vm.swappiness=1

# You can persist these directly in:
/etc/sysctl.conf
# Run this after adding it to the sysctl.conf at the end:
sudo sysctl -p

# Disable Transparent Huge Pages (causes latency spikes)
echo never > /sys/kernel/mm/transparent_hugepage/enabled

Redis Configuration:
# Security and access
protected-mode yes              # Keep enabled unless you're running behind strict network rules
bind 127.0.0.1                   # To allow connections from outside you can do 0.0.0.0 for all or set a specific IP you want to allow: bind 192.168.1.100, 127.0.0.1, etc
requirepass YOUR_PASSWORD_HERE # Always set a password in production!

# Memory handling
maxmemory 8gb                 # Optional: Limit memory usage

# TCP stack tuning
tcp-keepalive 0                # Keep connections alive (0 = OS default)
tcp-backlog 65536             # Allow more simultaneous TCP connections

# Persistence
appendonly yes                 # Use AOF (Append Only File) persistence
appendfsync everysec           # Default; sync every second (can be heavy under high writes)
# appendfsync no              # Consider this if using `save` + RDB-only strategy
# You can swap appendfsync no and use:
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 512mb

# Save snapshots
save 900 1                     # Save if at least 1 key changed in 15 min
save 300 10                    # Save if at least 10 keys changed in 5 min
save 60 10000                  # Save if at least 10k keys changed in 1 min

# You'll need to debug your own redis and adjust these as needed.

Development

Database migrations are handled by Alembic

Create an alembic file to create the migration:

alembic revision -m "my_cool_migration_name"

Run the migration locally before pushing to the repository:

alembic upgrade head

This command will create a new migration file in the alembic/versions/ directory. Make sure to commit the migration file to the repository.

Testing

test/ holds a self-contained test stack and scripts that replay real failure cases (a database outage mid-flush, Redis restarting under the app, a crash during leader election, a slow Redis under heavy load) and benchmark the ingest path. Each script prints PASS/FAIL and saves its numbers in test/results/.

Nothing here touches your own PsyduckV2, Redis or database. The stack is a separate Docker Compose project (psyduck-test) with its own containers and ports, built from your working copy:

Service	What it is	Port on your machine
`app`	PsyduckV2 built from this repo (`test/app.env`, `test/config.json`)	15050
`redis`	Redis 7, password `testpass`, no persistence (like in-memory mode)	16379
`mariadb`	MariaDB 10.11, user/password `psyduck`	13306
`fake-koji`	Serves 12 synthetic geofences, so no Koji is needed	18088

Events come from test/feeder.py, a stand-in for Golbat: one POST per second with every event since the last one, no timeout and no retry, just like Golbat's webhook sender.

Requirements

Docker (Docker Desktop on Windows/macOS) running.
The project's Python environment with requirements.txt installed. The scripts run on your machine and talk to the containers.

Commands below use python; on Windows with the project venv that is .venv/Scripts/python, on Linux/macOS .venv/bin/python.

Start the stack

docker compose -f test/docker-compose.yml build app      # after every code change you want to test
cd test
python -c "import harness as h; h.down(); h.up()"        # fresh stack (empty Redis + DB), waits until the app is ready

h.down() removes the test containers and their volumes, so every run starts clean. Stop everything with python -c "import harness as h; h.down()".

Failure scenarios

Run from test/ with the stack up. Each one leaves the stack running for the next.

Script	Takes	Checks
`python scenario_flush_requeue.py`	~5 min	MariaDB goes down while buffers flush: every event still reaches SQL exactly once, and a batch stranded by a crash is recovered
`python scenario_restore_guard.py`	~15 min	Restarts, crashes and Redis restarts never rewind live counters, resurrect deleted keys, or count anything twice; a Redis that lost its data gets it back
`python scenario_leader_failover.py`	~3 min	The app restarted right after a crash still becomes leader and keeps flushing
`python scenario_backpressure.py`	~4 min	Under a slow Redis the app sheds load with 503s instead of growing memory without bound, then recovers

Benchmarks

bench_ingest.py replays webhook traffic at a rate you choose, so start from your own feed's rate. Your running PsyduckV2 logs it once a minute in the 🩺 ingest line:

🩺 ingest | ... | last 60s: accepted 28,200 ev in 60 POSTs, ...

Accepted events ÷ seconds = your rate, here 28,200 / 60 = 470 messages/s. Then benchmark your normal load, and multiples of it to see how much headroom you have:

python bench_ingest.py --rate <your_rate> --mode realistic --label normal
python bench_ingest.py --rate <your_rate x 2> --mode realistic --label double
python bench_ingest.py --rate <your_rate x 5> --mode realistic --label five_times

--rate: messages per second.
--mode realistic: a production-like mix - about half of the Pokémon messages are non-encounter sightings that get filtered out, plus re-sent encounters. exact (the default) sends only valid, unique events, the worst case per message.
--seconds (default 30) and --label (any name; the results go to test/results/bench_ingest_<label>.json).

Each run reports the app's CPU per 1,000 messages, POST latency (p50/p95/max) and Redis commands and time per message. Golbat posts once a second, so as long as the slowest POST stays well under 1 s the app keeps up; past that, a backlog builds and it is time for more UVICORN_WORKERS. To compare code changes, run the same rate and mode before and after (rebuilding the image in between) under two labels.

Focused checks

Script	Checks
`python check_transaction_autocommit.py`	`transaction()` really runs its block as one transaction
`python check_counter_retrieval.py`	Counter reads (generated key names) return exactly what `KEYS` used to
`python smoke_counter_api.py`	Every counter endpoint answers, and no `KEYS` reaches Redis (feed a little first, e.g. one benchmark run)
`python check_geofence_index.py`	The geofence index picks the same area as a linear scan, and how much faster it is
`python check_incremental_backup.py`	Incremental Redis → MySQL backups leave MySQL identical to Redis, across time zones (stops the test app while it runs)
`python check_reencode_hashes.py`	`reencode_hashes` shrinks closed counter hashes without changing a value
`python check_repair_weekly_totals.py`	`repair_weekly_totals` restores exactly what a mid-week Redis wipe took

The check_* scripts use Redis databases 2–3 and scratch tables of the test stack only. Run every script from test/: it keeps your own .env from redirecting them anywhere else.

Dashboard

The dashboard is a separate Dash-based analytics application that provides visual analytics for Pokemon GO events.

Running the Dashboard

# Start the dashboard (runs on port 8050 by default)
python run_dashboard.py

Important: The dashboard requires the main application (psyduckv2.py) to be running, as it fetches data from the API endpoints.

Features

Home: Landing page with overview statistics
Pokemon: Pokemon spawn analytics with IV distributions, timeseries charts, shiny rates, and heatmaps
Raids: Raid occurrence analytics by level and Pokemon
Quests: Quest analytics by reward type and pokestop
Invasions: Team Rocket invasion analytics by type and location

Multi-Language Support

The dashboard supports multiple languages:

English (en)
Portuguese (pt)
German (de)
French (fr)

Switch languages using the flag dropdown in the navbar.

Background Tasks

The dashboard runs background tasks to periodically fetch and cache data from the API:

Daily tasks (hourly refresh): Areas, Pokestops, Pokemon, Raids, Invasions, Quests
Alltime tasks (daily refresh): Historical aggregations for all event types

On startup, daily tasks run first, followed by "alltime" tasks after a 30-second delay to avoid overwhelming the API.

Dashboard Customization

The dashboard can be customized via dashboard/dashboard_config.json:

{
  "map_name": "YourMapName",
  "icon_url": "/assets/custom_images/your_icon.png",
  "custom_navbar_links": [
    {
      "name": "API Docs",
      "url": "https://your-api-docs-url.com/",
      "icon": "bi bi-book-fill",
      "external": true
    }
  ]
}

map_name: Displayed as "PsyduckV2 {map_name}" in navbar and page titles
icon_url: Custom icon displayed between "PsyduckV2" and map name (place images in dashboard/assets/custom_images/)
custom_navbar_links: Additional navbar links with Bootstrap Icons

Custom Images: Place your custom images in dashboard/assets/custom_images/ (gitignored). This directory is ignored by git to keep your customizations private.

Icon Caching: The dashboard automatically caches Pokemon and reward icons locally on startup for faster loading. Icons are downloaded from the WatWowMap UICONS repository and stored in dashboard/assets/ subdirectories (gitignored). First startup may take 2-5 minutes to download all icons; subsequent starts are instant.

Configuration Reference (`config/config.json`)

`SQL`

Controls which event types are persisted to MySQL long-term storage.

Key	Default	Description
`store_sql_pokemon_aggregation`	`true`	Write Pokemon IV events to partitioned SQL tables
`store_sql_pokemon_shiny`	`true`	Write shiny username events to SQL
`store_sql_raid_aggregation`	`true`	Write raid events to SQL
`store_sql_quest_aggregation`	`true`	Write quest events to SQL
`store_sql_invasion_aggregation`	`true`	Write invasion events to SQL
`store_sql_weather`	`true`	Write Golbat `weather` webhooks (one row per S2 cell weather change) to `weather_daily_events`. Golbat must send the `weather` webhook type

`CLEAN_SQL`

Retention for partitioned SQL tables. Old daily/monthly partitions are automatically dropped.

Key	Default	Description
`clean_pokemon_older_than_x_days`	`15`	Drop Pokemon IV partitions older than N days
`clean_raid_older_than_x_days`	`15`	Drop raid partitions older than N days
`clean_quest_older_than_x_days`	`15`	Drop quest partitions older than N days
`clean_invasion_older_than_x_days`	`15`	Drop invasion partitions older than N days
`clean_weather_older_than_x_days`	`90`	Drop weather partitions older than N days
`clean_pokemon_shiny_older_than_x_months`	`3`	Drop shiny partitions older than N months
`partition_drop_batch`	`2`	Expired partitions dropped per `ALTER TABLE`. Small batches keep each table lock short, e.g. after lowering a retention
`partition_drop_pause_seconds`	`10`	Pause between those `ALTER TABLE`s so inserts get through

`IN-MEMORY`

Controls Redis timeseries storage and the optional MySQL backup/restore feature.

Key	Default	Description
`store_pokemon_timeseries`	`true`	Store Pokemon timeseries in Redis
`store_pokemon_tth_timeseries`	`true`	Store Pokemon TTH timeseries in Redis
`store_raids_timeseries`	`true`	Store raid timeseries in Redis
`store_invasions_timeseries`	`true`	Store invasion timeseries in Redis
`store_quests_timeseries`	`true`	Store quest timeseries in Redis
`store_max_battles`	`true`	Count Golbat `max_battle` webhooks in Redis (daily/weekly, each battle once). Golbat must send the `max_battle` webhook type
`store_tappables`	`true`	Count Golbat `tappable` webhooks in Redis (hourly/daily/weekly, each tappable once). Golbat must send the `tappable` webhook type
`REDIS_MYSQL_BACKUPS`	`false`	Recommended for production. Periodically backs up Redis counter and timeseries keys to MySQL, then restores them on startup. This is the optimal way to run PsyduckV2 — it allows Redis to operate without AOF/RDB persistence (lower I/O, faster restarts) while guaranteeing no data loss across restarts.
`backup_interval_seconds`	`3600`	How often (seconds) the backup cycle runs
`redis_restore_timeout_seconds`	`600`	How long follower workers wait for the leader to finish restoring Redis data on startup. Increase if your Redis restore takes longer than 10 minutes. Only applies when `REDIS_MYSQL_BACKUPS` is enabled.
`full_backup_interval_hours`	`24`	Backups normally only rewrite keys that can have changed since the previous one; every N hours (and after any restore) a full backup rewrites everything. `0` = always full

`REDIS`

Key	Default	Description
`redis_connections`	`600`	Max Redis connection pool size. With multiple uvicorn workers set this to at least `UVICORN_WORKERS × (pool / 2) + 50`.

`flusher`

Controls how buffered events are batch-inserted into MySQL.

Key	Default	Description
`{type}_flush_interval`	`15`	The leader flushes each buffer to MySQL every N seconds (`weather_flush_interval` defaults to `60`)
`{type}_max_threshold`	`10000`	No longer used: webhook requests never flush, so buffers only drain on the interval above

`WEBHOOK`

Protects the webhook receiver when it falls behind. Limits apply per uvicorn worker.

Key	Default	Description
`max_inflight_events`	`20000`	While this many events are still being processed, new webhook batches are refused with HTTP 503 and dropped (Golbat does not retry). Keeps memory bounded when Redis or MySQL stall. ~20000 is about 40 s of a busy feed; raise it with your volume. `0` = no limit
`max_rss_mb`	`0`	Also refuse batches while the process uses more than this many MB. A last-resort cap; set it well above normal usage (see the `🩺 ingest` log line), since Python rarely returns memory to the OS. `0` = off
`abort_on_disconnect`	`true`	Stop processing a batch whose sender has hung up
`health_log_interval_seconds`	`60`	How often the `🩺 ingest` line is logged (in-flight events, accepted/shed/aborted, event-loop lag, memory). It is a WARNING when anything was shed or the app lagged

`weather`

Weather sync from Golbat's own weather table (needs the GOLBAT_DB_* connection). Golbat only sends a weather webhook when a cell's weather changes, so without it a cell has no data until its first change, and changes missed while PsyduckV2 was down are lost. Each pass writes a row for every cell overlapping an area that has none yet, or whose stored weather differs from Golbat's, timed at the start of the hour Golbat last read the cell (weather only changes on the hour).

Key	Default	Description
`golbat_sync_minutes`	`15`	How often the sync runs. `0` = off
`golbat_sync_max_age_hours`	`2`	Skip cells Golbat has not refreshed for this long (no scanner reads them, so their stored weather is a guess). Golbat refreshes a cell about once an hour. `0` = trust every cell

`geofences`

Key	Default	Description
`expire_cache_seconds`	`3600`	How long geofence data is cached in Redis
`refresh_cache_seconds`	`3500`	How often the background task re-fetches geofences from Koji

`golbat_pokestops`

Key	Default	Description
`pokestop_cache_expiry_seconds`	`86400`	How long pokestop data is cached in Redis
`pokestop_refresh_interval_seconds`	`86300`	How often the background task syncs pokestops from Golbat's DB

`retention_hours`

How long Redis keeps hourly counter keys and timeseries keys. Set to 0 to keep all data indefinitely.

Key	Default	Description
`timeseries_pokemon`	`72`	Pokemon timeseries retention (hours)
`tth_timeseries_pokemon`	`72`	Pokemon TTH timeseries retention (hours)
`timeseries_raid`	`72`	Raid timeseries retention (hours)
`timeseries_invasion`	`72`	Invasion timeseries retention (hours)
`timeseries_quest`	`72`	Quest timeseries retention (hours)
`counter_pokemon_hourly`	`168`	Hourly Pokemon counter key retention (hours)
`counter_tth_pokemon_hourly`	`168`	Hourly Pokemon TTH counter key retention (hours)
`counter_raid_hourly`	`168`	Hourly raid counter key retention (hours)
`counter_invasion_hourly`	`168`	Hourly invasion counter key retention (hours)
`counter_quest_hourly`	`168`	Hourly quest counter key retention (hours)
`counter_tappable_hourly`	`168`	Hourly tappable counter key retention (hours)

`retention_days`

How long Redis keeps daily counter keys (one key per day per area, aggregated from hourly keys). Set to 0 to keep all data indefinitely.

Key	Default	Description
`counter_pokemon_daily`	`30`	Daily Pokemon counter key retention (days)
`counter_tth_pokemon_daily`	`30`	Daily Pokemon TTH counter key retention (days)
`counter_raid_daily`	`30`	Daily raid counter key retention (days)
`counter_invasion_daily`	`30`	Daily invasion counter key retention (days)
`counter_quest_daily`	`30`	Daily quest counter key retention (days)
`counter_max_battle_daily`	`30`	Daily max battle counter key retention (days)
`counter_tappable_daily`	`30`	Daily tappable counter key retention (days)

`CLEAN_REDIS_TS`

Key	Default	Description
`cleanup_interval_seconds`	`1800`	How often the background task scans and removes expired Redis keys

API Documentation

Available here.

Name		Name	Last commit message	Last commit date
.vscode		.vscode
alembic		alembic
config		config
dashboard		dashboard
examples		examples
my_redis		my_redis
server_fastapi		server_fastapi
sql		sql
test		test
utils		utils
webhook		webhook
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
Dockerfile		Dockerfile
README.md		README.md
alembic.ini		alembic.ini
config.py		config.py
example.docker-compose.yml		example.docker-compose.yml
psyduckv2.py		psyduckv2.py
requirements.txt		requirements.txt
run_dashboard.py		run_dashboard.py

Latest commit

History

Folders and files

Repository files navigation

Psyduck V2 - A Data Analyst's Pokemon Experimental Tool

Configuration

Docker Setup

If you want to run python natively:

Docker stand alone container options:

Redis Tuning (Recommended for Production)

System-Level Optimizations

Development

Testing

Requirements

Start the stack

Failure scenarios

Benchmarks

Focused checks

Dashboard

Running the Dashboard

Features

Multi-Language Support

Background Tasks

Dashboard Customization

Configuration Reference (config/config.json)

SQL

CLEAN_SQL

IN-MEMORY

REDIS

flusher

WEBHOOK

weather

geofences

golbat_pokestops

retention_hours

retention_days

CLEAN_REDIS_TS

API Documentation

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages