GeoRiva

WARNING: This project is in active development and breaking changes may occur without backwards compatibility. Until we reach a stable release, we recommend treating the codebase as not production-ready. We welcome contributions and feedback on the architecture, but please be aware that APIs, data models, and plugin interfaces may change as we iterate.

Geospatial Raster Ingestion, Visualization & Analysis

GeoRiva is a geospatial backend platform for automated ingestion, processing, visualization, and analysis of gridded raster data. Built on Django/Wagtail, it provides a plugin-driven architecture for pulling data from diverse sources, serving it through modern standards-compliant APIs, and enabling analytical workflows on top of it.

Status: Active development — core ingestion, the two-tier STAC data model, the derivation engine, multi-tenancy, STAC API, tile serving (Titiler + Martin), WMTS, and the analysis modules are built; some areas (EDR data-retrieval plane, generic analysis-plugin framework) are still in progress. See the Architecture Design Document for the as-built design and open discussion points.

---

What It Does

• Ingest gridded data from multiple sources via plugin apps or by dropping files into a MinIO directory
• Process data into cloud-optimized COGs through an async Celery pipeline
• Index everything as STAC-compliant Catalogs, Collections, and Items with time-series optimized storage
• Serve tiles and data through a STAC API, Titiler and Martin tile servers, and on-demand encoded textures for browser-side rendering
• Derive new products from ingested data — climatologies, anomalies, promotions — with declarative recipes run by a generic derivation engine
• Analyze data using pluggable modules that integrate with the Xarray-compatible scientific Python ecosystem
• Visualize with modern browser-side rendering (WeatherLayers GL), moving beyond legacy WMS

Architecture at a Glance

For the detailed architecture with diagrams, data model, and design decisions, see docs/architecture/README.md.

---

Tech Stack

Component	Technology
Core Framework	Django 5.x + Wagtail 7.x
Database	PostgreSQL 18 + TimescaleDB + PostGIS (via PgBouncer)
Object Storage	MinIO (S3-compatible), multi-bucket
Task Queue	Celery + Redis (three queues)
Tile Servers	Titiler (raster COGs) + Martin (vector/MVT)
Discovery APIs	STAC API + OGC API – EDR
Data Formats	COG, virtual Zarr (kerchunk / Icechunk)
Event Bus	MinIO → Redis list → minio-consumer
Multi-tenancy	Org-scoped rows, org-first storage paths, API keys
Containerization	Docker Compose
Package Manager	uv (pyproject.toml + uv.lock)

---

Getting Started

Prerequisites

• Docker and Docker Compose (v2+)
• Git

Quick Start (Production)

1. Clone the repository

   git clone https://github.com/wmo-raf/georiva.git
   cd georiva

2. Configure environment variables

   cp .env.sample .env

   Edit .env and set the required values. At minimum, you need to set:

   • SECRET_KEY — Django secret key
   • GEORIVA_DB_USER, GEORIVA_DB_NAME, GEORIVA_DB_PASSWORD — database credentials
   • MINIO_ROOT_USER, MINIO_ROOT_PASSWORD — MinIO credentials
   • ALLOWED_HOSTS — comma-separated list of allowed hostnames
   • CSRF_TRUSTED_ORIGINS — comma-separated list of trusted origins

   See .env.sample for the full list of options. (MinIO events are delivered to a Redis list and drained by the minio-consumer service — there is no webhook endpoint or token to configure.)

3. Start the stack

   docker compose up -d

   On first run, the entrypoint automatically handles database migrations and static file collection.

4. Bootstrap the central organisation

   GeoRiva is multi-tenant: every row, and the first segment of every storage key, belongs to an organisation. A fresh install has none, so create one before ingesting anything.

   docker compose exec georiva python manage.py bootstrap_central_org

   This is idempotent, and claims Wagtail's default Site at your base domain. Additional institutions get their own org (and their own hostname) via the create_organisation command.

5. Access GeoRiva

   Open http://localhost in your browser.

   Additional services:

   • STAC Browser: http://localhost/stac-browser/

Development Setup

The dev setup uses a compose override that mounts your source code for hot reloading.

1. Follow steps 1–2 from Quick Start above.

2. Start with the dev override

   docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d

   This gives you:

   • Django dev server with auto-reload on code changes
   • Celery worker with auto-reload
   • Source code mounted from ./georiva into the container

   Optionally, create a shortcut in your shell:

   alias dc-dev="docker compose -f docker-compose.yml -f docker-compose.dev.yml"

3. Useful commands

   # View logs
   docker compose logs -f georiva
   
   # Run management commands
   docker compose exec georiva python manage.py createsuperuser
   docker compose exec georiva python manage.py shell
   
   # Rebuild after dependency changes
   docker compose build georiva
   
   # Restart a single service
   docker compose restart georiva-celery-worker

Managing dependencies: GeoRiva uses uv. Core dependencies live in georiva/pyproject.toml and are locked in georiva/uv.lock (there is no requirements.txt). Add packages with make uv-add pkg="<package>", then rebuild. For local IDE/test tooling outside Docker, run uv sync --all-packages at the repo root. See docs/contributing.md for details.

Installing Plugins

Plugins can be installed at build time or at runtime.

Build time — declare plugins in plugins.toml (copy plugins.toml.sample and edit), then build:

docker compose build georiva

Runtime — set GEORIVA_PLUGIN_URLS in your .env file and restart. Ensure GEORIVA_DISABLE_PLUGIN_INSTALL_ON_STARTUP is not set to "true".

Local development — clone plugin repos into dev-plugins/ and run with the dev override (make dev-up OV=1); each subdirectory is auto-discovered and editable-installed. See the Plugin Installation Guide.

Project Structure

georiva/src/georiva/      # Main Django/Wagtail application
├── config/               # Settings (base/dev/production), URLs, Celery, WSGI/ASGI
├── core/                 # Published STAC data models, multi-bucket storage, machine plane
├── staging/              # Staging tier — source-grained STAC models + DerivationLink lineage
├── organisations/        # Multi-tenancy: row-level scoping, membership, per-org page trees
├── accounts/             # Per-user API keys (grv_…) + DRF authentication
├── ingestion/            # Async ingestion pipeline, IngestionLog, MinIO event consumer
├── formats/              # Format handler plugins (GRIB, NetCDF, GeoTIFF) + registry
├── sources/              # Source-plugin framework (DataSource, LoaderProfile, fetch strategies)
├── processing/           # Derivation engine, recipe registry, derivation tasks
├── geoprocessing/        # Pure compute library (algebra, regrid, temporal, zonal) — no Django
├── stac/                 # STAC API
├── edr/                  # OGC API – EDR (metadata plane)
├── wmts/                 # WMTS capabilities (GetTile is served by titiler-app)
├── analysis/             # Time-series + zonal-statistics modules
├── virtual_zarr/         # Per-Variable virtual Zarr (kerchunk / Icechunk) manifests
├── visualization/        # Wagtail admin hooks for map/tile config
└── pages/                # Wagtail CMS pages

titiler-app/              # Custom Titiler tile server (FastAPI)
source-plugin-boilerplate/# Cookiecutter template for new source plugins
deploy/                   # Nginx, Martin, plugin install scripts
docs/                     # Architecture, data model, plugin, and storage docs

For a deeper map of conventions and patterns, see docs/.

---

Contributing

GeoRiva is in its early stages and contributions are welcome — especially feedback on the architecture.

Where to start:

1. Read the Architecture Design Document to understand the system design
2. Check the Open Questions section for areas where input is needed
3. See docs/contributing.md for development setup and guidelines

Ways to contribute:

• Review and comment on the architecture
• Build a source plugin for a data provider you know well
• Build an analysis module for your domain
• Improve documentation
• Report bugs and suggest features via issues

---

Documentation

Start at the documentation index, which ties everything together. Key documents:

Document	Description
Documentation Index	Map of all docs and a suggested reading order
Architecture Design Document	Full system architecture, data model, and design decisions
Data Model Guide	How to organize data into Catalogs, Collections, Variables
Format Plugin System	Reading GRIB/NetCDF/GeoTIFF; writing a new format plugin
Storage & Ingestion Architecture	Buckets, event-driven ingestion, IngestionLog
Download Deduplication	Multi-collection feeds and download dedup
Source Plugin Contract	CollectionDefinition contract and the setup wizard
Architecture Decision Records	Every architectural decision, with the reasoning behind it
Contributing Guide	How to set up a dev environment and contribute

---

License

[TBD]