Analytics/ML Data Platform

Overview

This repository provides a reference architecture for building production-grade data platforms designed for small to medium-sized businesses. It demonstrates a complete data stack—from ingestion through transformation to machine learning deployment—using modern, open-source technologies that can scale with organizational growth.

The platform addresses common challenges faced by organizations building data capabilities: fragmented tooling, unclear data flow, lack of orchestration, and difficulty productionizing machine learning models. This implementation offers a tested blueprint that can be adapted to various use cases including business intelligence, analytics applications, and ML-driven products.

Architecture

The platform follows a layered architecture pattern with clear separation of concerns:

┌─────────────────────────────────────────────────────────────────────┐
│                         Data Sources                                │
│                    (REST APIs, Databases, etc)                      │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                     Orchestration Layer                             │
│                           Dagster                                   │
│  - Asset-based pipeline definitions                                 │
│  - Incremental loading with dlt                                     │
│  - Scheduling and sensor-based triggers                             │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      Data Warehouse                                 │
│                    MotherDuck (DuckDB)                              │
│              Centralized storage for all data assets               │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Transformation Layer                             │
│                            dbt                                      │
│                                                                     │
│    Raw Layer → Staging Layer → DWH Layer → Data Mart Layer         │
│  (Source data) (Type casting)  (Business   (Aggregations,          │
│                                  logic)     Analytics)              │
└─────────────────────────┬───────────────────────────────────────────┘
                          │
          ┌───────────────┼───────────────┐
          │               │               │
          ▼               ▼               ▼
┌──────────────┐  ┌─────────────┐  ┌────────────────┐
│  ML Pipeline │  │   Data API  │  │   Analytics    │
│              │  │             │  │                │
│  - Training  │  │  FastAPI    │  │  Data Marts    │
│  - Inference │  │  - REST API │  │  - Reporting   │
│  - MLflow    │  │  - OpenAPI  │  │  - Ad-hoc      │
│    Registry  │  │             │  │    Analysis    │
│              │  │             │  │                │
│  Runs in K8s │  └──────┬──────┘  └────────────────┘
│  Pods        │         │
└──────────────┘         ▼
                  ┌─────────────┐
                  │   Dash App  │
                  │             │
                  │ Interactive │
                  │    Viz      │
                  └─────────────┘

Technology Stack

Orchestration: Dagster

• Asset-based pipeline definitions for clear data lineage
• Integrates dlt (data load tool) for API extraction with state management
• Supports scheduled runs and event-driven triggers
• Provides web UI for monitoring and debugging

Data Warehouse: MotherDuck

• Cloud-hosted DuckDB for OLAP workloads
• Columnar storage optimized for analytical queries
• Accessible from multiple components (orchestrator, transformation, serving)
• Supports structured and semi-structured data

The current implementation uses MotherDuck, but the architecture supports any cloud data warehouse (Snowflake, BigQuery, Redshift, etc.) with appropriate connection configuration changes.

Transformation: dbt

• SQL-based transformations with version control
• Four-layer data model architecture (Raw, Staging, DWH, Data Mart)
• Built-in data quality testing
• Multi-environment support (development, QA, production)
• SQLFluff integration for SQL linting

Machine Learning: MLflow + Kubernetes

• Model training orchestrated through Dagster
• ML workloads execute in isolated Kubernetes pods
• MLflow for experiment tracking and model registry
• Batch inference pipelines
• Uses scikit-learn and Polars for data processing

Data Serving: FastAPI + Dash

• FastAPI backend provides REST API access to warehouse data
• OpenAPI documentation auto-generated
• Dash frontend for interactive data visualization
• Both components query warehouse directly

Infrastructure: Kubernetes

• Containerized deployment for all components
• Kubernetes manifests and Helm charts included
• Designed for any K8s environment (cloud or on-premises)
• Scalable based on workload requirements

Data Flow

1. Ingestion: Dagster orchestrates data extraction from external sources using dlt, which handles incremental loading and state persistence. Data lands in the Raw layer of the warehouse.

2. Transformation: dbt reads from Raw, applies type casting and light transformations in Staging, builds dimensional models in the DWH layer, and creates aggregated data marts for consumption.

3. Consumption: Transformed data is accessed by three primary channels:

   • ML pipelines for training and inference
   • FastAPI for programmatic data access
   • Data marts for analytics and reporting

4. ML Lifecycle: Models are trained in Kubernetes pods, tracked in MLflow, and deployed for batch inference through Dagster pipelines. Predictions can be written back to the warehouse or served via API.

Deployment

The platform is designed for containerized deployment:

• Each component includes Dockerfile and Kubernetes manifests
• Helm charts available for streamlined deployment
• CI/CD pipeline implemented via GitHub Actions (testing, linting, deployment)
• Environment-specific configuration through environment variables
• Designed for multi-tenant usage (can host multiple projects or products)

While production deployment targets Kubernetes, individual components can run locally for development purposes.

Local Development

Components can be developed and tested independently:

1. Clone the repository
2. Navigate to the specific component directory (dagster-proj, dbt-proj, etc)
3. Set required environment variables for warehouse connection
4. Run component-specific commands:
   • Dagster: dagster dev
   • dbt: dbt build or dbt run
   • FastAPI: uvicorn or fastapi dev
   • ML pipelines: Python scripts or Jupyter notebooks

Each component directory contains its own README with detailed setup instructions.

Current Implementation

This repository includes a working example using NBA sports data sourced from the ESPN API. The implementation demonstrates the full data lifecycle: API ingestion, dimensional modeling of game and player data, and a binary classification model for game outcome prediction.

This platform is fully deployed and running on Kubernetes in production, providing a real-world validation of the architecture and deployment patterns.

The sports analytics use case serves as a reference implementation. The platform architecture is domain-agnostic and can be adapted to other industries and data sources.

Scalability and Extensibility

The architecture supports growth in multiple dimensions:

• Data volume: MotherDuck and dbt handle increasing data volumes through partitioning and incremental models
• Pipeline complexity: Dagster's asset-based approach scales to hundreds of pipelines
• ML workloads: Kubernetes enables horizontal scaling of training and inference jobs
• Data sources: dlt supports 200+ connectors for various APIs and databases
• Team size: Component separation allows multiple teams to work independently

The platform currently handles structured and semi-structured data. Extension points exist for unstructured data processing (images, video, documents) through additional pipeline stages.

Documentation

Technology-specific documentation is located in each component directory:

• dagster-proj/ - Orchestration and ingestion details
• dbt-proj/ - Data modeling and transformation guide
• ai-ml/ - ML pipeline and MLflow setup
• data-app/ - API and visualization application setup
• deployment/ - Kubernetes deployment instructions

Requirements

• Python 3.11+ (managed via uv package manager)
• Kubernetes cluster (for production deployment)
• Cloud data warehouse (current implementation uses MotherDuck; alternatively Snowflake, BigQuery, Redshift, or other SQL-based warehouses)
• Docker (for containerization)